EVALUATION REPORT ENDLINE PERFORMANCE EVALUATION OF USAID-FUNDED MATERNAL AND CHILD SURVIAL PROGRAM (MCSP) IN MYANMAR October 2018 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Social Impact, Inc. Photo Credit: Dr. Myat Pan Hmone, at the MNMA L&PIC site, 2018 EVALUATION REPORT ENDLINE PERFORMANCE EVALUATION OF USAID-FUNDED MATERNAL AND CHILD SURVIAL PROGRAM (MCSP) IN MYANMAR USAID/Burma Program Development Office AID-486-I-14-00001 (IDIQ) 720-482-18-F-00003 (Task Order) October 2018 Dr. Myat (Crystal) Pan Hmone (Team Leader) Dr. May Thet Kyaw (Research Specialist) Social Impact, Inc. 2300 Clarendon Blvd., Suite 1000, Arlington, VA 22201 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States government. TABLE OF CONTENTS ABSTRACT.......................................................................................................................................................................... I ACKNOWLEDGEMENTS .............................................................................................................................................. II ACRONYMS ...................................................................................................................................................................... III EVALUATION PURPOSE AND QUESTIONS.......................................................................................................... 3 EVALUATION PURPOSE........................................................................................................................................... 3 EVALUATION QUESTIONS..................................................................................................................................... 3 METHODOLOGY ............................................................................................................................................................ 4 DATA COLLECTION METHODS .......................................................................................................................... 4 SAMPLING ..................................................................................................................................................................... 5 ETHICAL REVIEW........................................................................................................................................................ 5 FIELD WORK................................................................................................................................................................ 6 DATA ANALYSIS ......................................................................................................................................................... 6 LIMITATIONS ................................................................................................................................................................ 6 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS .............................................................................. 7 EQ 1: TO WHAT EXTENT DID MCSP ASSISTANCE INFLUENCE CHANGES IN IN-SERVICE TRAINING PRACTICES AND RELATED SYSTEMS TO IMPROVE MATERNAL, NEONATAL, AND CHILD HEALTH?.......................................................................................................................................................... 7 EQ 2: HOW HAVE MCSP’S APPROACHES CONTRIBUTED TO THE POTENTIAL SUSTAINABILITY OF PROJECT RESULTS?........................................................................................................ 21 EQ 3: WHAT ARE THE SPECIFIC LESSONS THAT CAN BE LEARNED TO INFORM FUTURE PROGRAMS THAT AIM TO STRENGTHEN SYSTEMS FOR CAPACITY BUILDING RELATED TO MCH, PARTICULARLY AT THE TOWNSHIP LEVEL?.................................................................................... 24 ANNEXES ......................................................................................................................................................................... 29 ANNEX A. EVALUATION SCOPE OF WORK................................................................................................. 29 ANNEX B. DOCUMENT REVIEW SOURCES................................................................................................... 38 ANNEX C. DATA COLLECTION TOOLS......................................................................................................... 40 ANNEX D. EVALUATION TEAM MEMBERS (COMPONENT A) ............................................................... 66 ANNEX E. EVALUATION DESIGN MATRIX .................................................................................................... 67 ANNEX F. INTERVIEWEE LIST.............................................................................................................................. 68 ANNEX G. DISCLOSURE OF CONFLICT OF INTEREST ............................................................................. 71 TABLE OF FIGURES FIGURE 1. MAP SHOWING THE GEOGRAPHIC COVERAGE OF MCSP PROJECT SITES 4 FIGURE 2. STANDARD OPERATING PROCEDURES FOR SMALL BABIES (KMC) 7 FIGURE 3. MINISTER OPENING THE L&PIC AT WOMEN AND CHILDREN HOSPITAL, TAUNGGYI 9 FIGURE 4. EFFECTIVENESS OF THE IN-SERVICE CAPACITY BUILDING TRAINING AT L&PIC SHOWN BY MASTER MENTORS’ SKILLS AND KNOWLEDGE RETENTION 12 FIGURE 5. AVERAGE QUALITY IMPROVEMENT PERFORMANCE SCORE OF FIVE L&PIC AFFILIATED CLINICAL SITES (SITTWE, TAUNGGYI, LASHIO, MAGWAY, PATHEIN) 13 FIGURE 6. IMNCI UPDATED GUIDELINE PRE- AND POST-TEST COMPARISON 13 FIGURE 7. HUSBANDS PRACTICING KMC AT TAUNGGYI WOMEN’S HOSPITAL 14 FIGURE 8. BEMONC WORKSHOP FOR EHO HEALTH CARE PROVIDERS AT MYAWADDY GENERAL HOSPITAL 15 TABLE OF TABLES TABLE 1. SAMPLE SIZES, ACCORDING TO LOCATION, RESPONDENT TYPE, AND DATA COLLECTION METHOD FOR EACH ACTIVITY 5 i ABSTRACT The United States Agency for International Development in Myanmar engaged Social Impact to assess the Maternal and Child Survival Program (MCSP), implemented by Jhpiego (lead agency), Save the Children, John Snow, Inc., and Broad Branch Associates. The program was carried out to respond to strategic priorities of the Ministry of Health and Sports (MOHS) for improving maternal, newborn, and child health by strengthening the institutional capacity of in-service midwifery. MCSP improved the in-service capacity of health staff by developing and updating maternal and child health-related policies, guidelines, and standard operating procedures. It also provided in-service training in best practices and fostered an enabling environment. The activity was aligned with MOHS interests and yielded strong government buy-in. MCSP’s support for the existing cascade training model, quality improvement approaches, and setup of the Learning and Performance Improvement Center (L&PIC) sites were in line with the MOHS’ policy. The effectiveness of MCSP was demonstrated by improved staff performance scores and high knowledge retention rates. MCSP supported the creation of professional development opportunities for midwives, lady health visitors, and other basic health staff via continuous education, accreditation, and a clinical skills-based assessment relicensing system. MCSP successfully engaged with ethnic health organizations (EHOs) and increased trust and coordination between the MOHS and EHOs. MCSP has the potential to be sustainable due to the improved staff capacity and the establishment of L&PIC sites. ii ACKNOWLEDGEMENTS The evaluation team would like to thank many individuals and organizations for their support. This evaluation would not have been possible without support from the implementing partners (IPs): Jhpiego, Save the Children, and John Snow, Inc. (JSI), particularly the Chief of Party and staff from Jhpiego, who provided invaluable information, support, and facilitation throughout the evaluation process. The team would also like to extend its gratitude to staff from the Department of Public Health (DPH) and Myanmar’s Ministry of Health and Sports (MOHS), particularly senior staff from Nay Pyi Taw who helped ensure that necessary letters were issued in a timely manner prior to field data collection. Special thanks to all the respondents from the MOHS: the state and regional health team from Taunggyi, Ayeyarwady, and Yangon (Thanlyin) from DPH and the Department of Medical Services; the president and executive members of the Myanmar Nurses and Midwives Association (MNMA); the president and training team members from the Myanmar Nurses and Midwives Council (MNMC); and senior staff from the Karen Department of Health and Welfare (KDHW). They articulated insightful, experienced-based reflections on how MCSP supports in-service midwifery institutional capacity to deliver lifesaving maternal, newborn, and child health (MNCH) care, the potential for sustainability, what worked well, and what did not work well. Lessons learned and challenges shared from their experience will inform future USAID-supported program design. The team would also like to acknowledge USAID staff who met with the evaluation team and/or facilitated the evaluation team’s work and ensured an objective and comprehensive evaluation process. Finally, the team appreciates the support of Social Impact’s headquarters staff, who have been instrumental in all phases of the evaluation process. iii ACRONYMS 3MDG The Three Millennium Development Goal Fund AMW Auxiliary Midwife ANC Antenatal Care BHS Basic health Staff CSSAC Clinical Skills Standardization and Assessment Center DMS Department of Medical Science DPH Department of Public Health EHO Ethnic Health Organization EmOC Emergency Obstetric Care EQ Evaluation Question ERC Ethics Review Committee F-IMNCI Facility-based Integrated Management of Newborn and Childhood Illness FGD Focus Group Discussion GoM Government of Myanmar HMIS Health Management Information System HR Human Resources IMNCI Integrated Management of Newborn and Childhood Illness INGO International Non-governmental Organization IP Implementing Partner IR Intermediate Result IST In-service Training Jhpiego Johns Hopkins Program for International Education in Gynecology & Obstetrics JSI John Snow Incorporated KDHW Karen Department of Health and Welfare KII Key Informant Interview KMC Kangaroo Mother Care L&PIC Learning and Performance Improvement Center LHV Lady Health Visitor LBW Low Birth Weight M&E Monitoring and Evaluation MCH Maternal and Child Health MCHIP Maternal and Child Health Integrated Program MDHS Myanmar Demographic and Health Survey MNCH Maternal, Neonatal, and Child Health MNMC Myanmar Nurse and Midwife Council MOHS Ministry of Health and Sports MRH Maternal and Reproductive Health NGO Non-governmental Organization NHP National Health Plan NIMU NHP Implementation Monitoring Unit NPT Nay Pyi Taw O&G Obstetrics and Gynecology PE Performance Evaluation PE/E Pre-eclampsia / Eclampsia PHS Public Health Supervisor PPH Postpartum Hemorrhage QA Quality Assurance QI Quality Improvement RHC Rural Health Center RS Research Specialist S/R/T State/Regional/Township SCI Save the Children International SI Social Impact SN Senior Nurse SOP Standard Operating Procedure TA Technical Assistance TTBA Trained Traditional Birth Attendant TL Team Leader UNICEF United Nations Children’s Emergency Fund UNOPS United Nations Office for Project Services USD United States Dollar WCH Women and Child Hospital (Taunggyi) iv Executive Summary PROGRAM BACKGROUND As part of a global cooperative agreement to introduce and support high-impact health interventions to end preventable child and maternal deaths, the United States Agency for International Development (USAID) in Myanmar invested in USAID’s global Maternal and Child Survival Program (MCSP) from July 2015 to June 2018. Jhpiego (lead agency), Save the Children International, John Snow, Inc. (JSI), and Broad Branch Associates were the lead MCSP health and development partners working in Myanmar who implemented the program in seven out of 17 states and regions (S/R), aiming to strengthen the midwifery institutional capacity to deliver lifesaving maternal, newborn, and child health (MNCH) interventions. Key partners were the Ministry of Health and Sports (MOHS), Myanmar Nurse and Midwife Council (MNMC), and Myanmar Nurse and Midwife Association (MNMA). EVALUATION PURPOSE This evaluation report examines the extent to which MCSP’s interventions supported Myanmar’s capacity and systems for in-service training of health care providers to deliver quality MNCH services. The evaluation analyzed the effectiveness of in-service capacity building approaches utilized by MCSP, including the Learning and Performance Improvement Center (L&PIC) model, the rollout approach for competency-based capacity building at the S/R level and below, the standards-based quality improvement (QI) model, and efforts to strengthen MNMC and MNMA. Findings will not only be used to inform approaches for enhancing in-service training at lower levels under USAID’s follow-on Essential Health Program, but also to generate recommendations for USAID and other development partners on how to optimize support to the MOHS to deliver integrated in-service training interventions and build related country systems. EVALUATION DESIGN AND METHODOLOGY Three main evaluation questions (EQs) were: 1. To what extent did MCSP assistance influence changes in in-service training practices and related systems to improve maternal, neonatal, and child health? 2. How have MCSP’s approaches contributed to the potential sustainability of project results? 3. What are the specific lessons that can be learned to inform future programs that aim to strengthen systems for capacity building related to MCH, particularly at the township level? Data Collection Methods: The evaluation team used a mixed-method evaluation design involving document review, primary qualitative data from key informant interviews (KIIs) and focus group discussions (FGDs), and secondary quantitative evidence. Respondents included senior staff from Department of Public Health (DPH) and S/R training team members (Department of Public Health and Department of Medical Services), staff from implementing partners (IPs), professional bodies (MNMA and MNMC), Karen Department of Health and Welfare (KDHW, an Ethnic Health Organization), and USAID. Sampling: Data were collected from Yangon/Thanlyin, Nay Pyi Taw (NPT), Shan, Ayeyarwady, and Kayin and 64 purposively selected respondents, in 27 KIIs, seven FGDs (31 participants in total), and two additional mothers’ group discussions (6 in total). Data Analysis: The evaluation team triangulated evidence across qualitative and quantitative data sources as well as responses from various stakeholders. Data were disaggregated by types of stakeholder and administrative level (central or state and regional), and the evaluators used content and comparative analysis of coded KII and FGD interview notes to answer each question. Key Challenges/Limitations: Staff turnover, outdated contact phone numbers, and budget constraints for some state/regional training team members to travel from rural health centers made it difficult to reach the originally targeted respondents. The evaluation team tracked down respondents, identified new potential respondents with inputs from MOHS staff, and coordinated to overcome travel constraints. A flood delayed v data collection in Kayin, however, the team managed to interview representatives from KDHW in a timely manner. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS EQ 1: To what extent did MCSP assistance influence changes in in-service training practices and related systems to improve maternal, neonatal, and child health? 1a. MCSP supported changes in policies, practices, and the enabling environment for in-service capacity building (training) at different system levels. MCSP supported changes in in-service training practices and related systems across the central and S/R levels of the MOHS, as well as in MCSP project townships. The evaluation found evidence that MCSP enhanced and updated policies and guidelines such as antenatal care (ANC) and Integrated Management of Neonatal and Child Illness (IMNCI), increased capacity of targeted health care providers to apply MNCH best practices, and supported the creation of an enabling environment for in-service capacity building (e.g., through Learning and Performance Improvement Centers (L&PIC); support to professional councils). MCSP approaches such as the cascade training model, quality improvement (QI) approach, post￾training follow-up and supervision activities were found to align with the existing MOHS plans and policies for improving MNCH. The post-training approach allowed MCSP and local health departments to monitor and assess the impact of training for a longer period and to reinforce correct practices. MCSP, in collaboration with the MOHS, successfully demonstrated the implementation of updated MNCH practices. The effectiveness of L&PIC in-service capacity building was supported with evidence in improved capacity of training team members, which was shown by the increase in post-test scores and an 80 percent retention rate of master mentors’ skills and knowledge six months post-training. Evidence found that MCSP implemented an effective and standardized approach to in-service capacity building, which was endorsed by the MOHS. Both state and non-state EHO actors had access to the standardized training approach and curriculum. With a high-level commitment to MCSP interventions, MCSP-supported policies, guidelines, and standard operating procedures (SOPs) may be used in future nationwide trainings. The integration of MCSP-supported models into Myanmar’s National Health Plan (NHP) Annual Operational Plans, such as cascade training models, QI, and post-training follow-up, and the MOHS’ interest in expanding L&PIC sites in other S/R, demonstrated how MCSP supported in-service capacity building at the system level. Capacity development in maternal care was evidenced by changes in practice in the management of postpartum hemorrhage (PPH), hypertension in pregnancy, and effective use of partographs. However, some respondents reported being unconfident about applying new skills, and shared challenges of using condoms and magnesium sulfate in managing PPH and eclampsia. Capacity development in newborn and child health care was observed through reported changes in practice around newborn care, cord care, and kangaroo mother care (KMC) for small babies, as well as diagnosis and referral for high-risk child illnesses. One key finding was that L&PIC models were being used effectively at all sites visited, and the desk review suggested that all sites were functioning well. Respondents from MNMA shared that MCSP’s L&PIC sites enabled student nurses and midwives to learn, apply, and practice their skills using practice models. Around 60 percent of evaluation respondents felt that the continuing nursing education approach contributed to increases in health workers’ capacity. However, the evaluation team identified maintenance challenges of L&PIC equipment. 1b. Align to health system realities to address key barriers for strengthening in-service training MCSP was aligned with the Myanmar government’s interests and the MOHS’ agenda. It supported the MOHS’ strategic priority of strengthening human resources for health by building the capacity of existing health workers to deliver MNCH services. Consultations with the central MOHS at the design stage contributed to program success. However, some MOHS staff from child health division in NPT and S/R staff suggested the need for more engagement with respective departments at the central and S/R levels in future programs. The evaluation team found that when planning for training and other related activities, MCSP had more engagement with the central MOHS than with the S/R directors. This can be attributed to MCSP’s main scope focusing on demonstration and documentation at the central and S/R levels. vi Demonstration at the township level was added to the project cycle at a later stage. MCSP was found to have addressed health system barriers mainly at the central and S/R levels and MCSP-focused townships. Many basic health staff stressed the importance of information sharing between frontline health staff and the MOHS. They shared that if they were able to share implementation challenges in the field with the management staff directly, they might receive better guidance on addressing the challenges. Although MCSP addressed most key barriers to strengthening in-service training, some factors were out of MCSP’s scope, such as government budget and financial constraints for training, human resource issues including staff shortages, attrition, and turnover, and geographical constraints. Similarly, space constraints at the MNMC site was out of MCSP’s scope, however, MNMC senior leadership expressed a need for addressing the space constraint at their L&PIC site, and suggested that this site could serve as an exam center, which provided a window of opportunity for MCSP. 1c. Align to address drivers of maternal and child mortality and morbidity MCSP’s approach was aligned to address the drivers of MNCH morbidity and mortality. Based on KII and FGD responses and the desk review, the evaluation team found that MCSP’s approach not only strengthened the capacity of health workforce to deliver quality MNCH service through capacity building, but also maintained staff performance through the QI initiative. The quality of care was reinforced by standardized practices using policies, guidelines, and SOPs developed with the support of MCSP. Preterm and low-birth-weight (LBW) neonates (the leading cause of neonatal deaths) had a better chance of survival via KMC. According to MOHS staff, it was too early to determine the level of MCSP’s contribution to reducing MNCH mortality and morbidity. Jhpiego was working on an impact modeling analysis at the time of this evaluation, and the report could help estimate MNCH mortality and morbidity. EQ1 Recommendations For USAID: (1) Continue engagement with the MOHS and maintain the good practice of central-level buy￾in from the MOHS. At the S/R and township levels and below, the Mission should encourage the implementing partners to engage at all levels in future programming while searching for opportunities to reach out at all levels for a realistic field experience as resources and time permit. (2) Strengthen and expand the effective collaboration of the MOHS and EHOs, not limited to KDHW alone, to align with and complement the NHP aim of delivering the essential package of health services. (3) Provide support and assistance to the MOHS to reach the township level and below while ensuring cost effectiveness and efficiency. For the IP: (1) Support should not be limited to the central-level staff in NPT and should include representative staff from both DPH and DMS at the S/R and township levels in pre-planning consultation. (2) Review resources put in L&PICs and ensure the sites’ ability to maintain these resources. For the MOHS: (1) Continue engagement between the MOHS and EHOs to ensure access to a standardized curriculum and approach. (2) Continue briefing meetings between midwives and township-level staff in the work plan so that central- or township-level health staff can share updated training with midwives and midwives can share challenges encountered (perhaps meet on paydays, when midwives from rural areas come to township offices). (3) To encourage the trained health staff on the use of condoms as a tamponade to stop massive PPH and magnesium sulfate for eclampsia in managing patients. (4) Strengthen the professional bodies (especially MNMC) and use the opportunity of leadership change in MNMC to review the Clinical Skills Standardization and Assessment Center (CSSAC) site at MNMC for possible space expansion. (5) Support the L&PIC site at MNMA as a training and exam preparation resource center that can serve as a fund-raising source. Provide support for continued professional education of MNMA members via the continuous nursing education approach, standardized training modules, and user-friendly training models. EQ 2: How have MCSP’s approaches contributed to the potential sustainability of project results? 2a/b. Potential sustainability and supporting key factors/evidence The sustainability of MCSP’s work was enhanced by the MOHS’ endorsement of guidelines and policies developed with MCSP’s support, as well as the government’s commitment to apply and vii replicate MCSP activities. For example, the MOHS planned to expand the L&PIC sites and integrate the standardized and modularized training curricula in nationwide training. In addition, the technical approach of cascading training from the central to the S/R levels to the MCSP-focused townships also contributed to MCSP’s sustainability. The MOHS’ NHP Annual Operational Plan intended to adopt MCSP’s cascade training model in developing a nationwide integrated training model and plan. MCSP’s work was further enhanced by senior government level commitment and leadership support of the Activity. MCSP-trained staff retained a high-level of MNCH knowledge and skill. The evaluation team found that 80 percent of master mentors retaining knowledge six months after training. Other MCSP contributions included increasing the post-test skills of basic health staff, improved QI performance assessments of clinical staff, and increases in training team members’ confidence and motivation. MCSP-supported tools for KMC were a scalable and sustainable model, which was evidenced by the MOHS’ interest in the activity and the high acceptance of the KMC model and staff commitment at Taunggyi Women’s Hospital and Thanlyin General Hospital. The evaluation team found that the curricula would be more sustainable if it were more comprehensive and included aspects of nutrition, exclusive breastfeeding and other key topics and if all curricula across pre-service training institutions were updated at the same time. Having maternal health training materials in Burmese would have contributed to the sustainability of MCSP, as midwives reported having difficulty using PowerPoint presentations and guidelines in English. EQ 2 Recommendations For USAID: (1) Support efforts of the MOHS to sustain government ownership and commitment even after leadership changes. (2) Support application and replication of effective MCSP interventions in USAID’s new Essential Health Program, which is also aligned with the NHP Essential Health Care package. (3) Identify, encourage, and support motivated staff from the MOHS who are at decision-making levels and recognize them through a “champion” pool. For IPs: (1) Continue advocating for the integration of MCSP cascade training modules into the MOHS National Health Plan. (2) Include training modules with comprehensive information on the nutrition component, such as exclusive breastfeeding, maternal and child nutrition, etc. (3) Follow up on and implement the sustainability plan developed in consultation with the MOHS and key stakeholders and secure support from other sources if feasible (e.g., Essential Health, Access to Health, etc.). (4) Support (technically and financially) the MOHS in integrating MCSP activities into the MOHS NHP, particularly the integrated training plan, QI process, and L&PIC extension. For the MOHS: (1) Integrate MCSP best-practice activities in the MOHS NHP Operational Plan and annual work plans. (2) Strengthen a dedicated training team at the S/R level to focus on trainings, equipment, and models used in L&PIC and the functioning of L&PIC sites. (3) Update clinical and training guidelines in DPH, DMS, and pre-service training at nursing universities at the same time so that MCSP and non-MCSP trained staff have an equal opportunity to learn updated MNCH techniques and practices. (4) Allocate a training budget in the annual work plan and train MOHS township-level staff to understand ministry budget and finance systems to be able to request training resources from S/Rs and the central level in a bottom-up approach. (5) Develop training materials and resources in Burmese and, if feasible, ethnic languages, particularly in preparation for township health staff- and below-level trainers. EQ 3: What are the specific lessons that can be learned to inform future programs that aim to strengthen systems for capacity building related to MCH, particularly at the township level? 3a: Approaches/interventions that should/should not be supported/replicated through future assistance There was no negative feedback on MCSP’s strategy and approaches. The evaluation team found that MCSP￾implemented programs were in line with interests and policies of the MOHS. Some interventions are likely to be replicated, such as trainings, post-training follow-up, QI, and KMC. The L&PIC model was also praised by most respondents. One of the key lessons learned was the importance of the MOHS’ leadership at the central level and in clinical sites, the inclusion of which at the design stage played a key role in the success of program implementation. Collaboration within the MOHS system and among all stakeholders, as well as the orientation of stakeholders at all levels, were important lessons for the Activity. viii 3b. Challenges to address for future programs, particularly at the township level and below MCSP supported in-service capacity building of the health workforce via a multipronged approach. It also addressed health system capacity building challenges from various perspectives. However, MCSP, which mainly reached the S/R level, would have generated a greater impact if had reached the township level. The evaluation team found that the mitigation plan for replacing broken models at L&PIC sites was not in place during field visit. Some challenges were out of MCSP’s scope, such as human resource constraints (staff turnover, attrition), coordination between government departments, and political challenges (e.g. conflict in Rakhine). The effectiveness of the training team depended on human resources, financial resources, leadership dedication, and team motivation. 3c: Interventions/supports that should be removed/modified to address health system realities Flexible approaches to developing work plans maximized outcomes but required planning in advance, since all relevant stakeholders needed timely information about program activities. MCSP promoted effective collaboration between the Maternal and Reproductive Health Division and Child Health Development, DPH, and the MOHS. Respondents believed that activities and training plans should be communicated ahead of time and to a broader range of relevant stakeholders, including S/R leaders (director level) and relevant authorities from the clinical side for future programming. 3d: Modifications to the models and interventions, including their future mode of delivery There has been a need for decentralization or inclusive approach to extend the MCSP’s reach to the township level and below. The MOHS should pay attention to staff replacement due to turnover, retirements, and personal or ministry-assigned absences. The theory of change and the training plan needed to be developed in advance and with all stakeholders. Future work needs to be done with the professional bodies to ensure a sustainable stream of income. The criteria for selecting training team members must be realistic and in accordance with available workforce capacity. The evaluation team found that only very few child specialists met the high qualification requirements for Facility-based Integrated Management of Neonatal and Childhood Illness (F-IMNCI) post-training supervisors. Due to busy schedule and geographical constraints, many were unable to conduct frequent post-training follow-up visits. EQ 3 Recommendations For USAID: (1) Consider supporting the MOHS and IPs to reach out at the township level and below in a future similar program by setting up mini L&PICs at district and township levels, which would be based on the MCSP L&PIC model but tailored to fit the peripheral level in consultation with staff from districts and townships. (2) Consider providing technical and financial assistance to the MOHS to promote capacity of MOHS staff, particularly at the lower level, and encourage decentralization by linking with the MOHS NHP Operational Plan. (3) Ask that IPs include the MOHS in discussing potential changes in work plans, funding mechanisms, and expected deliverables. (4) Consider supporting the skilled lab (L&PIC) site at MNMA as a training and exam preparation resource center, and as a fundraising source, by allowing nursing students and medical students to practice using models at training sites. For IPs: (1) Continue the momentum of high-scoring intervention models based on the findings from this evaluation, such as training, post-training follow-up, and QI models. Continue the best practice of engaging the MOHS at the central level. (2) Support capacity building of professional bodies at the organizational level. For the MOHS: (1) Consider an appropriate approach to overcome the challenges of finding available and qualified post-training supervisors for F-IMNCI by reviewing the existing selection criteria. (2) Develop an action plan and strategy to reach the township level and below and make sure to involve staff from clinical sites (both hospital management and clinical staff) in all steps in programming. (3) Develop a knowledge￾sharing model between newly assigned staff and existing or transferred trained staff. Identify ways to pass knowledge gained and to develop clear job descriptions for knowledge handover. (4) Forecast HR management (transfer, replacement) in line with future program yearly plans and develop a plan to address challenges arising from staff turnover. 1 Box 1. MCSP Program Description Goal: To respond to MOHS strategic priorities for improving MNCH by demonstrating, documenting, and transitioning capacity to counterparts to make sustainable improvements in the health system via the MCSP program Aim: To build the capacity of midwives in Myanmar by introducing and supporting high-impact, sustainable MNCH interventions. Period of Performance: July 1, 2015, to June 30, 2018 Type of Assistance: Cooperative Agreement No. AID-OAA-A-14-00028 IP: Jhpiego, lead agency and technical lead for maternal health; Save the Children (SC), technical lead for newborn health; John Snow International (JSI), technical lead for children 2–59 months; Broad Branch (supporting National Health Plan Implementing Monitoring Unit) Geographic Presence: Out of 17 states/regions (S/R), seven with high MNCH rates: Mandalay Region, Rakhine State, Magway Region, Ayeyarwady Region, Shan State, Yangon Region, and Kayin State (total population 28,049,90) Key Partners: MOHS’ Maternal and Reproductive Health (MRH) Division, Child Health Development Division, Department of Public Health (DPH) Department of Medical Services (DMS), NHP Implementation and Monitoring Unit (NHP-MU), and state/regional DPH; professional bodies (Myanmar Nurse and Midwife Council [MNMC] and Myanmar Nurse and Midwife Association [MNMA]); the 3MDG fund; and American College of Nurse-Midwives (ACNM) Total Funding Amount: USD $8.1 million Introduction BACKGROUND AND CONTEXT Myanmar ended nearly 50 years of military rule in 2011 and inaugurated a democratically elected government in 2016. Since the official reopening of the U.S. Agency for International Development (USAID) Mission in 2012, the U.S. government has supported the country’s transition to democratic governance, national reconciliation, economic integration, and healthy and resilient communities. USAID prioritizes the health sector as a key arena for enhancing stability and resilience in Myanmar, as its population continues to face some of the highest mortality and morbidity rates in the region, including among mothers and children under 5. Improving maternal, neonatal, and child health – particularly among underserved populations – is a key national priority supported by USAID health assistance. The Maternal and Child Survival Program (MCSP) is a global cooperative agreement supported by USAID to introduce and support high-impact health interventions in 25 priority countries in support of the global goal of reducing preventable maternal and child deaths. In Myanmar, MCSP began implementation in July 2015. USAID’s initial vision for MCSP in Myanmar was to support discrete activities related to strengthening of in￾service training in midwifery, complementing work funded by the multi-donor Three Millennium Development Goal (3MDG) Trust Fund. With initial funding from USAID1, MCSP focused on supporting discrete in-service training and continuing professional development of frontline health workers (midwives), partnering with professional associations, and complementing 3MDG support for pre-service training and accreditation of training institutions. In mid-2016, USAID provided additional funding to expand MCSP programming to support discrete activities related to malaria in pregnancy and additional health systems strengthening and health workforce development priorities of the Ministry of Health and Sports (MOHS). Following a consultative process to identify potential priority activities with MCSP counterparts at the national level, MCSP developed a combined work plan in early 2017 to fully integrate interventions across malaria 1 MCSP. 2018. Summary of the evolution of USAID’s Maternal and Child Survival Program changes over time, inception to date. 2 Box 2. MCSP Targeted Intermediate Results (IR), Key Interventions and Results IR1: Policy environment strengthened for improving quality and equitable access to MNCH services Key Interventions • Provide technical support to MOHS for the implementation of the National Health Plan (NHP). • Build the capacity of the Myanmar Nurse and Midwife Council (MNMC) to administer competency-based assessment and licensure for midwives. • With MOHS, develop national antenatal care (ANC) and malaria in pregnancy (MIP) guidelines. • Update the national IMNCI guidelines (newborn and child) and collaborate with stakeholders for their endorsement IR2: Health workforce strengthened to support effective delivery of MNCH components of the EPHS Key Interventions • Support MOHS in developing and demonstrating an effective, standardized approach to in-service capacity building and continuing professional development in selected states and regions • Provide support to the MOHS Child Health Development (CHD) Division for updating and rolling out updated IMNCI and F-IMNCI training as part of national efforts to strengthen newborn and child health services • Provide support to EHOs and community-based organizations (CBOs) on the Myanmar-Thailand border region to improve and standardize skills of EHO providers • Strengthen the technical capacity and teaching approach of the faculty of Myanmar’s two nursing universities • Build the capacity of the Myanmar Nurse and Midwife Association (MNMA) to provide continuing professional development to members IR3: Quality health service delivery strengthened in targeted technical and geographical areas Key Interventions • Introduce a standards-based approach for improving the quality of care in the clinical sites affiliated with L&PICs • Implement and document the feasibility and effectiveness of Kangaroo Mother Care (KMC) for small, preterm, and low-birth￾weight) babies at two facilities efforts, in-service training, and health systems under a common theory of change, and an overarching focus on strengthening the health workforce. The total commitment of USAID funding to MCSP over the three-year period, from July 2015 to June 2018, was $8.1 million. The objective of MCSP in Myanmar was to build the capacity of midwives by introducing and supporting high-impact, sustainable maternal, neonatal, and child health (MNCH) interventions, and to support the MOHS’ strategic priority of strengthening human resources for health (HRH) by building the capacity of existing health staff to deliver lifesaving MNCH interventions. The focus on strengthening HRH and midwifery was selected given the importance of and poor access to midwifery services in rural and peripheral areas. Although the MCSP-led in￾service capacity-building training on MNCH has concluded, demands remain for technical support in interactive training for basic health staff (BHS) and in updating some MNCH curriculum and guidelines. MCSP aligned its activities with the national priority of improving health worker capacity to deliver high￾quality, lifesaving care under Myanmar’s Essential Package of Health Services (EPHS), which is currently under development by the MOHS, and thereby contribute to improved health outcomes. This report presents evaluation findings on USAID-supported activities on the technical capacity building of midwifery. The evaluation is part of a larger USAID Capacity Assessment in health sector-related projects in Myanmar being conducted by Social Impact, Inc. (SI). 3 EVALUATION PURPOSE AND QUESTIONS EVALUATION PURPOSE The purpose of this performance evaluation (PE) is to examine the extent to which MCSP’s interventions supported Myanmar’s capacity and systems for in-service training of health workers as a means to improve the availability and quality of maternal and newborn care services. The evaluation analyzed the effectiveness of in-service capacity building supported by MCSP, including the Learning and Performance Improvement Center (L&PIC) model, the rollout approach for competency-based capacity building at the state/regional level and below, the standards-based quality improvement model introduced at select training sites, and complementary efforts to strengthen institutions such as the Myanmar Nurse and Midwifery Council (MNMC) and Myanmar Nurse and Midwives Association (MNMA). Evaluation findings will be used to inform approaches for strengthening in-service training at lower levels of the health system under USAID’s follow-on Essential Health Program. They will also be considered when proposing recommendations for USAID or other development partners on how to optimize support to the MOHS to deliver integrated in-service training interventions and build relevant country systems. EVALUATION QUESTIONS There were three main evaluation questions (EQs) outlined in the Scope of Work (see Annex A). EQ 1: To what extent did MCSP assistance influence in-service training practices and related systems to improve maternal, neonatal, and child health? a) To what extent have the MCSP’s in-service capacity-building activities, including the models outlined in Section B and associated interventions, influenced policies, practices, and the enabling environment for in-service training at different levels of the system (regulatory and professional bodies, central MOHS, state/regional level, and township level and below)? To what extent have health system actors been able to apply and replicate interventions introduced by MCSP? b) To what extent were MCSP’s approach and interventions aligned to health system realities to address the key barriers to strengthening in-service training at the state/region and township levels? c) To what extent were MCSP’s interventions and program design aligned to address drivers of maternal and child mortality and morbidity? EQ 2: How have MCSP’s approaches contributed to the potential sustainability of project results? a) Which activities or technical approaches could be beneficial to incorporate in future MOHS in￾service training models? b) What are the key factors/evidence that support such conclusions? EQ 3: What are the specific lessons that can be learned to inform future programs that aim to strengthen systems for capacity building related to MCH, particularly at the township level? a) Are there any similar approaches/interventions that should/should not be supported/replicated in future assistance? Why/why not? b) Are there any challenges in the health system that MCSP did not address that would need to be addressed for future programs to be successful, particularly in effecting improvements at the township level and below? c) Should any interventions/supports be removed or modified to better adjust interventions to health system realities? d) Are there any necessary modifications to the models and interventions supported by MCSP, including their mode of delivery, if future replication is considered? 4 METHODOLOGY DATA COLLECTION METHODS The SI team employed a mixed-method design, which includes document review, primary qualitative data collection, and review and analysis of available secondary quantitative data. Document Review A comprehensive document review provided background knowledge on existing national policies and guidelines, international standards and best practices, regional programming with similar scope, capacity-strengthening initiatives undertaken, and critical information on the status and outcomes of MCSP-related activities. A list of all documents consulted for this evaluation appears in Annex B. Although the evaluation team did not conduct a special desk-based gender assessment, gender and social considerations underlined data collection, analysis, and report preparation as described throughout this report. Key Informant Interviews (KIIs) KIIs explored the progress of MCSP activities across the seven states/regions (S/R) and 14 townships. Respondents were the primary counterparts and beneficiaries of MCSP activities such as S/R health departments and training teams and selected secondary beneficiaries such as L&PIC clinical site staff from the Women and Children Hospital, the Taunggyi and Thanlyin District Hospital, and the staff who were part of the S/R training teams that MCSP revitalized during implementation. The evaluators asked respondents about changes since MCSP’s inception, including perceptions on increased capacity in providing quality service, staff skill development, and quality assurance (KII tools can be found in Annex C). Focus Group Discussions (FGDs) FGDs were held with MCSP beneficiaries such as executive members of MNMC, members of MNMA, S/R health training teams from the MOHS, and MOHS clinicians and nurses from DMS who participated in the trainings. The FGDs gathered information about MCSP training and capacity-building models. FGDs sought to understand how MCSP supported staff capacity to deliver MNCH practices, how MCSP aligned with the current health system, and how MCSP results contributed to overall capacity development efforts in addressing maternal and child mortality, as well as how stakeholders perceived the efficacy of each intervention. The team explored all stakeholders’ perceptions on the sustainability outcomes of MCSP and lessons learned to inform the upcoming USAID-funded activity, particularly at the township level. The team also conducted two group discussions with beneficiary mothers from kangaroo mother care (KMC) units in Thanlyin General Hospital and Taunggyi Women’s Hospital. Mothers were asked about the KMC service they received from the hospital, status improvements of low-birth-weight and preterm newborns, any challenges they encountered, their trust and acceptance of the KMC service, and how their husbands and family members responded to it. Figure 1. Map showing the geographic coverage of MCSP project sites 5 Direct Observation Although direct observation was not an explicit feature of the original evaluation design, the evaluation team observed three L&PIC sites in Yangon (MNMA, MNMC) and Taunggyi. Although the team did not use an observation checklist, they assessed how these L&PIC sites functioned and how they used models. SAMPLING Data Collection Sites The evaluation team selected five of the seven MCSP implementation S/Rs for primary data gathering: Yangon/Thanlyin, Nay Pyi Taw (NPT), Shan, Kayin, and Ayeyarwady. Evaluators purposely selected these locations based on stakeholder types (central-level MOHS, S/R training teams, ethnic health organizations [EHOs], professional bodies), the presence of L&PIC and KMC sites, and geographical coverage. The team conducted two KIIs remotely because one respondent was overseas during the data collection period and one was in a remote location. The team conducted a total of 27 KIIs and seven FGDs, plus two group discussions with KMC mother groups. All respondents were purposely selected. Table 1 presents target and actual sample sizes. In total, 64 respondents participated in the evaluation. TABLE 1. SAMPLE SIZES, ACCORDING TO LOCATION, RESPONDENT TYPE, AND DATA COLLECTION METHOD FOR EACH ACTIVITY MATERNAL AND CHILD SURVIVAL PROGRAM LOCATION RESPONDENT TYPE DATA COLLECTION METHOD KIIs FGDs TARGET ACTUAL TARGET ACTUAL Yangon IPs 12–14 6 1–2 0 Professional Body (MNMA, MNMC) 2 2 Donors 3 0 Thanlyin Thanlyin Hospital 1 1 NPT Central MOHS 5 3 0 0 Shan Shan State DPH 7–9 6 2–3 3 Women and Child Hospital, Taunggyi 2 1 Ayeyarwady Ayeyarwady DPH (Remote Interview) 1–2 1 2 0 Kayin KDHW (EHO) (Meeting at Yangon) 1–2 3 1–2 0 ALL LOCATIONS/ RESPONDENT CATEGORIES FOR THE MCSP 26-32 27 6-9 7 Additional mother groups for kangaroo mother care at Taunggyi and Thanlyin Hospital 0 2 Among 27 respondents representing MNMA, MNMC, MOHS, IPs, and donors, 63 percent (n=17) were female and 37 percent (n=10) were male. 30 respondents representing MNMC, MNMA, and state and regional training teams participated in seven FGDs. All were female. ETHICAL REVIEW The evaluation was conducted in accordance with USAID’s Ethics Standards in Research Policies and the ethical guidelines and processes of the DPH’s Ethics Review Committee (ERC). The deputy director general (DDG) and senior staff of DPH supported the process. 6 FIELD WORK A two-person team, including the team leader and a research specialist, conducted KIIs and FGDs in the field. The M&E specialist from USAID’s Program Development Office attended and actively participated in three KIIs conducted with senior MOHS key informants in NPT. Two of the 27 KIIs were conducted remotely, and 2 KIIs were conducted in English. The remaining KIIs, FGDs, and discussions were conducted in the Burmese language. The team leader primarily facilitated all interviews. DATA ANALYSIS In answering the EQs, the evaluation team triangulated evidence across stakeholders and data sources. The evaluation team disaggregated the data by activity and used content and comparative analysis, coding interview notes from the KIIs and FGDs to identify recurring themes and key factors for each EQ. The team analyzed the qualitative data in tandem with any available quantitative data (e.g., in quarterly and annual progress reports, work plans, key implementation documents, performance monitoring plans, relevant documents from the MOHS, and other records and information shared by the IPs). LIMITATIONS Busy schedules of senior MOHS staff, staff turnover and transitions by S/R training team members for postgraduate study, and outdated contact information for some S/R training team members made it more difficult to gain access to originally targeted respondents and capture the perspectives of some respondent types. Due to the tight schedule, the evaluation team chose to travel mainly to Shan and Kayin states. Another limitation might be the small number of respondents from some regions (3 respondents from Kayin state and 1 respondent from Ayeyarwady region); however, it did not affect the planned and targeted respondents. The evaluation targeted 26–32 respondents for KIIs and 6–9 respondents for FGDs, and conducted 27 KIIs and 7 FGDs. This fell within the targeted range, albeit on the lower end. It was difficult to recruit BHS staff from the rural health centers, as some were stationed at sub-rural health centers about 100 kilometers away from Taunggyi. Surveying at the state and regional level was limited. Mitigation Strategy: The evaluation team needed several follow-ups to get interviews with senior staff. For S/R training team members, the team identified potential respondents with inputs from the director of the Maternal and Reproductive Health Division (MRH), the deputy director, and senior staff of S/R DPH. The team successfully tracked down several originally targeted respondents. If the planned respondents were not available, the team identified new potential respondents who were also training team members with input from senior central and S/R health staff. The team used “snowball” referrals in identifying S/R training team members. After phone discussions with the deputy director of Shan state and checking with the USAID contracting office, SI supported travel costs for BHS staff from rural health centers to participate in FGDs. Some FGDs comprised three or four respondents owing to S/R training team composition. Mitigation strategy: This did not affect the quality and nature of FGDs, as those who participated were key persons involved in the MCSP process and representative of their departments. Participants provided wide-ranging information to help the evaluators better understand the experiences of each respondent group. Unexpected flooding delayed data collection with KDHW, an EHO in Kayin state. Mitigation strategy: The team monitored the situation for one week, and ultimately elected to interview KDHW (EHO) staff when they came to Yangon. 7 There was a possibility of recall bias, reducing accuracy and completeness of recollections, especially when respondents participated in training activities long before the evaluation, and response bias, whereby respondents skewed toward what they expect the interviewer to favor, partly in hope of continued donor support. Mitigation strategy: SI’s data collection instruments specified the time period being referenced in order to aid respondents with recall. The evaluation team triangulated KII and FGD data across multiple types of respondents and with secondary data and information from the document review. The evaluation team explained the nature of the study, which included no inherent benefits or risks to participants and ensured confidentiality. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS EQ 1: TO WHAT EXTENT DID MCSP ASSISTANCE INFLUENCE CHANGES IN IN￾SERVICE TRAINING PRACTICES AND RELATED SYSTEMS TO IMPROVE MATERNAL, NEONATAL, AND CHILD HEALTH? EQ 1a. To what extent have the MCSP’s in-service capacity-building activities, including the models outlined in Section B and associated interventions, supported or enabled changes in policies, practices, and the enabling environment for in-service training at different levels of the system (regulatory and professional bodies, central MOHS, state/regional level, and township level and below)? To what extent have health system actors been able to apply and replicate interventions introduced by MCSP? FINDINGS There is evidence that MCSP supported in-service training practices and related systems in several respects. KII and FGD respondents acknowledged that MCSP has strengthened policies for delivering quality and equitable access to MNCH services (IR 1), enhanced the capacity of the health workforce to support effective MNCH service (IR 2), and improved quality health service delivery (IR 3). More detail is provided below. 1. MCSP support for the policy environment strengthened the quality of service delivery and ensured more equitable access to MNCH services Among respondents, 63 percent of KII participants (17 out of 27) and 90 percent of FGD participants (28 out of 31) were familiar with MCSP activities. They were able to describe the value of these activities and the extent of MCSP’s support for local practices and systems. Technical support for the National Health Plan (NHP): MCSP contributed to the formulation of the NHP operational plan and supported the development of the NHP by providing technical assistance. Among respondents, eight KII participants representing donors, central-level MOHS staff, and staff from the IPs were familiar with this work. 75 percent of all respondents felt that the activities were effective. MCSP provided technical support to NHP implementation monitoring unit activities (NIMU) by providing basic costing to the Essential Package of Health Services (EPHS), the formulation of the NHP operational plan, NHP budgeting, terms of reference for the Figure 2. Standard Operating Procedures for Small Babies (KMC) (source: MCSP. Preventing the Needless Deaths of Women and Their Families. 2018. PowerPoint presentation) 8 development of the M&E framework for the NHP, the preparation of S/R and township health working groups, and a template for inclusive township health plans and the NHP NIMU communication strategy2.These products are being rolled out nationwide to guide the NHP. Around 50 percent of KII respondents who were familiar with the NHP noted that MCSP’s role in the development of these products was instrumental. Support for the MOHS to develop national antenatal care guidelines: MCSP supported the MOHS Maternal and Reproductive Health (MRH) division to develop the first national antenatal care (ANC) guidelines (in English and Burmese) and successfully facilitated a consensus-building meeting to finalize the guidelines. About 60 percent (16 out of 27 KIIs) of respondents representing various stakeholders such as donors, central and S/R MOHS staff, MNMA, MNMC, and IPs were familiar with MCSP support for MNCH policy and guideline development. Respondents shared that the ANC guidelines helped them standardize maternal care guidelines so that they could apply standard operating procedures (SOPs) and ensure quality of care. About 80 percent of respondents from S/R training teams reported that there was a high level of acceptance of the ANC guidelines and that they used the techniques in the guidelines to prevent and manage post-partum hemorrhage (PPH), monitor labor progress using partograms, and practice delayed cord clamping. Support the MOHS in updating newborn and child health-related policies, guidelines, and SOPs: Nine of 27 KII respondents from USAID, IPs, and the MOHS at central and S/R levels were familiar with MCSP’s support for neonatal and child health care guidelines. They described how MCSP supported updates to newborn and child health-related policies, guidelines, and SOPs. Staff from the MOHS, Save the Children International (SCI), and John Snow International (JSI) (7 interviewees) reported how MCSP activities assisted MOHS staff, pediatricians, and neonatologists to review and revise the Integrated Management of Neonatal and Childhood Illness (IMNCI) manuals. The 2017 annual summary report from MCSP reported that newborn guidelines allowed basic health staff (BHS) to perform essential practices that were previously disallowed in nursing schools, such as nasogastric tube insertions3 Also, MCSP successfully supported the implementation of kangaroo mother care (KMC) for small (preterm and low-birth-weight) babies delivered at Taunggyi Women and Children Hospital and Thanlyin General Hospital, according to interviews with neonatologists and pediatricians from these two facilities. From July 2017 to March 2018, staff at the Thanlyin hospital applied KMC to approximately 70 percent eligible newborns (201 out of 287). In 2017, staff at the Taunggyi hospital applied KMC to around 64 percent of eligible newborns (135 out of 212)4. Additionally, MCSP supported an SOP for KMC endorsed by the MOHS for broader utilization. These changes suggest strong buy-in from these institutions and signal not only sustained programming for MCSP’s work toward preterm and LBW infant care, but also scaling beyond the project scope. Hence, MCSP supported the MOHS’ newborn care-related practices and systems by improving the management of low-birth-weight babies and improving preterm health outcomes. Building MNMC’s capacity for competency-based assessment and licensure for midwives: According to MNMC staff, MCSP strengthened MNMC’s functions, provided technical and infrastructure support for relicensing of nurses and midwives, and supported continuous nursing education (CNE) accreditation, which contributed to midwives’ and nurses’ professional development. MCSP supported L&PIC at MNMC, which MNMC refers to as the Clinical Skills Standardization and Assessment Center (CSSAC) and still functioning and being used. According to the interview and group discussion with MNMC’s president and executive committee members, the center now serves as part of the official 2 MCSP. Myanmar Annual Summary Program Year 3, October 2016–September 2017. p. 5. 3 MCSP. Myanmar Annual Summary Program Year 3, October 2016–September 2017. p. 6. 4 MCSP.2018. Strengthening Small Baby Care at Thanlyin General Hospital through Kangaroo Mother Care.p.9. and MCSP. 2018. The Feasibility and Acceptability of Kangaroo Mother Care in Neonatal Unit, Women’s and Children’s Hospital, Taunggyi, Myanmar, Preliminary Results, Report Brief and Presentation. Slide no. 24. 9 exam center for nurse and midwife relicensure. To date, 31 percent of nurses (721 out of 2326) and 54 percent (131 out of 244) of midwives whose licenses have lapsed have been relicensed after passing the relicensure exam conducted at MNMC’s L&PIC site (as of FY 2018, Q3).5 By supporting licensure and registration of nurses and midwives through the L&PIC, MNMC members have increased their capacity to administer competency-based assessments and licensure for midwives. Management commitment and political will within the MOHS: The MOHS leadership showed commitment and political will to work with MCSP, particularly in strengthening the MNCH policy environment. Various stakeholders from KIIs and FGDs (23 of 27 respondents from donors, MOHS, IPs, and professional bodies (MNMA, MNMC) plus 5 FGDs with S/R MOHS staff) supported the finding that commitment and buy-in from the central government was strong. During the interview, senior staff from NPT, S/R training team members, and IP staff cited the minister’s personal commitment and shared about the minister traveling to Taunggyi to open the L&PIC site and to NPT for a broader launch of the L&PIC model. The permanent secretary delivered an opening speech at the launch ceremony for a few initiatives, including Early Essential Newborn Care (EENC), the care for small babies initiative, and the rollout of the updated Integrated Management of Neonatal and Childhood Illness guidelines. In addition, MCSP models were reflected in the NHP. For example, the MOHS recognized the MCSP cascade training model, which was a multiplier training model carried out at the central and S/R levels and demonstrated at MCSP-focused townships. MCSP supported the development and updating of various guidelines and SOPs in line with international standards and best practices. MCSP’s success was also demonstrated by the fact that senior MOHS staff from NPT cited the receptivity of NIMU and other entities to a range of MCSP technical support inputs and the possibility of replicating the L&PIC model in other S/Rs across Myanmar. 2. MCSP support for improved provider practice There were several indications that MCSP supported changes in MNCH health care delivery practices in the field and improved health care staff communication skills and male involvement. Changes in maternal health-related practice: MCSP supported changes to state-of-the-art maternal health-related clinical practices. Training team members and master mentors (staff from NPT) reported key changes, including management of pregnancy-induced hypertension (pre-eclampsia/eclampsia); use of partograms to monitor the progress of labor and identify high-risk cases for early referral; active management of the third stage of labor, including application of controlled cord traction, aortic compression, or bimanual compression; use of prophylactic uterotonic (intra-muscular oxytocin) to prevent and manage post-partum hemorrhage (PPH); and birth positioning in compliance with respectful maternity care. Adopting these updated maternal care practices following MCSP training allowed midwives and lady health visitors to control PPH and refer high-risk mothers to a health facility. However, not all respondents were completely comfortable with these new skills. Among respondents (from S/R teams) who reported practicing aortic or bimanual compression to control PPH, six of 16 5 MCSP. Myanmar Quarterly Progress Report. Program Year 4, Quarter 3: April-June 2018. Figure 3. Minister opening the L&PIC at Women and Children Hospital, Taunggyi. (source: MCSP. Preventing the Needless Deaths of Women and Their Families. 2018. PowerPoint presentation) 10 Box 3. Success story based on IMNCI training “Receiving the IMNCI updated guideline training has helped to save lives of people living in rural areas. The baby was lucky as I had received the training recently and I could apply the knowledge received directly from the IMNCI training. I will continue executing the IMNCI approach in future.” –Daw Win Nilar, Midwife of Han Ngaing Rural Health Center (Source: Final Report of IMNCI Training Courses in Five Selected States and Regions 2016– 2018, p. 69) respondents remained unconfident in using a condom as a tamponade for intrauterine pressure to stop massive PPH.6 FGDs with midwives and lady health visitors revealed challenges in using magnesium sulfate to manage hypertension in pregnancy. According to the FGDs with MNMA staff and S/R training team members (6 KIIs and 4 FGDs), the occurrence of vaginal tears, PPH, and neonatal asphyxia has reduced over past one to two years. Respondents assumed that the cause was midwives, including themselves, now following the SOPs and guidelines learned from MCSP, though there is no clinical or documented project data to corroborate this finding. Changes in newborn and child health care practices: MCSP also supported the MOHS in changing practices in newborn and child health care to align with national standards, such as cord care, thermoregulation via skin-to-skin care, and kangaroo mother care (KMC) for management of preterm/LBW babies, and provided support to update the IMNCI guidelines per the request from the Child Health Division of the MOHS. MCSP enabled S/R team members (reported from 10 KIIs and 5 FGDs) to be stationed at township MNCH centers and rural health centers to learn about the “golden minute,” a critical time to prevent avoidable neonatal death. According to these respondents, they are applying delayed cord clamping to improve neonates’ blood supply from the cord, placing neonates on the mother’s chest for skin-to-skin care, and applying suction to manage birth asphyxia. Similarly, all respondents gave high scores to penguin suction7. (Note: penguin newborn suction is a reusable suction bulb intended for removal of fluids from a newborn’s nasal and oral cavities and easy to use, clean, and carry.) Respondents shared how they managed birth asphyxia cases via suction in their catchment areas. Respondents familiar with MCSP’s support for KMC gave positive feedback on the skin-to-skin method, saying it was easy to apply without special skills and could increase mother-child bonding. After visits to KMC sites in Taunggyi and Thanlyin, the team found that MCSP successfully introduced the KMC model for preterm and low-birth-weight neonates. All interviewed health workers in charge of the KMC units in Taunggyi and Thanlyin hospitals believed in the efficacy of the approach and were enthusiastic to practice KMC. They did note that there were still unfulfilled needs for more health staff in their units, more space, and pronounced commitment by the leadership. As reported in the section above, the desk review revealed that in the Thanlyin hospital, 70 percent of eligible newborns were introduced to KMC from July 2017 to March 2018,8 while in the Taunggyi Women’s Hospital, 198 eligible newborns (32 percent of low-birth-weight and 64 percent of newborn babies less than 2,000 grams) were admitted to the KMC unit in 2017.9 6 For more information on this technique, see Hasabe, R., Gupta, K., & Rathode, P. (2016). Use of Condom Tamponade to Manage Massive Obstetric Hemorrhage at a Tertiary Center in Rajasthan. Journal of Obstetrics and Gynaecology of India, 66(Suppl 1), 88–93. 7 For more information on penguin suction, see Smith, J. M., de Graft-Johnson, J., Zyaee, P., Ricca, J., & Fullerton, J. (2015). Scaling up high-impact interventions: How is it done? International Journal of Gynecology & Obstetrics, 130, S4-S10. 8 MCSP. Myanmar Annual Summary. Program Year 3, October 2016–September 2017. p. 8. MCSP.2018. Strengthening Small Baby Care at Thanlyin General Hospital through Kangaroo Mother Care. p.9. 9 Hnin, Thein Thein. Quality Improvement of Newborn Care at Women’s and Children’s Hospital Taunggyi. July 2018. PowerPoint presentation. Slide no 54. 11 The team conducted brief discussions with mothers who benefited from KMC services in hospitals in both Thanlyin and Taunggyi. The interviewed mothers noted that wraps, blankets, and portable privacy screens for immediate skin-to-skin contact after birth were critical in helping them implement KMC practice.10 They would recommend this practice to others as it increases the bodyweight of the baby according to daily weight measurements conducted at the hospital. A study on KMC conducted by MCSP shared a similar finding: 100 percent of respondents (n=103) recommended KMC services to others, while 91 percent of mothers and 86 percent of fathers were satisfied with their baby’s progress as a result of practicing KMC.11 Respondents did report that there was an opportunity cost for adopting and adhering to optimal KMC practices during long hospital stays for the mother and baby. Long stays result in a reduction in family income as mothers and family members were not able to work and family members incurred additional costs in accompanying them during the stay. MCSP supported changes in infant and child health practices at the central/institutional level as well as the service-delivery level. MCSP supported the child health division of the MOHS in updating IMNCI and rolling out the IMNCI and F-IMNCI training in selected townships. MCSP also supported post-training supervision as part of a national effort to strengthen newborn and child health services. According to the Final Report of IMNCI Training produced by MCSP’s child health team in June 2018, as part of IMNCI training, MCSP introduced continuous nursing education modules on pneumonia and diarrhea based on updated IMNCI guidelines,12 while adherence to these practices was noted during FGDs and KIIs with S/R teams. A midwife shared in her interview that her capacity to check, identify, and refer children with diarrhea or pneumonia cases had improved. Prevention of avoidable child deaths as a result of changes in practice was demonstrated by a success story published in the program report: an IMNCI-trained midwife saved a 2-year-old girl from Mong Kai township, Southern Shan state (a remote area), via timely diagnosis of danger signs.13 Factors favoring changes in MNCH practice: MCSP provided international best practice models for safe delivery. In addition to improving knowledge and skills through training, MCSP empowered health staff to use their newly acquired skills—a critical aspect of any behavior change intervention. According to respondents (KIIs with staff from NPT-MOHS and IPs), midwives had an opportunity to acquire, learn, and practice updated knowledge and skills through the standardized training modules and interactive teaching modality. In addition, MNMA staff who attended the MCSP training reported that they had the confidence to deliver quality care and act as trainers. Staff equipped with updated MNCH management knowledge through the MCSP training reported they were able to identify high-risk patients and were capable of monitoring and supervising at the township level. The learning aid model used in L&PIC increased staff confidence in dealing with real patients, as they had hands-on practice with simulations of child birth, PPH, and newborn care (as shared by S/R training team members: 5 out of 10 KIIs and 12 out of 16 in 4 FGDs). Respondents claimed that fresh graduates had an opportunity to practice with dummies and learning aids prior to frontline service. Secondary data from Jhpiego supported and validated this finding. MCSP improved staff communication skills: According to interviews with MNMA members and S/R trainings teams, MCSP training improved basic health staff’s facilitation and communication skills. Respondents had positive feedback for the interactive teaching methods, as most of them had only 10 Source: Group discussions with mothers hospitalized at KMC units and Jhpiego. 2017. Myanmar Annual Summary. Program Year 3, October 2016–September 2017. p. 9. 11 The feasibility and acceptability of Kangaroo Mother Care in Neonatal Unit, Women and Children Hospital, Taunggyi, Myanmar, June 2018. 12 MCSP Child Health Team. June 2018. Final Report of IMNCI Training Courses in Five Selected States and Regions 2016- 2018. June 2018. p 24. 13 MCSP Child Health Team. June 2018. Final Report of IMNCI Training Courses in Five Selected States and Regions 2016– 2018. Annex 13, p. 73. 12 received classroom-based, lecture-oriented training previously. They appreciated the interactive and skills-based training, as well as the module on respectful mother care. Respondents (16 respondents from MNMA and S/R training teams) reported that they have learned to respect mothers and changed their attitudes in dealing with patients. In addition, midwives and lady health visitors stated that they have gained trust from mothers and families via effective communication with patients. Health staff learned to prioritize mothers’ wishes, and mothers were now free to deliver in the position of their preference (including standing or sitting), which contributed to easier deliveries; previously, mothers were only permitted to deliver in the lithotomy position. 3. MCSP supported the enabling environment As reported below, MCSP has supported the enabling environment for in-service capacity building of health staff responsible for MNCH health status improvements. Updated policies and guidelines finalized: As stated above, MCSP supported an enabling environment for in-service training through their support in reviewing, revising, and developing updated MNCH guidelines such as the ANC guideline (3 KIIs from the MOHS, 3 KIIs from IPs). MCSP provided technical assistance to the government and other stakeholders such as WHO and UNICEF, among many others.14 Supported standardized approach to in-service capacity building: MCSP supported the establishment of a learning hub at the S/R level as a repository for capacity-building materials and learning opportunities. It also revitalized S/R health training teams so that they were able to deliver competency-based in-service training to township training teams. Such support for capacity building was widely reported in by KIIs and FGDs with donors, MOHS staff, S/R training teams, MNMC, and MNMA. Cascade and modularized trainings: S/R staff were empowered by the strengthening of master mentors in NPT (central level) and training teams. The program targeting central and S/R levels was in line with project design. At the township level and below, MCSP demonstrated the standardized and modularized MNCH training to S/R health training teams and provided guidance on bringing training offerings to the township level. The rollout of training and the L&PIC approach in townships for midwives and lady health visitors occurred during the last quarter of the program.15 Similarly, midwives and lady health visitors also received child health training at the township level. A total of 21 lady health visitors and 231 midwives received the IMNCI training16. The use of interactive sessions, energizers, and improved facilitation skills were positive attributes of the training for trainees (9 out of 10 KIIs, 1 FGD with MNMA, and 4 out of 5 FGDs with S/R training teams). As the training modules were modularized into different components17, midwives had an opportunity to 14 MCSP. 2017. MCSP Myanmar. Implementation Plan Program Years 3-4. March 2017–June 2018. 15 MCSP. Myanmar Quarterly Progress Report. Program Year 4, Quarter1: October–December 2017. 16 MCSP Child Health Team.2018. Final Report of IMNCI Training Courses in Five Selected States and Regions, 2016-2018. P27 17 Modules to date include: Respectful Maternity Care (RMC), Clinical Decision Making (CDM), Use of the Partograph, Normal labor and immediate newborn care (NL), Newborn Resuscitation (NBR)/HBB, Management of Severe Pre-eclampsia (PE) Figure 4. Effectiveness of the in-service capacity-building training at L&PIC shown by master mentors’ skills and knowledge retention 13 learn the updated MNCH techniques in a systematic way and had the flexibility to choose topics. Hence, midwives were able to apply what they learned easily in a clinical setting. MCSP also trained MOHS staff on using modularized training materials, which allows them to replicate MCSP’s approach in the future. MCSP post-training supervision and follow-up were critical for ensuring quality of care. Post-training follow-up and supervision activities were implemented in line with existing MOHS supervision plans. The application of training skills could be monitored and assessed several months after the training through quality improvement (QI) performance scores assessed during supervision visits. Six months after training, 80 percent of trained master mentors from S/R training teams scored on average 80 percent on knowledge retention assessments in the areas of normal labor, PPH, newborn resuscitation, and knowledge assessment (Figure 5). A comparison of knowledge levels of IMNCI updated guidelines before the test training and post-test training demonstrated that central-level TOT training has increased post-test scores from 5.15 to 5.53. State- and township-level TOT training has increased scores from 3.74 to 7.19, while township-level basic health staff training has increased from 4.1 to 7.6718 (Figure 6). MCSP-initiated quality improvement activities: MCSP supported QI assessments and documented meaningful QI achievements. The desk review19 showed that in Program Year 3, MCSP introduced a modified version of the QI approach to hospital staff at five facilities affiliated with the L&PICs. This approach, which was first introduced in Myanmar through a Jhpiego￾led, General Electric Foundation-funded project, included provision of technical updates, QI committee formation, and baseline QI assessment for infection prevention and normal labor. The program conducted three QI assessments (baseline, midline, and endline) at these facilities to examine the performance scores of health staff who completed the MCSP training and used checklists to assess practices in normal labor and infection prevention. The average performance score at five L&PIC affiliated clinical sites on QI verification criteria /Eclampsia and Management of Post-partum Hemorrhage (PPH)]. (Source: MCSP.2018. MCSP Myanmar Workplan, March 17- June 18. 18 MCSP. Preventing the Needless Deaths of Women and Their Families. 2018. PowerPoint presentation. 19 MCSP. Myanmar Annual Summary. Program Year 3, October 2016–September 2017. p. 8. 45 80 0 10 20 30 40 50 60 70 80 90 100 Baseline Endline Percent Scores Figure 5. Average quality improvement performance score of five L&PIC affiliated clinical sites (Sittwe, Taunggyi, Lashio, Magway, Pathein) Figure 6. IMNCI updated guideline pre and post-test comparison 14 increased from 45 percent at baseline to more than 80 percent at endline when measuring performance scores against standard QI verification criteria.20 The MCSP Annual Reporting for Year 3 reported that L&PIC sites achieved 100 percent QI verification at endline. Effective application of the L&PIC model at 10 sites: Interviews and FGDs indicated that L&PICs were integrated into local institutions in a number of ways that bode well for their sustainability, though some sites indicated that maintenance and resources for long-term maintenance were potential challenges. S/R training teams and EHOs used ten L&PIC sites established in seven S/Rs as education and resource centers. Midwives and lady health visitors reported that they had benefited from the practical training, especially learning aids and models, offered by these centers. L&PIC sites served as skill laboratories, classrooms for clinical assessment training, examination centers, and learning hubs for rolling out modular competency-based courses for improving MNCH skills and knowledge. A desk review of a report for January-March 2018 supported the qualitative findings by stating that 10 L&PIC sites, established for both education and training of midwives with state-of-the-art clinical skills standardization and assessment centers, have been effectively utilized in Yangon and five S/R, MNMC, MNMA, and EHO areas21. The number of trainees using L&PIC through MCSP and non-MCSP funds were reported to be 1,422 for the MCSP project duration, in which a total of 937 health staff were trained with MCSP funding and 485 were trained by non-MCSP funding, indicating the effective usage of L&PIC sites by local stakeholders27. During an FGD, MNMC executive committee members mentioned space constraints in the MNMC L&PICs as an implementation challenge. The limited space made it difficult to arrange full exam tables as suggested by MCSP, while the evaluation team acknowledged that space constraints at MNMC were out of MCSP’s scope. Furthermore, interviews with a person responsible for the L&PIC in Taunggyi and FGDs with MNMA and MNMC revealed the possibility of maintenance challenges in the future and resource constraints for long-term functioning. Development of professional bodies: MCSP supported two midwifery institutions: MNMC and MNMA. MNMC has a regulatory function (licensing, registration, and accreditation) while MNMA is a professional association whose primary mandate is to support its members and provide continuing professional education. At both MNMC and MNMA, MCSP established L&PICs and provided MNMA training that enabled team members to learn, apply, and practice standardized skills using hands-on practical models. As shared in the FGDs with MNMC and MNMA members, about 60 percent of participating respondents felt that the continuing nursing education approach contributed to increases in members’ capacity and provided a window of opportunity to orient midwives and lady health visitors in frontline service to new modules. For example, a CNE module session conducted with 49 midwives and lady health visitors in Mong Kai township, Southern Shan state, introduced a new IMNCI guideline.22 MCSP reported that the member association capacity assessment tool (MACAT) achievement increased from 65 percent at baseline to 92 percent at endline.23 MNMC was able to successfully use its L&PIC site as an exam location to relicense midwives who had been out of service. Likewise, during an FGD, MNMC participants expressed how the Clinical Skills Standardization and Assessment Center (CSSAC) (MNMC’s name for its L&PIC) at MNMC creates 20 MCSP. Myanmar Annual Summary. Program Year 3, October 2016–September 2017, p. 23; and a PowerPoint presentation on the Prevention the Needless Deaths of Women and Their Families (QI data were not collected for Sittwe for endline QI assessment). 21 MCSP. Quarterly Progress Report. Program Year 4, Quarter2: January-March 2018.p 37,47 22 MCSP. Final report of IMNCI training course in five selected states and regions 2016–2018, p. 13. 23 MCSP. Myanmar Quarterly Progress Report. PY 4, Quarter1: October–December 2017. Indicator 2.12, p. 48. Figure 7. Husbands practicing KMC at Taunggyi Women’s Hospital (Source: Photo taken by SI team during data collection) 15 opportunities for nurses and midwives to update their skills and knowledge. According to those respondents, MCSP enhanced the capacity of MNMC members and strengthened council functions by making the passage of a CSSAC practical test a requirement for the national relicensure exam. 4. Other Findings MCSP addressed gender equality and social inclusion: There were indications that MCSP-supported training positively addressed gender equality through emphasis on male involvement. Interviews and FGDs with mothers of preterm and LBW babies in Thanlyin and Taunggyi hospitals, KMC units, and staff in charge indicated that MCSP-supported KMC practice encouraged husbands to take ownership and help their wives in child rearing and care. Respondents shared that husbands encouraged, accepted, and even helped hold babies for hours to support their wives. Photos (Figure 10) were posted on the wall of the KMC unit of Taunggyi Hospital to serve as an indication of positive shifts in promoting male participation in MNCH care at delivery sites. Additional value provided by MCSP includes more male/husband involvement in child delivery, as husbands could be present at delivery sites and support their wives. Some FGD participants held the cultural belief that if a man stays in the delivery place or room, it could lower his willpower. However, with effective communication, husbands are now willing to stay during delivery and support their wives. In support of the MOHS, MCSP addressed both male and female involvement at S/R and township levels by choosing training participants. Due to the nature of MCSP, midwives—the primary beneficiaries— were female. But by including other basic health staff such as public health supervisors, male staff would have an opportunity to benefit from MCSP training. According to MCSP training reports, 8 percent (4) of all 50 trainees were male in 2016, while 11 percent (186) of the 1,749 attendees were male in 2017.24 Engagement with EHOs: Engaging EHOs in MCSP has strengthened MOHS’ engagement with EHOs, which increased mutual understanding and addressed social inclusion. All senior EHO staff participating in KIIs appreciated MCSP’s efforts in establishing an L&PIC site in an EHO area, setting up an EHO training team, and facilitating trainings delivered by senior MOHS staff from NPT to EHO staff. For the first time, the MOHS issued training certificates to EHO staff members after the completion of trainings, which showed positive prospects for future engagements and collaborations between the MOHS and EHOs. During the life of the project, the number of EHO providers certified by the MOHS increased from zero at baseline (2015) to 72 (2017).25 A senior staff member from the central MOHS reported that in the NHP, the government will engage/integrate with non-state actors to support the Universal Health Coverage (UHC) approach. In addition, lessons learned and achievements from MCSP will provide useful insight for future activities. 24 MCSP. 2017. Participant Training Report for 2016. Excel spreadsheet. 25 Performance Indicator Report, Myanmar Annual Summary Program Year 3, October 2016–September 2017, p. 22. Figure 8. BEmONC workshop for EHO health care providers at Myawaddy General Hospital (Source: MCSP. Preventing the Needless Deaths of Women and Their Families. 2018. PowerPoint presentation) 16 All six IPs, three senior-level MOHS staff, and two USAID staff perceived that MCSP provided a platform to support the MOHS in unprecedented engagement of EHOs in ceasefire areas. As a result, the MOHS has gained more experience in engaging/collaborating with EHO staff. USAID staff also recognized the greater coordination and collaboration between EHOs and the government, as well as the successful engagement of MCSP in EHO areas. CONCLUSIONS In general, the evaluation found qualitative evidence showing MCSP activities and approaches have successfully supported clinical practice as well as in-service training practices and related systems, particularly at the central and S/R levels. MCSP-supported existing cascade training approaches, such as the cascade training model, new QI approaches, and post-training follow-up and supervision activities, were implemented in line with the MOHS’ plans to improve MNCH. Supervision activities specifically allowed MCSP and local health departments to monitor and assess the outcomes from training for a longer period and to reinforce correct practice. The effectiveness of in-service capacity-building training conducted at L&PIC sites was evidenced by the improved capacity of training team members, shown by increases in post-test scores. It was also supported by the high skills and knowledge retention assessment scores of master mentors—80 percent on average at six months after training. Endorsed by the MOHS, MCSP supported the MOHS and EHO counterparts to implement an effective and standardized approach to in-service capacity building. Both state and non-state actors (EHOs) had access to the standardized training approach and curriculum. MCSP-supported policies, guidelines, and SOPs are now being used in nationwide trainings. Integration of cascade training models, QI, post￾training follow-ups, and other MCSP models into the NHP operational plans, together with the MOHS interest in expanding L&PIC sites to other S/Rs, suggests that MCSP was effective in creating local ownership around its interventions and supporting in-service capacity building at the system level. Capacity development in maternal care was evident via reported changes in practice at the local level, management of PPH, hypertension in pregnancy, effective use of partograms, neonatal care via newborn care, cord care, KMC care for small babies, early diagnosis, and referrals for high-risk child illnesses. There was effective usage of L&PIC models in sites visited, and the desk review suggested that all 10 L&PIC sites were functioning, though there were maintenance challenges noted for the future after MCSP that could pose a challenge for long-term functioning. Professional bodies had an opportunity to strengthen organizational capacity and develop CNE practices for their members. Examinations and relicensing of health care workers will strengthen the professional bodies and clinical care in the country. Improved capacity and skills of trained health workers was evident in post-training follow-up assessments showing high knowledge retention and quality improvement initiatives showing improved performance. MCSP successfully engaged with EHOs and increased trust and coordination between the MOHS and EHOs. There was a spillover benefit in the areas of male involvement in delivery and child-rearing practice, particularly in KMC units. EQ 1b. To what extent were the MCSP approaches and interventions aligned to health system realities to address the key barriers for strengthening in-service training at the state/region and township levels? FINDINGS Barriers to strengthening in-service training at the S/R and township levels are different across regions and related to various factors, such as resource availability, health workforce (e.g., assignment and 17 attrition rates), budget, as well as the existence of learning aids, updated curricula, leadership commitment, capacity at the S/R and township levels, and other supporting elements. How MCSP aligned to health system realities: The Myanmar health care system acknowledges that basic health staff, particularly midwives and lady health visitors, are the backbone of quality MNCH service delivery.26 The MCSP filled the gap by providing continuous learning for this cadre and supporting in-service training.27 MCSP was aligned with health system realities to address existing barriers faced by the MOHS, particularly through the use of a standardized training model. All respondents from KIIs (27/27) and members of 7 FGDs reported that MCSP was aligned with health system realities to tackle MNCH health problems and that MCSP complemented the national health system rather than overlapping with it. Findings from the desk review supported these reports, as MCSP efforts to strengthen health workforce capacity at the institutional level has contributed to the NHP policy and operational plan. Consultation with MOHS staff at the central level confirmed the interest of the MOHS in improving capacity. According to all six respondents representing MCSP IPs, MCSP consulted with MOHS counterparts in the design stage, developed a work plan based on MOHS inputs, and used existing government curricula in developing guidelines and tools. Respondents in the central￾level MOHS evaluation shared information on collaborative and consultative work conducted by MCSP. They had the perception that MCSP’s objective, which was to improve the capacity of MOHS staff to deliver quality MNCH care, was in line with the MOHS. One senior MOHS staffer in NPT, a key player in development of the NHP, shared that the government had taken ownership of and was committed to reducing the high MNCH mortality and morbidity in Myanmar, while MNCH components were involved in the majority of the basic essential health package outlined in the NHP. This respondent even claimed that “We always start all activities with MNCH, as it is the center of health care, and [MNCH] topics came as a top priority when situational analysis is done for the NHP prioritization process.” MCSP activities to strengthen MNMC for accreditation and relicensing support activities were in line with NHP Document Review 2017–2021 which states that “Competency-based licensing and relicensing of health professionals should be further developed and professional councils will be the focal licensing bodies.”28 MCSP’s use of a checklist system for clinical skills assessment, post-training, and quality improvement follow-up assessment enabled MOHS to monitor performance of trained health staff. 2. Addressing barriers to strengthening in-service training at all levels: Low levels of funding available for health care training activities is one of the many existing barriers the MOHS faces in strengthening in-service trainings at the S/R and township levels. There is generally insufficient budget allocated for training activities, including funding for training aids and materials.29 There are also shortages of competent staff to act as trainers and prevalent use of outdated training methods. Although the MOHS has a pool of master trainers, they have competing priorities and cannot cover all areas. Moreover, trained staff at the S/R level need refresher trainings to stay up-to-date on MNCH care and techniques. There is also the issue of accessibility of remote regions and barriers due 26 United Nations Population Fund in Myanmar. Myanmar Sexual, Reproductive, Maternal, Newborn and Adolescent Health (SRMNAH) Workforce Assessment Final Report. UNFPA, 2016. pp. 1–90. 27 MCSP. Summary of the evolution of USAID’s Maternal and Child Survival Program changes over time, inception to date. 28 Ministry of Health and Sports. 2016. Myanmar National Health Plan 2017-2021. 2016. 29 Ministry of Health and Sports. Myanmar National Health Plan 2017-2021. Quote from page 11: ‘Myanmar currently allocates 3.65 percent of its total budget on health, which is extremely low by global and regional standards’. 18 to conflict which can affect the replication of trainings at all levels. These challenges are compounded by human resource constraints, including staff shortages, attrition, and turnover.30 To address the barriers mentioned above, MCSP increased the number of competent staff at all levels via training, which allowed replication of trainings at all levels (central to S/R staff, and S/R training team members to the project township level and below). MCSP increased the number of competent staff trained in MNCH by using standardized, modularized, and updated training curricula, along with an interactive teaching model and a participatory approach. Staff trained in new methods allowed midwives to receive training with a focus on practice instead of classroom-based learning only. Learning models, aids, equipment, and resources provided to L&PIC sites minimized the constraints of limited availability of training aids and materials. MCSP was not able to address some barriers, including the adequate allocation of budget for training; human resource issues such as staff shortages, attrition, and turnover; and geographical and conflict constraints. These barriers were largely beyond MCSP’s scope. An interview with USAID staff highlighted that MCSP tried to introduce new tools for in-service MOHS trainings at the S/R and township levels; however, it was not clear to which these tools were disseminated beyond townships where training was directly supported by MSCP. While S/R training team staff have the capacity to deliver trainings on a wider level and in conflict-affected areas, covering these areas in the future depends on budget allocations and political will. EHOs have been included in the Myanmar Health Workforce Strategic Plan (2012–2017),31 which like the NHP recognizes the need to engage and partner with nongovernmental/EHO health workers in ethnic areas. The strategic plan suggested standardization of clinical skills as the first step toward full recognition of EHO health workers. A first step to support this was MCSP’s establishment of the L&PIC site at Karen Taw Nor Teaching Hospital in 2017. MCSP supported EHO MNCH trainers to be certified by the MOHS as BEmONC trainers, which was the first attempt of its kind. Respondents believed that continuing to provide certification for EHO trainers in all areas would support further high-quality training in all geographies of Myanmar. CONCLUSIONS MCSP activities were aligned with the Myanmar government’s interests and the MOHS’ agenda, as the program supported the MOHS’ strategic priority of strengthening human resources for health by building existing health workers’ capacity for delivering MNCH services. Consultation with the central MOHS in the design stage might contribute to program success while staff from the child health division in NPT and some state and regional staff suggested the need for more engagement with the respective departments in future programs. MCSP addressed health system barriers mainly at the central and S/R levels, as well as in MCSP-focused townships. A possible reason for limited scaling beyond MCSP townships was that MCSP’s main scope was demonstration and documentation, and township-level activities were added late in the program cycle. MCSP was not able to address other key barriers, including government budget allocations for training and human resource issues such as staff shortages, attrition, and turnover, as these were beyond MCSP’s scope. Supporting collaboration between the ministry and ethnic groups in conflict areas was broadly aligned with health system realities, and was important even if limited in scope and scale. 30 Sources: (1) Interview and FGD findings from MCSP evaluation; (2) United Nations Population Fund in Myanmar. Myanmar SRMNAH Workforce Assessment Final Report. UNFPA, 2016. pp. 1–90; (3) Ministry of Health. 2012. Health Workforce Strategic Plan 2012-2017. pp. 1–30. 31 Ministry of Health. 2012. Health Workforce Strategic Plan 2012-2017. 19 EQ 1c. To what extent were MCSP’s interventions and program design aligned to address drivers of maternal and child mortality and morbidity? FINDINGS Addressing the drivers of maternal neonatal and child mortality and morbidity: Maternal, newborn, and child mortality and morbidity are driven by multiple factors. Most drivers of mortality and morbidity can be attributed to the “three delays:” delays in the decision to seek care, delays in reaching health facilities, and delays in receiving effective and appropriate care once the mother/child contacts a service delivery site/provider. The first and second delays are mainly based on community and individual factors and determined by sociodemographic characteristics (poverty, husband and family support), health seeking behavior (knowledge and attitude to access health care), and accessibility factors (travel time, distance, infrastructure in remote areas, and road and weather conditions); both of these delays were outside of MCSP’s intended scope. MCSP was intended to help address the third delay, which relates to quality of care and other factors (staff, drugs, supplies, and equipment availability) in health facilities. MCSP addressed the skill gap in health staff by strengthening in-service staff capacity to deliver quality, effective, and appropriate MNCH health care. All respondents from the MOHS, four IP staff, and one USAID team member participating in KIIs said it might be too early to say whether MCSP addressed the drivers of MNCH mortality and morbidity. However, all agreed that the MCSP approach will plausibly contribute to reducing maternal mortality and morbidity in the longer term, specifically through improved capacity in prevention and management of PPH (the leading cause of maternal death) and by identifying high-risk cases and transferring them to the appropriate health facilities in time. MCSP reduced gaps in health service quality delivery by supporting MNCH-related national policies, guidelines, and SOPs; linking with the MOHS NHP, and reinforcing professional bodies. Introducing and providing evidence of QI achievements in normal labor and infection control practices using standard checklist systems in the five hospitals in Sittwe, Taunggyi, Lashio, Magway, and Pathein addressed drivers of morbidity and mortality, as staff were empowered to deliver quality health service. As described above, clinical staff at the MCSP-affiliated centers had increased performance scores on QI verification criteria from 45 percent to 80 percent. L&PIC sites achieving at least 60 percent of QI verification criteria also increased from less than 60 percent to 100 percent from baseline to endline. Regarding neonatal and child health, respondents from S/R training teams had positive perceptions about improving child health status due to training and improved practice of penguin suction, skin-to-skin care, and early identification and management of high-risk cases of infant and child illness. A study on causes of under-five deaths in Myanmar between 2014 and 2015 showed that newborn deaths contributed to about 48 percent of all deaths in children under 5, where 89 percent of newborn deaths were preventable and treatable, such as prematurity/low birth weight (36 percent), birth asphyxia (26 percent), neonatal jaundice (15 percent), and neonatal sepsis (12 percent).32 MCSP supported changes in neonatal and child health care practices by addressing the drivers of neonatal and child health morbidity and mortality via immediate newborn care, KMC care for preterm and low-birth-weight babies, and management of birth asphyxia via use of penguin suction or bag and mask. FGD respondents from S/R training teams shared how they utilized penguin suction and immediate newborn care and their observation of reduced asphyxia cases in their areas. Likewise, Thanlyin and Taunggyi pediatricians and nurses involved in KMC shared that KMC could contribute to reducing neonatal and child death rates, as neonatal death contributes to a large proportion of under-five child deaths in Myanmar. The evaluation team was not able to verify these observations with data. 32 MCSP. 2018. Newborn Health and Survival: The Adaptation and Implementation of Updated Newborn Guidelines to the Myanmar Context, Report. 20 MCSP’s ongoing impact modeling exercise could provide more insight for linkage of quality health care service to health impacts, as the program expected that increased numbers of competent midwives stationed at health facilities with the necessary equipment and supplies would encourage pregnant women to seek health care and reduce mortality and morbidity. CONCLUSIONS The main conclusion drawn from the above findings was that MCSP’s approach was consistent with addressing some known drivers of MNCH morbidity and mortality, as its activities and approaches strengthened health workers’ and the health system’s capacity for delivering in-service training of quality MNCH services using state-of-the-art tools. Quality of care in the form of using correct clinical protocols was reinforced by standardized policies, guidelines, and SOPs developed with support from MCSP. Preterm and low-birth-weight neonates (the leading cause of neonatal deaths) had a better chance of survival via KMC care. However, it was too early to determine how much MCSP contributed to reducing MNCH mortality and morbidity. While MCSP was working on an impact modeling analysis at the time of evaluation, there were no specific data that could verify estimates in the reduction of MNCH mortality and morbidity. Recommendations for EQ 1 For USAID: (1) Continue engagement with the MOHS and maintain the good practice of central-level buy-in from the MOHS. At the S/R and township levels and below, the Mission should encourage the implementing partners to engage at all levels in future programming while searching for opportunities to reach out at all levels for a realistic field experience as resources and time permit. (2) Strengthen and expand the effective collaboration of the MOHS and EHOs, not limited to KDHW alone, to align with and complement the NHP aim of delivering the essential package of health services. (3) Provide support and assistance to the MOHS to reach the township level and below while ensuring cost effectiveness and efficiency. For the IP: (1) Support should not be limited to the central-level staff in NPT and should include representative staff from both DPH and DMS at the S/R and township levels in pre-planning consultation. (2) Review resources put in L&PICs and ensure the government’s ability to maintain these resources. For the MOHS: (1) Continue engagement between the MOHS and EHOs to ensure access to a standardized curriculum and approach. (2) Continue briefing meetings between midwives and township￾level staff in the work plan so that central- or township-level health staff can share updated training with midwives and midwives can share challenges encountered (perhaps meet on paydays, when midwives from rural areas come to township offices). (3) To encourage the trained health staff on the use of condoms as a tamponade to stop massive PPH and magnesium sulfate for eclampsia in managing patients. (4) Strengthen the professional bodies (especially MNMC) and use the opportunity of leadership change in MNMC to review the Clinical Skills Standardization and Assessment Center (CSSAC) site at MNMC for possible space expansion. (5) Support the L&PIC site at MNMA as a training and exam preparation resource center that can serve as a fund-raising source. Provide support for continued professional education of MNMA members via the continuous nursing education approach, standardized training modules, and user-friendly training models. 21 EQ 2: HOW HAVE MCSP’S APPROACHES CONTRIBUTED TO THE POTENTIAL SUSTAINABILITY OF PROJECT RESULTS? 2a. How have MCSP’s approaches contributed to the potential sustainability of project results? FINDINGS Close consultation and planning with the MOHS contributed to sustainability: MCSP consulted with the government at the central level and worked closely with the S/R MOHS, professional bodies such as MNMA and MNMC, and non-state actors such as EHOs. This collaboration and consultation contributed to the sustainability of their work (9 KIIs with key staff from the MOHS NPT, MOHS S/R, MNMA, MNMC, EHOs). Link with the NHP and the MOHS’ policy: Respondents participating in 16 KIIs and two FGDs representing various stakeholders (donors, IPs, central and S/R MOHS staff) reported that MCSP interventions were supportive of the NHP and MOHS policies in general. For example, training team capacity development, the QI approach, and the development of L&PIC sites could be sustained after the activity’s conclusion. Integration of the MCSP model into the NHP itself is also key and will ensure sustainability of many approaches. According to five respondents (donors, directors, MOHS, IPs), the MOHS planned to expand L&PIC sites after the project, which shows their interest in sustaining the approach. Senior MOHS respondents noted that the program was embedded in the MOHS health system and that the central level has full ownership of the MCSP approach. A key player in implementing the NHP shared the view that MCSP’s cascade training and participatory training approach would be applied and replicated as the “integrated training model” in the NHP operational plan. According to this respondent, the NHP operational plan includes an “integrated training plan” which will streamline the number of trainings required for midwives and basic health staff. Observed use of guidelines and SOPs points toward sustainability: The development of national ANC guidelines, update and rollout of IMNCI, and rollout of F-IMNCI guidelines in partnership with the MOHS is evidence of a strong, sustainable approach that is already owned by the Myanmar government. The MOHS’ adoption of KMC SOP is another piece of evidence of the strong, sustainable nature of this work. Training approach, team commitment, and motivation: According to many respondents, the training model had the highest sustainability potential, followed by L&PIC sites (9 KIIs with key stakeholders and 5 FGDs). They reported that the standardized and modularized training curriculum and participatory training model increased trainees’ motivation and commitment to apply and replicate knowledge they learned even after MCSP. Revitalization of the S/R training team and interactive training methods allowed for participatory learning. The use of training models, teaching of facilitation skills, and the addition of the respectful mother care approach were all reported to be approaches that respondents wanted to maintain. 60 percent of respondents from both KIIs and FGDs stated that they would continue applying the knowledge and skills they learned even without external funding. While respondents generally thought the training approach was highly sustainable, there were several areas that were noted as areas where sustainability could be expanded or improved. ● Several respondents called for the curricula to be expanded further (e.g., more information on nutrition and exclusive breastfeeding). ● Some maternal health training materials such as PowerPoint presentations and guidelines were in English, and midwives had difficulty using them (KIIs with training coordinators and FGDs with midwives). ● There was also a need for training on how to develop a budget for training to ensure adequate allocation of resources. 22 ● Staff from S/R training teams suggested that if a training plan, timeline, and curriculum could be developed for both maternal and newborn child health, they could learn both at a single training, as they had to manage delivery and newborn care at the same time in real life. Post-training follow-up assessment, supervision, and clinical skilled assessment via QI verification improved staff quality and supported sustainability. The L&PIC platform: The MOHS’ interest in replicating and applying the MCSP-developed L&PIC model promotes sustainability of the project. This finding was supported by interviews and group discussions with 20 respondents representing various stakeholders with varied expertise. While none of the documents reviewed showed a specific budget allocation for L&PIC sites from the government, interviews with MOHS directors (2 KIIs) suggested that the MOHS is interested in setting up L&PIC sites in other S/Rs, which indicates the possibility of sustaining the approach. Interviews with S/R health staff and MNMA staff (5 KIIs and 3 FGDs) suggested that existing L&PIC sites could be sustained, but others shared that for the sites to be maintained, a specific budget allocation is needed. Further, there was a need to assign a specific workforce to the sites, which would ensure that there is adequate training and handover between staff when transferred. The inclusion of L&PICs as a priority in the NHP Annual Operational Plan is a positive sign of the MOHS’ buy-in to sustaining and replicating the model. L&PIC sites were affiliated with clinical sites, which contributed to the potential sustainability of the program. As seen in Taunggyi and Shan state, MCSP facilitated a series of multiplier training and training￾of-trainer workshops at L&PIC sites, which improved health workers’ clinical skills and training capacity. MCSP documented the improvement of midwives’ and lady health visitors’ MNCH practice skills via objective, structured clinical examination and pre- and post-test results. For example, participants of multiplier training in Hsi Hseng township in Southern Shan state showed an increase in the average skills score from 8 percent (baseline) to 90 percent (endline) for partogram use,33 which generated an effective rollout model for in-service capacity building and continuing professional development in the future. However, it is to be noted that the potential challenges of maintaining L&PIC sites might affect sustainability unless MOHS systems are put into place, such as plans to replace damaged equipment and models. Strengthened collaboration between key departments: MCSP facilitated a series of meetings and workshops which involved staff from various departments (particularly public health and medical services). This improved the effective collaboration among maternal and child health divisions as well as between public health professionals and clinical champions. Training and supervision visits encouraged effective collaboration between staff from different departments, which contributed to potential sustainability via increased cooperation between medical service and public health departments responsible for MNCH care. In addition, the MCSP model involved key stakeholders from the MOHS (DPH and DMS) and employed a two-pronged approach that enhances training teams’ capacity at the S/R and township levels. As a result, coordination and communication improved between DPH and DMS at all levels. Application and replication of MCSP intervention models: According to project documents and interview responses, MOHS staff have applied various components of MCSP, such as participatory training methodology, QI practices, SOPs, and quality of care in service delivery settings. A neonatologist from Shan state reported that the KMC model had been replicated in other hospitals in Shan state (e.g., Kyaing Taung Station Hospital, Ho Pone Township Hospital) .34 According to senior MOHS staff (2 KIIs), 33 MCSP. Myanmar Quarterly Progress Report. Program Year 4, Quarter1: October–December 2017. p. 18, Figure 2. 34 Hnin, Thein Thein. Quality Improvement of Newborn Care at Women’s and Children’s Hospital Taunggyi. July 2018. PowerPoint presentation. 23 the ministry plans to replicate or scale up interventions such as L&PICs, QI, cascade training models, post-training follow-up, and the KMC model. However, there are also barriers to sustainability related to some of the approaches introduced. Ten respondents from the S/R level, professional bodies, and IPs shared for example that the model “Noelle” was expensive and difficult to replace when broken. The evaluation team found that the model at MNMC was broken during direct observations. EHOs: The inclusion of EHOs allowed the MCSP activity to reach providers outside the public sector. Respondents from the MOHS, EHOs, donors, and IPs noted that bringing EHOs into MCSP activities increased coordination among different stakeholders, which contributed to the MOHS’ goal of Universal Health Coverage (UHC), as EHO staff could complement work in remote and conflict areas. According to the NHP, the government should integrate non-state actors to support expansion of UHC. Lessons learned and achievements from MCSP will provide useful insights for future activities. Another MCSP activity that contributed to sustainability was its effort in aligning non-state health actors’ services with best practice standards in Karen state via the EHO KDHW, which improved the coordination between state and non-state health actors. Other EHO-related activities that may contribute to sustainability include the application of standardized guidelines and curricula in the design of future trainings in EHO areas and the availability of an L&PIC in KDHW’s Taw Nor Teaching Hospital in Kayin state35. Professional development of midwife association and regulatory bodies contributed to sustainability: Respondents believed that L&PIC sites in MNMC and MNMA will be sustained even after MCSP. In Quarter 2 of Year 3 of the activity36, MCSP successfully conducted a planning workshop with MNMC members to plan for utilization and sustainability of the MNMC CSSAC beyond the life of MCSP. The plan developed from that workshop suggested the continued use of the CSSAC as a center for the administration of midwifery licensure and relicensure examinations, nursing professional education sessions, and skill lab coordinator training for new MNMC members. MNMA members, nurses, and midwives were relicensed via MCSP-supported L&PIC sites and will have opportunities to continue their professional development. All these activities contributed to the project’s potential sustainability. Additional findings In 2018, to inform inputs into the Inclusive Township Health Plan process, MCSP facilitated a sustainability workshop and defined plans for using and maintaining the L&PIC platform beyond the life of the MCSP program.73 According to interviews with MCSP staff members and documents reviewed, senior staff from S/R health departments and S/R training team members participating in the workshop were eager to sustain MCSP interventions. They developed a resource mobilization plan to maintain the L&PICs and proposed various fund-raising ideas during the workshop to support sustainability. CONCLUSIONS Overall, government commitment and buy-in to the MCSP approach is a strong point for sustainability. The MOHS’ commitment to applying and replicating MCSP activities and its endorsement of MCSP￾developed guidelines and policies contributed to the sustainability of the project. Future activities under the MOHS’ consideration include expansion of L&PIC sites and potential integration of MCSP interventions via the use of standardized and modular training curricula with nationwide trainings. In addition, MCSP’s technical approach of cascade training models via multiplier trainings at the central and S/R levels and MCSP-focused townships promoted program sustainability, as the models utilized MOHS training teams and structures and were well received. Furthermore, the MOHS’ NHP Operational Plan indicates the ministry’s intent to adapt MCSP’s cascade training model in developing a nationwide integrated training model and plan. As described above, sustainability of MCSP’s work is also supported 35 EHO will tailor the MCSP-developed curriculum based on their existing curriculum. 36 MCSP. Myanmar Quarterly Progress Report. Program Year 4, Quarter1: October–December 2017. 24 by the strong retention of training knowledge six months after training, increased post-test skills of trainees, improved QI performance assessments of clinical staff, and positive findings regarding training team members’ confidence and motivation. Lastly, high acceptance of the KMC model, staff commitment, and the MOHS’ interest suggest that KMC is a scalable and sustainable model. Recommendations for EQ 2 For USAID: (1) Support efforts of the MOHS to sustain government ownership and commitment even after leadership changes. (2) Support application and replication of effective MCSP interventions in USAID’s new Essential Health Program, which is also aligned with the NHP Essential Health Care package. (3) Identify, encourage, and support motivated staff from the MOHS who are at decision￾making levels and recognize them through a “champion” pool. For IPs: (1) Continue advocating for the integration of MCSP cascade training modules into the MOHS National Health Plan. (2) Include training modules with comprehensive information on the nutrition component, such as exclusive breastfeeding, maternal and child nutrition, etc. (3) Follow up on and implement the sustainability plan developed in consultation with the MOHS and key stakeholders and secure support from other sources if feasible (e.g., Essential Health, Access to Health, etc.). (4) Support (technically and financially) the MOHS in integrating MCSP activities into the MOHS NHP, particularly the integrated training plan, QI process, and L&PIC extension. For the MOHS: (1) Integrate MCSP best-practice activities in the MOHS NHP Operational Plan and annual work plans. (2) Strengthen a dedicated training team at the S/R level to focus on trainings, equipment, and models used in L&PIC and the functioning of L&PIC sites. (3) Update clinical and training guidelines in DPH, DMS, and pre-service training at nursing universities at the same time so that MCSP and non-MCSP trained staff have an equal opportunity to learn updated MNCH techniques and practices. (4) Allocate a training budget in the annual work plan and train MOHS township-level staff to understand ministry budget and finance systems to be able to request training resources from S/Rs and the central level in a bottom-up approach. (5) Develop training materials and resources in Burmese and, if feasible, ethnic languages, particularly in preparation for township health staff- and below-level trainers. EQ 3: WHAT ARE THE SPECIFIC LESSONS THAT CAN BE LEARNED TO INFORM FUTURE PROGRAMS THAT AIM TO STRENGTHEN SYSTEMS FOR CAPACITY BUILDING RELATED TO MNCH, PARTICULARLY AT THE TOWNSHIP LEVEL? 3a: Any similar approaches/interventions that should/should not be supported/replicated through future assistance? Why/ why not? FINDINGS All KII and FGD respondents agreed that existing MCSP interventions should be replicated in future programs. There was no major negative feedback on the model/approach employed by MCSP. All respondents agreed on the importance of leadership and collaboration. The evaluation team found that MCSP’s technical approaches were strong and in line with international best practices. Respondents favored the L&PIC sites, cascade and modularized training models, QI and KMC models, as well as post￾training supervision for performance assessment. In addition, none of the training team members complained about training methods or clarity. Respondents participating in KIIs and FGDs were able to articulate updated training techniques they learned by using models and pointing to clear SOP guidelines with pictures. However, it is noteworthy that six FGD participants from S/R training teams expressed concerns over the training duration and the fact that they had to leave their duty station to attend the training. MCSP demonstrated that a competency-based approach is workable. Respondents valued the coordination mechanisms set up by MCSP between different departments in the central MOHS, 25 between central and S/R MOHS, between S/Rs and designated townships, and between the MOHS and other stakeholders such as professional bodies, EHOs, and others. However, according to the respondents, IPs have more engagement with the central MOHS than the S/R level and they suggested having more communications and engagement with the S/R level for future programs. CONCLUSIONS In general, there was no negative feedback on MCSP’s strategy or technical approach. MCSP’s programming was in line with MOHS’ interests and policies to improve MNCH health status in Myanmar. Interventions such as trainings, post-training follow-up, QI, and KMC received high replicability ratings. The model of L&PIC sites was also highly praised by respondents. Lessons learned for this program suggested that MOHS leadership is key to successful program implementation. It is important to have MOHS leadership during the program design phase and at various implementation sites. Other key aspects that can be replicated in future program designs include engagement with various stakeholders at all levels and program activities that encourage inter- and intra-MOHS department collaboration. 3b. Any challenges in the health system that MCSP did not address that would need to be addressed for future programs to be successful, particularly in effecting improvements at the township level and below? FINDINGS The MCSP supported in-service capacity building of the health workforce via a multipronged approach, which addressed health system capacity-building challenges in various perspectives. Four out of six IP respondents reported that the availability of government staff for meetings, trainings (both as trainees and trainers), and supervision visits was limited due to their workloads and staffing constraints. According to the IPs, the training model was not adjusted for the existing human resource system, as it did not include a plan to manage staff turnover or transfer. Multitasking and heavy workloads sometimes hindered selected staff’s ability to attend and/or complete the training. MCSP did not address human resource challenges within the MOHS system, as this was out of scope. However, the program did help stakeholders become aware of human resource constraints. Another challenge was the quality of data. For example, MCSP had to rely on secondary relicensing data from MNMC, which were sometimes delayed in delivery. Other challenges that affected the timeliness and reliability of the data reported include MNMC’s election process, delayed new council law implementation, delayed activities due to violence and weather in Rakhine and Northern Shan state, and shifts in the scope of activities and work plans. As DPH was the main counterpart for MCSP’s midwife capacity-building activities, the training plan was mainly developed in consultation with DPH. However, there was a need to involve administrative and clinical staff from the Department of Medical Services, which oversees hospitals. Interviews and group discussions with respondents from the clinical side revealed that staff from clinical sites were employed on shift work with a rotating schedule. It was hard for them to attend full training courses if there was no one to cover their duties. According to various stakeholders, another challenge was that the information flow was sometimes not equally distributed among different departments (DPH and DMS). Finally, according to a midwife who participated in an FGD, there was sometimes a discrepancy in practice between trained midwives and staff in referral hospitals, particularly when hospital staff were not aware of updated guidelines and practices. Respondents (KIIs with 3 IPs and 2 S/R senior health staff) reported that limited authority and insufficient delegation of work was a barrier to effective program implementation. Many reported that the lack of decentralized authority reduced MCSP’s effectiveness at the S/R level, as people had to wait for central-level approval in some cases. 26 Respondents reported that the program has reached MCSP-focused townships only. As described above, the evaluation team acknowledged that MCSP aimed to document and demonstrate best practices, and the program design was intended to cover MCSP-focused townships only. However, it is noteworthy that many respondents reported a need to roll out the training to a broader audience below the central and S/R levels in the future. A USAID staff member also highlighted a broader need to build capacity at the township level and below. MNMC senior leadership expressed a need to address space constraints in their L&PIC site and suggested that this site can also serve as an exam center. They also expressed a desire for a sustainability plan to manage maintenance of the models used in the L&PIC site. According to MCSP’s second-quarter report in 2018, the program facilitated the development of a sustainability plan for MNMC, in which these issues were discussed and options were identified. The team found that the mitigation plan for replacing broken models at L&PIC sites was not in place during data collection in July 2018. MCSP was aware of this issue and later shared model suppliers’ contact information with relevant counterparts at the dissemination event of “L&PIC In-service Capacity Building Model: Implementation Guide” in September 2018. CONCLUSIONS MCSP supported in-service capacity building of the health workforce via a multipronged approach and addressed health system capacity-building challenges from various perspectives. MCSP coverage has reached central, S/R, and MCSP-focused townships only, according to the project design. MNMC has space constraints in hosting assessment exams. MCSP facilitated the development of sustainability plan for MNMC in which these issues and solutions were discussed. Likewise, although there was initially no mitigation plan for replacing broken models at L&PIC sites at the time of the evaluation team’s fieldwork, MCSP was aware of this issue and shared model suppliers’ contact information with respective counterparts later. Some challenges were out of MCSP’s scope, such as human resource constraints (staff turnover, attrition), coordination among government departments, and political challenges such as conflict in Rakhine. The effectiveness of the training implementation depends on human resources, financial resources, leadership dedication, and team motivation. 3c: Any interventions/supports that should be removed or modified to better adjust interventions to health system realities? FINDINGS With three distinct work plans, MCSP initially intended to take on smaller and discrete interventions. As funding became available, MCSP later integrated the three work plans into a comprehensive strategy in 2017. For future interventions, having a single work plan with a consistent theory of change would benefit program implementation. Changing the work plan in the middle of the project cycle had mixed results. On the one hand, it allowed the program to reprioritize activities. On the other hand, the effort spent on creating the work plans and having them approved delayed program implementation. Senior staff from S/R health departments shared that they were introduced to training plans with little notice sometimes, which made it difficult to assign staff properly. The evaluation team acknowledged that demonstration at the township level was added in the last year of the program, which led to a situation where MCSP was organizing many trainings for a short implementation period. This resulted in shorter preparation times for training, including short notice given to S/Rs. Respondents believed that activity plans should be communicated to a broader range of relevant stakeholders, including S/R leaders (director level), township medical officers under the DPH, medical superintendents, specialists, and matrons under the Department of Medical Science. 27 CONCLUSIONS Revision of work plans during the project cycle maximized outcomes but was also time-consuming and delayed implementation in some cases. Better planning for work plan revisions should be considered in future programming. MCSP promoted effective collaboration among Maternal and Reproductive Health Division, Child Health Division, DPH, and the MOHS. More collaboration and information sharing between different departments and among central, S/R, and township levels was needed. 3d: Any necessary modifications to the models and interventions supported by MCSP, including their mode of delivery, if future replication is considered? FINDINGS MCSP interventions comprised a number of discrete activities at the beginning. A comprehensive set of interventions with a theory of change was only developed in early 2017 when the project budget and scope was expanded at the request of USAID. Although the unified work plan entailed pulling the discrete interventions together under a common results framework and theory of change, it would have been easier to discern MCSP’s outcomes if the common results framework had been in place since the beginning. A senior staff member from the Child Health Division, NPT, suggested the program include training for emergency child health referral if future replication of the program is considered. Under MCSP, the criterion for becoming a post-training supervisor for F-IMNCI was being a child specialist, as clinical skills and experience were needed to adequately support and supervise township hospital-level staff. However, there are only two child specialists at the S/R level in some areas, and geographical constraints hindered their ability to participate in frequent trainings. The desk review of the report supported that finding as it states that “limited availability of the trainers or supervisors for conducting the post-training supervision visits and clinicians, who are the trainers and whose skills are best suited for the post-training supervision, are also very busy with their clinical duties”37. MCSP facilitated awareness of human resource barriers, but could not address the challenges directly, as human resource management was owned by the MOHS. The training model did not adjust for existing human resource limitations or plan for high staff turnover. EHO staff shared their willingness to practice at government township hospitals for better hands-on experience. One senior EHO staff member suggested that EHOs could serve as an implementation counterpart in future programs. To extend a similar intervention to the township level and below in future programs, the concept of decentralization or providing more authority or delegation at S/Rs and below should be considered. A quote from NHP report 2017-2021 states that “decision-making with respect to the deployment of human resources should be gradually decentralized to S/R.” ADDITIONAL FINDINGS Recognition or accreditation: EHOs appreciated that MCSP linked EHOs with the MOHS and central MOHS staff provided standardized trainings in EHO areas. EHOs requested to have licenses or equal recognition from the MOHS for their experienced and competent staff. From the MOHS’ perspective, MOHS senior staff shared that the quality improvement (QI) modality used in MCSP would be very useful in preventing poor-quality care and promoting standard-quality care, 37 MCSP Child Health Team. F-IMNCI Case Study. Southern Shan State, 2016-2018’.2018. Page 27 28 while health worker recognition and accreditation could be set up by the performance-based assessment as outlined in the NHP. MNMC’s executive committee members highlighted the upcoming private nursing and midwife training schools and institutions. CONCLUSIONS To extend support to the township level and below in future programs and ensure this type of external support is effective, there is a need to decentralize the MOHS’ authority or provide more authority to the S/R level within the health system. The MOHS should pay attention to staff replacement for turnover, retirements, and reallocation. A theory of change and training plans should be developed in advance, with all stakeholders included in future programs. Selecting child specialists as supervisors/trainers for F-IMNCI was sometimes a challenge, especially when there were staffing constraints, given there are very few such specialists. Recommendations for EQ 3 For USAID: (1) Consider supporting the MOHS and IPs to reach out at the township level and below in a future similar program by setting up mini L&PICs at district and township levels, which would be based on the MCSP L&PIC model but tailored to fit the peripheral level in consultation with staff from districts and townships. (2) Consider providing technical and financial assistance to the MOHS to promote capacity of MOHS staff, particularly at the lower level, and encourage decentralization by linking with the MOHS NHP Operational Plan. (3) Ask that IPs include the MOHS in discussing potential changes in work plans, funding mechanisms, and expected deliverables. (4) Consider supporting the skilled lab (L&PIC) site at MNMA as a training and exam preparation resource center, and as a fundraising source, by allowing nursing students and medical students to practice using models at training sites. For IPs: (1) Continue the momentum of high-scoring intervention models based on the findings from this evaluation, such as training, post-training follow-up, and QI models. Continue the best practice of engaging the MOHS at the central level. (2) Support capacity building of professional bodies at the organizational level. For the MOHS: (1) Consider an appropriate approach to overcome the challenges of finding available and qualified post-training supervisors for F-IMNCI by reviewing the existing selection criteria. (2) Develop an action plan and strategy to reach the township level and below and make sure to involve staff from clinical sites (both hospital management and clinical staff) in all steps in programming. (3) Develop a knowledge-sharing model between newly assigned staff and existing or transferred trained staff. Identify ways to pass knowledge gained and to develop clear job descriptions for knowledge handover. (4) Forecast HR management (transfer, replacement) in line with future program yearly plans and develop a plan to address challenges arising from staff turnover. 29 ANNEXES ANNEX A. EVALUATION SCOPE OF WORK EVALUATION OF HEALTH SECTOR CAPACITY DEVELOPMENT THROUGH USAID PROGRAMMING IN MIDWIFERY, DRUG QUALITY MONITORING, AND SURVEY IMPLEMENTATION I. PURPOSE OF THE EVALUATION This evaluation will examine how USAID-supported programming has affected the development of Myanmar national capacity in several areas, looking at key dimensions of human and institutional capacity and commitment. The study is divided into two components with two distinct deliverables, one focused on midwifery and in-service training for health care workers in maternal, neonatal and child health, and the other focused on two specific departments in the Ministry of Health and Sports (MOHS) that have received technical assistance through USAID programs: Department of Food and Drug Administration (DFDA) and the Department of Health Planning (DHP). At the conclusion of the studies, a dissemination event will be organized to share findings with key stakeholders, including implementing partners and the MOHS. A. Component A: End-line Performance Evaluation of Maternal and Child Survival Program (MCSP) Contractor will conduct an external endline performance evaluation for the Maternal and Child Survival Program (MCSP), a 3-year, $8.1 million field support buy-in to the MCSP global mechanism, and support dissemination of findings. The purpose of this evaluation will be to examine the extent to which MCSP’s interventions supported the country’s capacity and systems for in-service training of health workers to improve availability and quality of maternal and newborn care services. The evaluation will analyze the effectiveness of the in-service capacity building approaches supported by MCSP, including the Learning and Performance Improvement Center (L&PIC) model, the roll-out approach for competency based capacity building at the state/regional level and below, the standards-based quality improvement model introduced at selected training sites, and complementary efforts to strengthen institutions such as the Myanmar Nurse and Midwifery Council and Myanmar Nurse and Midwives Association (MNMC and MNMA). This information will be used to inform approaches for continued strengthening of in-service training at lower levels of the health system under USAID’s follow-on Essential Health program, and to generate recommendations for USAID or other development partners on how to optimize support to the MOHS to deliver integrated in-service training interventions and build related country systems through future programs. III. 30 A. Description of the Problem, Development Hypothesis(es), and Theory of Change 1. a. Component A: MCSP’s theory of change is: → If MCSP builds on past experience to… • Strengthen and build coordination among the institutions and systems that govern capacity development for health workers; • Introduce transformative, coordinated and targeted competency-based approaches to provider education (in-service training and continuing professional development), including on-the-job at facilities where quality improvement (QI) efforts, based on standards of quality care are implemented; and at the same time • Strengthen the regulation of practice to improve the governance and practice of health providers in maternal, newborn and child health; And if these pilot activities are well documented and shown to be effective, →Then they can be scaled up by the government and/or other actors; and as services improve, maternal, newborn and child lives will be saved. The intermediate results in the approved MCSP work plan for 2017-2018 include working with the MOHS and key partners to achieve the following: 1. Policy environment strengthened for improving quality and equitable access to maternal, newborn and child health services 2. Health workforce strengthened to support effective delivery of MNCH components of the Essential Package of Health Services 3. Quality health service delivery strengthened in targeted technical and geographical areas 1.b. Component A: Summary of MCSP’s goal and approaches to be assessed The activity’s stated goal is to respond to the Ministry of Health and Sports’ (MOHS) strategic priorities for improving maternal, newborn and child health by demonstrating, documenting and transitioning capacity to counterparts to make sustainable improvements in the health system. One purpose of the final performance evaluation is to assess MCSP’s efforts to build capacity and systems for in-service training, which covers a sub-set of the overall package of interventions supported by the project in Myanmar. Specifically, in-service training approaches have centered around four key “models” or approaches being introduced by MCSP, and for each, a documentation package intended to support adoption and scale up of these models will be developed by the implementing partner. The project implementation shifted over time from a focus on training to an increased systems strengthening-oriented approach. The assessment should account for the fact that the project emphasis and model shifted between project years. 31 The four models include: 1. The Learning and Performance Improvement Center (L&PICs) model for in-service capacity building, established in five states & regions and at MNMA, MNMC, and Taw Naw Teaching Hospital in Kayin State (L&PICs were also established to support pre￾service training in two Nursing Universities with USAID funding, and in midwifery training schools with 3MDG funding); 2. The roll out approach for competency based capacity building, using the L&PICs combined with complementary support to MOHS counterparts to plan and execute in￾service training at the state/regional level and below; 3. A model for strengthening the institutions that the International Confederation of Midwives (ICM) has identified as central to strengthening the midwifery profession (MNMC and MNMA); 4. A standards-based quality improvement model for clinical training sites affiliated with selected L&PICs. 2. a. Component B: If the PQM and DHS strengthen the institutional capacities of the two departments - DFDA and DHP, the performance on drug quality monitoring and health survey implementation will be improved. II.EVALUATION AND ASSESSMENT QUESTIONS A. Component A: End-line Performance Evaluation of Maternal and Child Survival Program (MCSP) Question 1: To what extent did MCSP assistance influence in-service training practices and related systems to improve maternal, neonatal and child health? In answering this question, the contractor must address the following: a) To what extent have the MCSP’s in-service capacity building activities, including the models outlined in Section B and associated interventions, influenced policies, practices and the enabling environment for in-service training at different levels of the system (regulatory and professional bodies, central MOHS, state/regional level, and township level and below)? To what extent have health system actors been able to apply and replicate interventions introduced by MCSP? b) To what extent were MCSP approach and interventions aligned to health system realities in order to address the key barriers for strengthening in-service training at State/Region and Township levels, and; c) To what extent were MCSP's interventions and program design aligned to address drivers of maternal and child mortality and morbidity? In addressing this question, the evaluator is to consider alternative ways of developing health human resource capacity and how the chosen model implemented compares with other alternatives in terms of effectiveness and efficiency in international best practice. 32 Question 2: How have MCSP’s approaches contributed to potential sustainability of project results? In answering this question, the Contractor must address the following: a) What interventions will likely be/not be sustained or scaled up by the Government of Myanmar? b) What are key factors/evidence that support such conclusion(s)? Question 3: What are the specific lessons that can be learned to inform future programs that aim to strengthen systems for capacity building related to maternal and child health, particularly at the township level? a) Any similar approaches/interventions that should/should not be supported/replicated through future assistance? Why/why not?; b) Any challenge(s) in the health system that MCSP did not address which would need to be addressed for future programs to be successful, particularly toward affecting improvements at the township level and below; c) Any intervention(s) /support(s) that should be removed or modified to better adjust interventions to health system realities; and d) Any necessary modifications to the models and interventions supported by MCSP, including their mode of delivery, if future replication is considered. III. DESIGN AND METHODOLOGY A. Component A: End-line Performance Evaluation of Maternal and Child Survival Program (MCSP) Qualitative and quantitative data should be collected and analyzed using case study methodology or other appropriate methods, such as systems analysis and complexity-aware methods that account for the short period of project intervention while helping to understand the suitability and replicability of MCSP-supported models and interventions around in-service training. The evaluation team will propose an appropriate method in consultation with USAID. There is no overall baseline data that would allow for before-after comparison or with a control group to assess change over time. The project only has limited baseline data on a few quality measures in selected sites available which may allow for some limited secondary analysis to show the extent or reach of MCSP interventions over time. Availability and comprehensiveness of project-produced data and documentation varies given that the approach for MCSP shifted significantly between work plan years. The existing data include performance statistics from MCSP on program implementation over time and detailed project documentation on project components, cost, and details of the model rollout. In addition to existing project data, the evaluation team may have to draw on evidence from other donor programs in similar contexts. To answer the evaluation questions, the evaluation team will have to collect supplementary qualitative (and, as relevant, quantitative) information through key informant interviews, focus 33 group discussions, and survey questionnaires. Questions may focus, for example, on perceived changes due to project activities, project sustainability, and intended and unintended outcomes. Key informants may include project staff, USAID staff, ministry staff at the central level and state health training team members and counterparts at the state and regional level, township medical officers and members of township training teams at the township level and below including midwives, patients, and targeted beneficiaries. Other donors and partners active in the MCH space (including 3MDG technical advisors, UNICEF, WHO, MNMC, MNMA, and MMA- OB/GYN Society), also will likely have valuable perspectives on the role and impact of USAID support for in-service training strengthening in midwifery. The design matrix and methods below are the illustrative and the contractor may propose other methods as appropriate. The evaluator may also propose alternative wording of evaluation questions if desired: Questions Suggested Data Sources (*) Suggested Data Collection Methods Data Analysis Methods Question 1: To what extent did MCSP assistance influence in-service training practices and related systems to improve maternal, neonatal and child health? In answering this questions, the contractor must address the following: a) To what extent have the MCSP’s in- service capacity building activities, MCSP learning agenda documentation (to be completed by Dec 2017), workplans and reports from MCSP project and other MCH projects reports in country, other relevant in country document such as Annual Operational Plan of National Qualitative (key informant interview and/or focus group discussions etc as relevant), Desk review, secondary analysis as necessary To be determine d by the contractor 34 including the models outlined in Section B and associated interventions, influenced policies, practices and the enabling environment for in-service training at different levels of the system (regulatory and professional bodies, central MOHS, state/regional level, and township level and below)? To what extent have health system actors been able to apply and replicate interventions introduced by MCSP? b) To what extent were MCSP approach and interventions aligned to health system realities in order to address the key barriers for strengthening in-service training at State/Region and Township levels, and; c) To what extent were MCSP's interventions and program design aligned to address drivers of maternal and child mortality and morbidity? In addressing this question, the evaluator is to consider alternative ways of developing health human resource capacity and how the chosen model implemented compares with other alternatives in terms of effectiveness and efficiency in international best practice. Health Plan (NHP), Yearly NHP implementation report if available, routine facility data, stakeholders & project beneficiaries, finding from the surveys. Question 2: How have MCSP’s approaches contributed to the potential sustainability of project results? What interventions will likely be/not be sustained or scaled up by the Government of Myanmar? What are key factors/evidence that support such conclusion(s)? MCSP learning agenda documentation (to be completed by Dec 2017), workplans and reports from MCS`P project and other MCH projects reports in country, other relevant in country document such AOP of NHP, Yearly NHP implementation report if available, routine facility data, stakeholders & project beneficiaries, finding from the surveys. Qualitative (key informant interview and/or focus group discussions etc as relevant), Desk review, secondary analysis as necessarily. To be determined by the contractor 35 What are the specific lessons that can be learned to inform future programs that aim to strengthen systems for capacity building and in-service training related to maternal and child health, particularly at the township level? a. Any similar approaches/interventions that should/should not be supported/replicated through future assistance? Why/why not?; b. Any challenge(s) in the health system that MCSP did not address at Township level which would need to be addressed for future programs to be successful; c. Any intervention(s) /support(s) that should be removed or modified to better adjust interventions to health system realities.; and d. Any necessary modifications to the models and interventions supported by MCSP, including their mode of delivery, if future replication is considered. MCSP learning agenda documentation (to be completed by Dec 2017), workplans and reports from MCSP project and other MCH projects reports in country, other relevant in country document such AOP of NHP, Yearly NHP implementation report if available, routine facility data, stakeholders & project beneficiaries, finding from the surveys. Qualitative (key informant interview and/or focus group discussions etc as relevant), Desk review, secondary analysis as necessarily. To be determined by the contractor V. FINAL REPORT FORMAT (THIS SECTION APPLIES TO BOTH COMPONENT A AND B) The final reports (for each Component) must include an abstract; executive summary; background of the local context and the project being assessed; the evaluation/assessment purposes and main evaluation/ assessment questions; the methodology or methodologies; the limitations to the evaluation/assessment; findings, conclusions, and recommendations. For more detail, see “How- To Note: Preparing Evaluation Reports” (Attachment 2) and ADS 201mah, USAID Evaluation Report Requirements. An optional evaluation report template is available in the Evaluation Toolkit. Each executive summary must be 2–5 pages in length and summarize the purpose, background of the project being assessed, main assessment questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The methodology must be explained in the report in detail. Limitations to the assessment/evaluation must be disclosed in the report, with particular attention to the limitations associated with the assessment/evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.) The annexes to each report must include: 36 ● The task order Statement of Work (SOW); ● Any statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the assessment/evaluation team; ● All data collection and analysis tools used in conducting the evaluation/assessment, such as questionnaires, checklists, and discussion guides; ● All sources of information, properly identified and listed; and ● Signed disclosure of conflict of interest forms for all evaluation/assessment team members, either attesting to a lack of conflicts of interest or describing existing conflicts of interest. ● Summary information about evaluation/assessment team members, including qualifications, experience, and role on the team. VI. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION/ ASSESSMENT REPORTS (THIS SECTION APPLIES TO BOTH COMPONENT A AND B) Per ADS 201maa, Criteria to Ensure the Quality of the Evaluation Report, the draft and final reports will be evaluated against the following criteria to ensure the quality. ● The report must represent a thoughtful, well-researched, and well-organized effort to objectively evaluate/assess the project. ● The report must be readily understood and should identify key points clearly, distinctly, and succinctly. ● The Executive Summaries of the report must present a concise and accurate statement of the most critical elements of the reports. ● The report must adequately address all questions included in the SOW, or the questions subsequently revised and documented in consultation and agreement with USAID. ● The evaluation/assessment methodology must be explained in detail and sources of information properly identified. ● Limitations to the evaluation/assessment must be adequately disclosed in the reports, with particular attention to the limitations associated with the methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). ● Findings must be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. ● Findings and conclusions must be specific, concise, and supported by strong quantitative or qualitative evidence. ● If findings assess person-level outcomes or impact, they should also be separately assessed for both males and females. If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. VII. OTHER REQUIREMENTS 37 All modifications to the required elements of the SOW of the contract, whether in technical requirements, evaluation/assessment questions, evaluation/assessment team compositions, methodology, or timeline, need to be agreed upon in writing by the Contracting Officer (CO). Any revisions must be updated in the SOW and only the final SOW shall be included as an annex to the Report. 38 ANNEX B. DOCUMENT REVIEW SOURCES MCSP. Summary of the evolution of USAID’s Maternal and Child Survival Program changes over time, inception to date. MCSP. 2017. MCSP Myanmar Workplans. MCSP. 2017. MCSP Myanmar. Implementation Plan Program Years 1-3. July 2015-September 2017. MCSP. 2017. MCSP Myanmar. Implementation Plan Program Years 3-4. March2017-June2018. MCSP. 2016. MCSP Myanmar. Performance Monitoring Plan (PMP). October 2015-September 2017. MCSP. 2017. MCSP Myanmar. Monitoring and Evaluation Plan Program Year 3-4. March 2017-June 2018. MCSP.2017. Myanmar Documentation and Dissemination plan of MCSP. MCSP. Annual Reports Program Year 1,2,3. MCSP. 2017. Myanmar Annual Summary Program Year 3, October 2016-September 2017. MCSP. 2017.Myanmar Quarterly Progress Report. Program Year 4, Quarter1: October-December 2017. MCSP. 2018.Myanmar Quarterly Progress Report. Program Year 4, Quarter2: January-March 2018. MCSP. 2018.Myanmar Quarterly Progress Report. Program Year 4, Quarter3: April-June 2018. MCSP. 2018. Learning and Performance Improvement Center In-Service Capacity Building Model: Myanmar, Implementation Guide (draft). MCSP.2018. Facility Quality Improvement for Maternal and Newborn Care, Implementation Guide (draft). MCSP.2018. Strengthening Small Baby Care at Thanlyin General Hospital through Kangaroo Mother Care. MCSP.2018. The Feasibility and Acceptability of Kangaroo Mother Care in Neonatal Unit, Women’s and Children’s Hospital, Taunggyi, Myanmar, Preliminary Results, Report Brief and Presentation. MCSP.2018. Newborn Health and Survival: The Adaptation and Implementation of Updated Newborn Guidelines to the Myanmar Context, Report. MCSP. Preventing the Needless Deaths of Women and Their Families. 2018. PowerPoint presentation. MCSP Child Health Team.2018. F-IMNCI Case Study. Southern Shan State, 2016-2018. MCSP Child Health Team.2018. Final Report of IMNCI Training Courses in Five Selected States and Regions, 2016-2018. Hnin, Thein Thein Professor. “Quality Improvement of Newborn Care at Women’s and Children’s Hospital Taunggyi.”, July 2018. PowerPoint Presentation. Htwe, Ye Min. “NHP 2017-2021: The Way Forward.” Nay Pyi Taw, December 2016. PowerPoint presentation. MCSP. Myanmar Documentation and Dissemination Plan. MCSP, 2018. Ministry of Health and Sports Department of Public Health. 2017. Public Health Statistics (2014-2016). Nay Pyi Taw. http://mohs.gov.mm/Main/content/publication/public-health-statistics-report-2014- 2016 Ministry of Health and Sports. Myanmar National Health Plan 2017-2021. MoHS. 2016.Pp. 2-40. 39 Ministry of Health and Sports. Myanmar National Health Plan 2017-2021. Annual Operational Plan 2017- 2018. MoHS, May 2017.Pp 1-48. Ministry of Health 2012. Health Workforce Strategic Plan 2012-2017. Myanmar Ministry of Health, 2012. Pp. 1-30. Ministry of National Planning and Economic Development and Ministry of Health. Myanmar Multiple Indicator Cluster Survey 2009 - 2010 Final Report. Nay Pyi Taw: Ministry of National Planning and Economic Development and Ministry of Health, 2011. Print. Ministry of Health and Sports & DHS Program ICF. Myanmar Demographic and Health Survey 2015-16. Nay Pyi Taw, Myanmar & Rockville, M.D.: DHS & MoHS, 2017. Pp. 1-453. Sin Htoo, Thant. “NHP 2017-2021: The Need to Prioritize.” Nay Pyi Taw, December 2016. The Three Millennium Development Goal Fund. Reasons for Optimism. Yangon: 3MDG, 2016. Pp. 2-15. Tin, Nilar. “NHP 2017-2021: Systems Building, Supportive Environment, Inclusive Township Health Planning (ITHP).” Nay Pyi Taw, December 2016. PowerPoint presentation. UNFPA Myanmar. Myanmar SRMNAH Workforce Assessment Final Report. UNFPA, 2016. Pp. 1-90. Union of Myanmar Ministry of Immigration and Population Department of Population. Country Report on 2007 Fertility and Reproductive Health Survey. Nay Pyi Taw: UNFPA, 2009. USAID. Evaluation: Learning from Experience. Washington, D.C.: USAID Evaluation Policy, 2011. USAID. How-To Note: Preparing Evaluation Reports. Washington, D.C.: USAID, 2012. USAID. Evaluation Report Template. Washington, D.C.: USAID. USAID. USAID Evaluation Report Requirements: A Mandatory Reference for ADS Chapter 201. Washington, D.C.: USAID, 2016. 2-4. 40 ANNEX C. DATA COLLECTION TOOLS Component A – End-line Performance Evaluation of Maternal and Child Survival Program (MCSP) Key Informant Interviews with MOHS staff Policy level staff (Director, Maternal and Reproductive Health Division, Department of Public Health (DPH); Acting Director/Director, Child Health Development Division, DPH; National Health Plan Implementing Monitoring Unit (NIMU) State and Regional Health Staff (Southern Shan State, Ayeyarwaddy Region), State and Regional Training Team Intervention Site: Kangaroo Mother’s Care-KMC): Thanlyin General Hospital, Taunggyi Women and Children Hospital, Department of Public Health and Medical Services, MOHS; L&PIC (Southern Shan State, Ayeyarwaddy Region) Informed Consent and Confidentiality: Hi, my name is X, and I work for Social Impact, which is an independent research company based in the Washington, DC area. We are collecting data about the Maternal and Child Survival Program (MCSP) program began in 2014 in Myanmar. The overall goal of the program is to reduce maternal and newborn mortality and morbidity in Myanmar and MCSP works with the MOHS and key partners to create enabling policy environment for the inclusion of MNCH best practices, improve quality and effectiveness of in-service midwifery training, assessment and performance and capacity-building interventions. MCSP program works with the MOHS to ensure that activities are in line with national priorities of improving health worker capacity to deliver the high￾quality life-saving care included in the Essential Package of Health Services (EPHS) and thereby improved health outcomes. Our evaluation is intended to inform the relevant departments in the Ministry of Health (MOHS) as well as the U.S. Government’s design of future technical assistance and institutional strengthening activities in Myanmar. We selected you and other respondents to interview because we understand that you may have perspective on the MCSP program and/or on relevant subject matter. We expect the duration of this interview to be one hour. We plan to ask you about the MCSP implementation, its achievement, lesson learned, constraint and your opinion on sustainability or future scaling up. There are no known risks or direct benefits related to your participation; however, your inputs may lead to recommendations that benefit actors engaged in MCSP program —and, thereby, the general public. All information that you share will be kept confidential. We will aggregate and present our findings to USAID in a way that cannot be attributed to any individual or organization. Therefore, please feel free to speak openly and candidly with us. Your participation is voluntary. Please feel free to ask to skip any question that you do not feel comfortable answering, end this interview at any point, or withdraw your responses after the interview. Do you confirm your consent to participate in this interview? ☐ Yes ☐ No To guarantee accuracy, we find it useful to keep an audio record of the conversation. If you prefer, however, we will not use recording devices. Do you confirm your consent for us to record this interview? ☐Yes ☐No Interview Place and Date: Interviewer(s): 41 Interviewee Name & Title: Sex: ☐ Female ☐ Male Part I: All Participants Introduction: a. What is your knowledge and understanding of the USAID funded MCSP program in Myanmar? b. What was your roles and responsibilities on the MCSP program? (Probe: how did you & in what ways were you involved? c. What would you say are the strengths and what are the weaknesses of this program (in general)? d. What worked well in this project? And what did not work very well? EQ1: To what extent did MCSP assistance influence in-service training practices and related systems to improve maternal, neonatal and child health? Probes: a. How successful and support, would you say, this program in improving capacity of the in-service training practices and related systems? Why and why not? Please explain. b. How does increased in the capacity of staff and institutional contribute for better maternal, neonatal and child health in Myanmar? In what way? c. Since the introduction of MCSP related activities in your sector, what changes have you observed? What has not changed that you would like to see changed? d. Were there any significant improvements in capacity of staff exposed to MCSP training and institutional capacity building program compared to those from non-MCSP Focus Township? Why? EQ1a: To what extent have the MCSP’s in-service capacity building activities, influenced policies, practices and the enabling environment for in-service training at different levels? To what extent have health system actors been able to apply and replicate interventions introduced by MCSP? Probes: a. How does the MCSP contribute in the institutional capacity building process and was it considered effective? Is the program reached to the intended objectives and targeted population? 38 b. How does the MCSP support in the development of guidelines and other policy to practice (operationalization of policies)? Please explain and how well are these policies being carried out? c. Any significant achievement, lesson learned, constraints and barriers based on the scope and implementation of activities? How did you overcome barriers? d. Any experience on applying or replicating what you have learned from MCSP? Please share? If not, why? 38 Note: All greyed-out questions are optional. 42 EQ1b: To what extent were MCSP approach and interventions aligned to health system realities in order to address the key barriers for strengthening in-service training at State/Region and Township levels (S/R/T)? Probes: a. What are the main barriers for strengthening in-service training at S/R/T levels? Are these addressed and by how? b. What changes do you observe in terms of midwifery capacity before and after MCSP? c. To what extent did MCSP align with current Myanmar health and do you perceive that MCSP program could address the barrier stated above? Why or why not? EQ1c: To what extent were MCSP’s interventions and program design aligned to address drivers of maternal and child mortality and morbidity? Probes: a. And in your opinion, how does increased in the capacity of staff and institutional contribute for better MNCH outcomes in Myanmar? In what way? b. How did MCSP’s address drivers of maternal and child mortality and morbidity? In what way? How did the project’s design contribute to country engagement and ownership? Probes: a. What are examples of government engagement towards this project? Do you think this contributes towards government ownership of the project design in near future? b. What aspects of the project’s design makes government to take ownership? c. To what extent does MCSP program’s approach align with national priorities and polices relating to MNCH implemented in NHP? And align with your organization/department interest? EQ2: How have MCSP’s approaches contributed to the potential sustainability of project results? Probes: a. How likely will this MCSP approach be sustained after this project ends? Why /Why not? b. What will be done to ensure that staff in MNCH sector remain up-to-date with training? EQ2a: In your opinion, what kind of activities or technical approaches will be feasible to apply/ replicate in relation to the future MOHS in-service training activities? Probes: a. To what extent have you been able to apply, and replicate interventions introduced by MCSP? b. Which elements of the project are likely to be sustained or expanded/replicated (e.g., through institutionalization or policies)? Any plan to scale-up the MCSP model and approach? Why and Why not? c. What are the issues around sustainability in capacity, funding, human resources, etc. with regard to midwife IST, L&PIC model, regulatory TA (MNMC), and association strengthening (MNMA)? d. To what extent have program activities been integrated into current activities/practices of MOHS? How can integration be enhanced? e. To what extent does MCSP program complement and synergize or overlap and duplicate with existing interventions? Any strategies or activities carried out to enhance synergies and avoid duplication? f. Were there any other ways of designing the program in a more cost-effective manner, without diminishing the quality of outputs? If yes, explain. g. How well is functioning/maintenance and possible scale-up of the L&PIC center, training team, etc.? Why /Why not, please elaborate? (S/R team only) 43 EQ2b: What are key factors/evidence that support such conclusion(s)? Probes: Important issues to listen for and probe into are: a. Comment on the training (State and Regional) activity, sustainability of training team, its capacity b. Comment on the L&PIC, KMC, QI at clinical site, support for in-service training and linkage with pre-service trainings. (Comments on the successes and challenges). c. Any suggestions and positive or negative aspects to share? What will you do differently for better outcome in future program? (S/R) EQ3: What are the specific lessons that can be learned to inform future programs that aim to strengthen systems for capacity building related to MCH, particularly at the township level? Probes: EQ3a: Any similar approaches/interventions that should/should not be supported/replicated through future assistance? Why/why not? EQ3b: Any challenges in the health system that MCSP did not address, which would need to be addressed for future programs to be successful, particularly toward affecting improvements at the township level and below? EQ3c: Any interventions/supports that should be removed or modified to better adjust interventions to health system realities? EQ3d: Any necessary modifications to the models and interventions supported by MCSP, including their mode of delivery, if future replication is considered? Final thoughts and conclusion How will you recommend the project to provide support differently in the future? Part 2: Specific Questions by Stakeholder Training related: (mainly at S/R level but probe at NPT level briefly) Probes: based on the various activities depend on the roles and responsibilities. a. What kind of training you receive from MCSP program? How long and how many times and content received? b. Could you provide feedbacks on the training received (in terms of technical and activity approach) e.g. resources, management, contents of the trainings, training curriculum and training time? Is attending training cause extra workload for you? How did you manage to cover your work when you are away for trainings? c. Any new information learned or have your skills reinforced? Did you applied the knowledge and skills learned and how? d. Any success stories to share in managing MNCH health care in your area (your own or other’s experiences) because of the MCSP activities? e. What are the main barriers to attend the trainings? What have/ can be done to improve sustainability of training team? f. In what ways has the MCSP related training impacted your capacity and up to the MNCH health status of area you cover? g. What can be done to better sustain this kind of activities? h. How likely will this (state health training team’s utilization of LPIC) be sustained after this project 44 ends? Why /Why not? i. In what ways has the MCSP support/improve state health training team’s performance till after the project? KMC unit (Respective staff from Thanlyin and Taunggyi hospital) a. What is your role and responsibilities in KMC unit? b. Can you share your experiences or feedbacks on the KMC related support via MCSP program (treatment, education, experience sharing sessions, data collection, follow-ups, etc.)? c. Can you share any significant achievement, lesson learned, and constraints encountered during implementation and documentation of KMC? How do you find about feasibility and effectiveness of it in relation to newborn health? Any success story you have to share? d. What will you do differently in future KMC facility set-up, in other areas? e. Will you recommend to continue KMC in your hospital? Why and Why not? f. Will you also recommend to practice in Township hospital and Rural Health Center? Why and Why not? g. Do you think KMC is feasible to implement in resource limited settings? L&PIC (Respective staff from L&PIC affiliated center: Ayeyarwaddy region and Southern Shan state) a. What is your roles and responsibilities in L&PIC unit setting-up, activities, functioning and sustainability? b. Can you share your experiences or feedbacks on the L&PIC (probe: training, developing sustainability plan, collaboration with MRH, CHD and officials from State and Regional Department, workshop, QI activities, etc.)? c. Do you think that resources provided for L&PIC including IMNCI and others are sufficient? Why? Any suggestion for future similar program? d. Can you share any significant achievement, lesson learned, and constraints encountered during implementation/ documentation of L&PI? Comment on feasibility and effectiveness? Any success story you have to share? e. How likely will this LPIC be sustained after this project ends? Why /Why not? f. In what ways has the MCSP supported to sustain LPIC’s performance till after the project? g. Do you think it is adequate in terms of effectiveness as well as sustainability? Why and why not? h. What will be your recommendation to provide support differently in future by project to improve the performance of the LPIC? What will you do differently in future L&PI facility set-up in other area? EHO, Central (NPT) a. How did you find about MCSP activity in EHO area and how MCSP program contributes in improving health care to ethnic and conflict affected area? (skip question if respondent did not aware of it) b. How do you find about collaboration and coordination with KDHW? Any feedbacks? What changes you would like to see? (skip question if respondent did not aware of it) c. In your opinion, is it feasible to scale up or replicate similar MCSP activity in other EHO areas in future program? What changes you would like to see? Tell me about potential strengths, constraints and barriers? 45 NHP related (National Level Staff: particularly with staff from NIMU unit only) a. To what extent do MCSP program’s approach contribute in the NHP health plan? b. To what extent did MCSP program complement/ synergize or overlap/ duplicate with NHP development and implementation? Any strategies or activities carried out to enhance synergies and avoid duplication? c. Any significant achievement, lesson learned, constraints and barriers based on the scope and implementation of activities? What worked what didn’t work and why? d. Is it likely that MCSP program achievements will be sustained after the program and if yes, how did the GoM or MOHS intend to align with NHP? e. What will be your recommendation to provide support differently in future for NHP by project? NHP related (State/Regional level) a. What extent does MCSP contribute in NHP efforts to build subnational capacity in current midwifery and IST? b. Have you observed any change in in-service capacity building, generally, and for midwives? c. To what extent have the program inputs (human, technical, and financial) been used efficiently? How and where and what improvements could have been made to improve efficiency and cost￾effectiveness without compromising quality? Are partnership arrangements organized effectively? d. What will be your recommendation to provide support differently in future for NHP by project? 46 Key Informant Interview: Regulatory Body: Myanmar Nurse and Midwife Council (MNMC) Professional Associations: Myanmar Nurse and Midwife Association (MNMA) Informed Consent and Confidentiality: Hi, my name is X, and I work for Social Impact, which is an independent research company based in the Washington, DC area. We are collecting data about the Maternal and Child Survival Program (MCSP) program began in 2014 in Myanmar. The overall goal of the program is to reduce maternal and newborn mortality and morbidity in Myanmar and MCSP works with the MOHS and key partners to create enabling policy environment for the inclusion of MNCH best practices, improve quality and effectiveness of in-service midwifery training, assessment and performance and capacity-building interventions. MCSP program works with the MOHS to ensure that activities are in line with national priorities of improving health worker capacity to deliver the high-quality life-saving care included in the Essential Package of Health Services (EPHS) and thereby improved health outcomes. Our evaluation is intended to inform the relevant departments in the Ministry of Health (MOHS) as well as the U.S. Government’s design of future technical assistance and institutional strengthening activities in Myanmar. We selected you and other respondents to interview because we understand that you may have perspective on the MCSP program and/or on relevant subject matter. We expect the duration of this interview to be one hour. We plan to ask you about the MCSP implementation, its achievement, lesson learned, constraint and your opinion on sustainability or future scaling up. There are no known risks or direct benefits related to your participation; however, your inputs may lead to recommendations that benefit actors engaged in MCSP program—and, thereby, the general public. All information that you share will be kept confidential. We will aggregate and present our findings to USAID in a way that cannot be attributed to any individual or organization. Therefore, please feel free to speak openly and candidly with us. Your participation is voluntary. Please feel free to ask to skip any question that you do not feel comfortable answering, end this interview at any point, or withdraw your responses after the interview. Do you confirm your consent to participate in this interview? ☐ Yes ☐ No To guarantee accuracy, we find it useful to keep an audio record of the conversation. If you prefer, however, we will not use recording devices. Do you confirm your consent for us to record this interview? ☐Yes ☐No Interview Place and Date: Interviewer(s): Interviewee Name & Title: Sex: ☐ Female ☐ Male Introduction: a. What is your knowledge and understanding of the USAID funded MCSP program in Myanmar? b. What was your roles and responsibilities on the MCSP program? (Probe: how did you & in what ways were you involved? c. What would you say are the strengths and what are the weaknesses of this program (in general)? d. What worked well in this project? And what did not work very well? 47 EQ1: To what extent did MCSP assistance influence in-service training practices and related systems to improve maternal, neonatal and child health? Probes: a. How successful, would you say, this program has been in strengthening institutions such as the Myanmar Nurse and Midwifery Council (MNMC) and Myanmar Nurse and Midwives Association (MNMA). Why and why not? Please explain. b. And in your opinion, how does increased institutional capacity contribute to better maternal, neonatal and child health in Myanmar? In what way? (Probe about organizational capacity, continuing professional development, licensure, MNMA master trainers, etc.) c. Since the introduction of MCSP related activities in your sector, what changes have you observed? What has not changed that you would like to see changed? d. In your opinion, were there any significant improvements in capacity of staff exposed to MCSP training and institutional capacity building program or other related activity compared to those from non-MCSP Focus Township? Why? EQ1a: To what extent have the MCSP’s in-service capacity building activities, supported policies, practices and the enabling environment for in-service training at different levels of the system? To what extent have health system actors been able to apply and replicate interventions introduced by MCSP? Probes: a. Was MCSP programs considered effective in building institutional capacity? Is the program reached to the intended objectives and targeted population? b. Tell me about MNMC role in L&PIC and CSSAC (Clinical Skills assessment Center)? To what extend these activities contribute in success of the MCSP program? Why or Why not? c. How many and what types of policies have been developed because of this capacity building program? How well are these policies being carried out? (Probe about organizational capacity, continuing professional development, licensure, MNMA master trainers, etc.) d. What are the strengths and weakness of MCSP program based on your role in it? e. Any significant achievement, lesson learned, constraints and barriers based on the scope and implementation of activities in comparing the agreed targeted results of MCSP regarding MNMC or MNMA? What worked what didn’t work and why? How did you overcome barriers? f. Collaboration and communication: i. What value-add has come of the joint statement signed between MNMA and MMA-OBGYN Society, facilitated by MCSP? What was expected this would achieve and has it? ii. Do you observe any improvement in communication or collaboration with other professional originations nationally and internationally as well as MOHS because of MCSP? What are they and any positive or negative aspects to share? What will you do differently for better outcome in future program? (Central level) L&PIC (Staff from L&PIC affiliated center: L&PIC at MNMA) a. What is your role and responsibilities in L&PIC unit setting-up, activities, functioning and sustainability? b. Can you share your experiences or feedbacks on the L&PIC related support via MCSP program (in terms of continuing professional education to members, training, developing sustainability plan, etc.)? c. Do you think that resources provided for L&PIC and others related activities are sufficient? Any comment on it and how do you plan for sustainability after the project end? Any suggestion for future similar program? d. Can you share any significant achievement, lesson learned, and constraints 48 encountered during implementation and documentation of L&PI? How do you find about feasibility and effectiveness of it in relation to the maternal and newborn health? Any success story you have to share? e. What will you do differently in future L&PI facility set-up in other area? EQ1b: To what extent were MCSP approach and interventions aligned to health system realities in order to address the key barriers for strengthening in-service training at State/Region and Township levels? Probes: a. b. What changes do you observe in terms of midwifery capacity before and after MCSP? c. What are the barriers in strengthening in-service training at State/Region and Township levels? d. Does MCSP related activities of L&PIC, CSSAC and credit point system could address the key barriers for strengthening in-service training and improve staff capacity (State/Regional/Township)?Tell me about activities such as identification of MNMA master trainers; post-training follow-up (PTFU) for trainers in terms of feasibility, appropriateness, sustainability and cascading to townships? Quality control measures for midwives? EQ1c: To what extent were MCSP’s interventions and program design aligned to address drivers of maternal and child mortality and morbidity? Probes: a. To what extent were MCSP approach and interventions aligned to health system realities b. And in your opinion, how does increased in the capacity of staff and institutional via MNMC or MNMA contribute for better MNCH outcomes in Myanmar? In what way? c. To what extent were MCSP’s interventions and program design aligned to address drivers of maternal and child mortality and morbidity? How did the project’s design contribute to country engagement and ownership? Probes: a. In your opinion, to what degree do you believe the government is now taking responsibility for this process? Please explain why or why not. b. What aspects of the project’s design supports the government to take ownership? c. How does MCSP’s program approach align with national priorities and polices relating to MNCH implemented in NHP? And align with your organization/department? EHO a. What is your perception of degree to which the EHO providers have participated in this program? b. What do you see as the prospects for further training and accreditation? c. What should be done to support WHOs to continue participation in training and accreditation? EQ2: How have MCSP’s approaches contributed to the potential sustainability of project results? Probes: a. How likely will this MCSP approach be sustained after this project ends? Why /Why not? Give reasons. b. What will be done to ensure that staff in MNCH sector remain up-to-date with training? c. In your opinion, what factors affect the sustainability of this program results (probe)? 49 d. To what extent does technical and organization development support to MNMC via MCSP contribute in sustainability of the project? Why and why not? e. What positive or negative effect do you expect for competency-based licensing testing? Any challenges and achievements and lesson learned? Feedbacks to share? EQ2a: In your opinion, what kind of activities or technical approaches will be feasible to apply/ replicate in relation to the future MOHS in-service training activities? Probes: a. We understand that there is/will be restructuring of MNMC and in what capacity will it affect future similar program? What are your perspectives on the advantages and disadvantage? b. Do you support the idea of the scaling up of MCSP model and approach? Why and Why not? What kind of activities or technical approaches will be feasible to apply/ replicate in relation to the future activities? c. Which elements of the project have been or are likely to be sustained or expanded (e.g., through institutionalization or policies)? i. What are the issues around sustainability in capacity, funding, human resources, etc. with regard to midwife IST, L&PIC model, regulatory TA (MNMC), and association strengthening (MNMA)? ii. To what extent have program activities been integrated into current activities/practices of MOHS? How can integration be enhanced? iii. Were there any other ways of designing the program in a more cost-effective manner, without diminishing the quality of outputs? If yes, explain. d. Any other factors or events affecting the quality of implementation? e. To what extent have you been able to apply and replicate interventions introduced by MCSP? EQ2b: What are key factors/evidence that support such conclusion(s)? Probes: Important issues to listen for and probe into are: a. Comments on the support received via MCSP program on institutional capacity strengthening in terms of technical and activity approach. b. Comment on the training (State and Regional) activity, sustainability of training team, its capacity, capacity building and technical assistance to MNMC (probe assessment for licensing) and MNMA (probe nurse law) c. Comment on the L&PIC, KMC, QI at clinical site, support for in-service training and linkage with pre-service trainings? d. Comments on the successes and challenges encountered. (technical and implementation) 50 EQ3: What are the specific lessons that can be learned to inform future programs that aim to strengthen systems for capacity building related to MCH, particularly at the township level? Probes: EQ3a: Any similar approaches/interventions that should/should not be supported/replicated through future assistance? Why/why not? EQ3b: Any challenges in the health system that MCSP did not address, which would need to be addressed for future programs to be successful, particularly toward affecting improvements at the township level and below? EQ3c: Any interventions/supports that should be removed or modified to better adjust interventions to health system realities? EQ3d: Any necessary modifications to the models and interventions supported by MCSP, including their mode of delivery, if future replication is considered? Final thoughts and conclusion How will you recommend the project to provide support differently in the future? 51 Key Informant Interview: Ethic Health Organization (EHO): Karen Department of Health and Welfare (KDHW) Informed Consent and Confidentiality: Hi, my name is X, and I work for Social Impact, which is an independent research company based in the Washington, DC area. We are collecting data about the Maternal and Child Survival Program (MCSP) program began in 2014 in Myanmar. The overall goal of the program is to reduce maternal and newborn mortality and morbidity in Myanmar and MCSP works with the MOHS and key partners to create enabling policy environment for the inclusion of MNCH best practices, improve quality and effectiveness of in-service midwifery training, assessment and performance and capacity-building interventions. MCSP program works with the MOHS to ensure that activities are in line with national priorities of improving health worker capacity to deliver the high-quality life-saving care included in the Essential Package of Health Services (EPHS) and thereby improved health outcomes. Our evaluation is intended to inform the relevant departments in the Ministry of Health (MOHS) as well as the U.S. Government’s design of future technical assistance and institutional strengthening activities in Myanmar. We selected you and other respondents to interview because we understand that you may have perspective on the MCSP program and/or on relevant subject matter. We expect the duration of this interview to be one hour. We plan to ask you about the MCSP implementation, its achievement, lesson learned, constraint and your opinion on sustainability or future scaling up. There are no known risks or direct benefits related to your participation; however, your inputs may lead to recommendations that benefit actors engaged in MCSP program—and, thereby, the general public. All information that you share will be kept confidential. We will aggregate and present our findings to USAID in a way that cannot be attributed to any individual or organization. Therefore, please feel free to speak openly and candidly with us. Your participation is voluntary. Please feel free to ask to skip any question that you do not feel comfortable answering, end this interview at any point, or withdraw your responses after the interview. Do you confirm your consent to participate in this interview? ☐ Yes ☐ No To guarantee accuracy, we find it useful to keep an audio record of the conversation. If you prefer, however, we will not use recording devices. Do you confirm your consent for us to record this interview? ☐Yes ☐No Interview Place and Date: Interviewer(s): Interviewee Name & Title: Sex: ☐ Female ☐ Male Introduction: a. What is your knowledge and understanding of the USAID funded MCSP program in Myanmar? b. What was your roles and responsibilities on the MCSP program? (Probe: how did you & in what ways were you involved? c. What would you say are the strengths and what are the weaknesses of this program (in general)? d. What worked well in this project? And what did not work very well? 52 EQ1: To what extent did MCSP assistance influence in-service training practices and related systems to improve maternal, neonatal and child health? Probes: a. How successful and influential has this program been in improving the quality and rigor of the in￾service training practices and related systems? Why and why not? Please explain. b. How does increased capacity of staff and the institution contribute to better maternal, neonatal and child health in your area and health system? In what way? c. Since the introduction of MCSP related activities in your sector, what changes have you observed? What has not changed that you would like to see changed? d. Were there any significant improvements in the capacity of staff exposed to MCSP training and institutional capacity building program before and after MCSP program? Why? EQ1a: To what extent has MCSP’s support contributed to the EHO health system? To what extent have health system actors in your area been able to apply and replicate interventions introduced by MCSP? Probes: a. How does the MCSP contribute in the institutional capacity building process and was it considered effective? Is the program reached to the intended objectives and targeted population? b. Could you share of your perceptions on the prospects of EHO collaboration with MOHS? c. Experience sharing and feedbacks on the MOHS’ support for training and certifying EHO providers in Basic Emergency Obstetric and Newborn Care (BEmONC)? Any improvement seen regarding BEmONC? d. What are the strengths and weakness of MCSP program based on your role in it? Any significant achievement, lesson learned, constraints and barriers based on the scope and implementation of activities? What worked what didn’t work and why? How did you overcome barriers? e. Collaboration and communication: i. How did you find support receive from central/policy level in implementing MCSP? Any suggestions and positive or negative aspects to share? What will you do differently for better outcome in future program? ii. We would like to explore about your reflections on scope for MOHS-EHO collaboration before/after the training; how does existence of L&PIC at Taw Nor help; why is MOHS training certification important. Training related: skills lab coordinator (SLC) workshop, Basic Emergency Obstetric and Newborn Care (BEmONC) Master Mentor training Probes: a. What kind of training did you receive from MCSP program? How long and how many times and content received? b. Could you provide feedback on the training received? Why are you saying it?(in terms of the resources, management, contents of the trainings, training curriculum and training time? Did you find that it is too much or too many trainings and do MCSP activities (trainings and related activities) cause extra workload for you?) c. Any new information learned or have your skills reinforced? Did you applied the knowledge and skills learned and how? d. Any success stories to share in managing MNCH health care in your area (your own or other’s experiences) because of the MCSP activities? e. What are the main barriers that prevent for you or others to attend the trainings? What can be done to improve sustainability of training team? f. In what ways has the MCSP related training impacted your capacity and up to the MNCH health 53 status of area you cover g. What can be done to better sustain this kind of activities? L&PIC a. What is your roles and responsibilities in L&PIC unit setting-up, activities, functioning and sustainability under MCSP program? b. Can you share your experiences or feedbacks on the L&PIC related support via MCSP program (in terms of training, developing sustainability plan, collaboration with MRH, CHD and officials from State and Regional Department, workshop, QI activities, etc.)? c. Do you think that resources provided for L&PIC is sufficient? Any comment on it and how do you plan for sustainability after the project end? Any suggestion for future similar program? d. Can you share any significant achievement, lesson learned, and constraints encountered during implementation and documentation of L&PI? How do you find about feasibility and effectiveness of it in relation to newborn health? Do you have any success stories to share? e. What will you do differently in future L&PI facility set-up in other area? EQ1b: To what extend MCSP programs contribute address the key barriers for strengthening staff or volunteer capacity in your area? Probes: a. What are the main barriers for strengthening capacities of staff or volunteer working in MNCH care in your area and what changes do you observe in terms of capacity before and after MCSP? EQ1c: To what extent were MCSP’s interventions and program design aligned to address drivers of maternal and child mortality and morbidity? Probes: a. And in your opinion, how does increased in the capacity of staff and institutional contribute for better MNCH outcomes in Myanmar? In what way? b. How did MCSP’s address drivers of maternal and child mortality and morbidity? In what way? EQ2: How have MCSP’s approaches contributed to the potential sustainability of project results? Probes: a. How likely will this MCSP approach be sustained after this project ends? Why /Why not? b. What will be done to ensure that staff in MNCH sector remain up-to-date with training? c. In your opinion, what factors affect the sustainability of this program results? EQ2a: In your opinion, which activities or technical approaches could be beneficial to incorporate in future capacity building training models in your area? a. Which elements of the MCSP project have been or are likely to be sustained or expanded in your area? b. What are the issues around sustainability in capacity, funding, human resources, etc. with regard to MCSP related activity in your area (for e.g.- BEmONC) c. To what extent have program activities been integrated into current activities/practices of KDHW? How can integration be enhanced? EQ2b: What are key factors/evidence that support such conclusion(s)? Probes: 54 Important issues to listen for and probe into are: a. Comments on the support received from central/IPs in terms of technical and activity approach. b. Comment on the MCSP supported activities (Probe: training, L&PIC, clinical training skills (CTS) workshops and work for standardization of EHO’s MNCH providers technical skills, competency-based training in EHO areas, pool of EHO BEmONC master mentors, etc.). c. Comments on the successes and challenges of the MCSP related activities. (technical and implementation) d. Any improvement or changes about EHO health care providers ‘s capacity in terms of clinical, facilitation skills? EQ3: What are the specific lessons that can be learned to inform future programs that aim to strengthen systems for capacity building related to MCH, particularly at the township level? Probes: EQ3a: Any similar approaches/interventions that should/should not be supported/replicated through future assistance? Why/why not? EQ3b: Any challenges in the health system that MCSP did not address, which would need to be addressed for future programs to be successful, particularly toward affecting improvements at the township level and below? EQ3c: Any interventions/supports that should be removed or modified to better adjust interventions to health system realities? EQ3d: Any necessary modifications to the models and interventions supported by MCSP, including their mode of delivery, if future replication is considered? Final thoughts and conclusion How will you recommend the project to provide support differently in the future? 55 Key Informant Interview: Implementing Partners: Jhpiego/Johns Snow/Save the Children Auxiliary Partners: 3MDG/ UNOPS Donor: Staff from USAID Informed Consent and Confidentiality: Hi, my name is X, and I work for Social Impact, which is an independent research company based in the Washington, DC area. We are collecting data about the Maternal and Child Survival Program (MCSP) program began in 2014 in Myanmar. The overall goal of the program is to reduce maternal and newborn mortality and morbidity in Myanmar and MCSP works with the MOHS and key partners to create enabling policy environment for the inclusion of MNCH best practices, improve quality and effectiveness of in-service midwifery training, assessment and performance and capacity-building interventions. MCSP program works with the MOHS to ensure that activities are in line with national priorities of improving health worker capacity to deliver the high-quality life-saving care included in the Essential Package of Health Services (EPHS) and thereby improved health outcomes. Our evaluation is intended to inform the relevant departments in the Ministry of Health (MOHS) as well as the U.S. Government’s design of future technical assistance and institutional strengthening activities in Myanmar. We selected you and other respondents to interview because we understand that you may have perspective on the MCSP program and/or on relevant subject matter. We expect the duration of this interview to be one hour. We plan to ask you about the MCSP implementation, its achievement, lesson learned, constraint and your opinion on sustainability or future scaling up. There are no known risks or direct benefits related to your participation; however, your inputs may lead to recommendations that benefit actors engaged in MCSP program —and, thereby, the general public. All information that you share will be kept confidential. We will aggregate and present our findings to USAID in a way that cannot be attributed to any individual or organization. Therefore, please feel free to speak openly and candidly with us. Your participation is voluntary. Please feel free to ask to skip any question that you do not feel comfortable answering, end this interview at any point, or withdraw your responses after the interview. Do you confirm your consent to participate in this interview? ☐ Yes ☐ No To guarantee accuracy, we find it useful to keep an audio record of the conversation. If you prefer, however, we will not use recording devices. Do you confirm your consent for us to record this interview? ☐Yes ☐No Interview Place and Date: Interviewer(s): Interviewee Name & Title: Sex: ☐ Female ☐ Male 56 KII with IPS Introduction: a. What is your knowledge and understanding of the USAID funded MCSP program in Myanmar? b. What was your roles and responsibilities on the MCSP program? (Probe: how did you & in what ways were you involved? c. What would you say are the strengths and what are the weaknesses of this program (in general)? d. What worked well in this project? And what did not work very well? e. How does implementation of MCSP in Myanmar compared to other countries or global MCSP? EQ1: To what extent did MCSP assistance influence in-service training practices and related systems to improve maternal, neonatal and child health? Probes: a. How successful, would you say, this program in improving capacity of the in-service training practices and related systems? Why and why not? Please explain. b. How does increased in the capacity of staff and institutional contribute for better maternal, neonatal and child health in Myanmar? In what way? c. Since the introduction of MCSP related activities in your sector, what changes have you observed? What has not changed that you would like to see changed? d. Were there any significant improvements in capacity of staff exposed to MCSP training and institutional capacity building program compared to those from non-MCSP Focus Township? Why? EQ1a: To what extent have the MCSP’s in-service capacity building activities, supported policies, practices and the enabling environment for in-service training at different levels of the system? To what extent have health system actors been able to apply and replicate interventions introduced by MCSP? Probes: a. How does the MCSP contribute in the institutional capacity building process and was it considered effective? Is the program reached to the intended objectives and targeted population? b. How many and what types of policies have been developed because of this capacity building program? How well are these policies being carried out? e. Any significant achievement, lesson learned, constraints and barriers based on the scope and implementation of activities? What worked what didn’t work and why? How did you overcome barriers? f. Any successes or challenges you would like to share? c. How do you find about collaboration among stakeholders? Any positive or negative aspects to share? Coordination and communications between central and State/Regional and Township? d. Any other factors or events affecting the quality of implementation? What are they? e. What are the barriers in implementing the planned activities (as per work plan)? EQ1b: To what extent MCSP programs contribute address the key barriers for strengthening in-service training at State/Region and Township levels? Probes: a. What are the main barriers for strengthening in-service training at S/R/T levels? Are these addressed and by how? What will be the most difficult and easiest barriers to overcome? b. What changes do you observe in terms of midwifery capacity before and after MCSP? 57 EQ1c: To what extent were MCSP’s interventions and program design aligned to address drivers of maternal and child mortality and morbidity? Probes: a. To what extent, do you think that MCSP design and approach could improve MNCH outcome? In what way? b. How did MCSP’s address drivers of maternal and child mortality and morbidity? In what way? How did the project’s design contribute to country engagement and ownership? Probes: a. In your opinion, to what degree do you believe the government is now taking responsibility for this process? Please explain why or why not. b. What aspects of the project’s design makes government to take ownership? c. To what extent does MCSP program’s approach align with national priorities and polices relating to MNCH implemented in NHP? And align with your organization/department? EQ2: How have MCSP’s approaches contributed to the potential sustainability of project results? Probes: a. How likely will this MCSP approach be sustained after this project ends? Why /Why not? b. To what extend MCSP activity of standardization and recognition of EHO MNCH providers has or will contribute to the potential sustainability of project results? c. In your opinion, what factors affect the sustainability of this program results (probe)? EQ2a: In your opinion, which activities or technical approaches could be beneficial to incorporate in future MOHS in-service training models? What are the issues around sustainability in capacity, funding, human resources, etc. with regard to midwife IST, L&PIC model, regulatory TA (MNMC), and association strengthening (MNMA)? Were there any other ways of designing the program in a more cost-effective manner, without diminishing the quality of outputs? If yes, explain. Do you have a proper exit strategy after MCSP and strategy for sustainability or scale-up? EQ2b: What are key factors/evidence that support such conclusion(s)? Probes: Important issues to listen for and probe into are: a. Comments on the experience on the working with different stakeholders at all levels (MOHS, EHO, Other IPs, 3MDG, Mission) during project duration. b. Comment on the training (State and Regional) activity, sustainability of training team, its capacity c. Comment on the L&PIC, KMC, QI at clinical site, support for in-service training and linkage with pre-service trainings. Comments on the successes and challenges. d. Any suggestions and positive or negative aspects to share? What will you do differently for better outcome in future program? 58 EQ3: What are the specific lessons that can be learned to inform future programs that aim to strengthen systems for capacity building related to MCH, particularly at the township level? Probes: EQ3a: Any similar approaches/interventions that should/should not be supported/replicated through future assistance? Why/why not? EQ3b: Any challenges in the health system that MCSP did not address, which would need to be addressed for future programs to be successful, particularly toward affecting improvements at the township level and below? EQ3c: Any interventions/supports that should be removed or modified to better adjust interventions to health system realities? EQ3d: Any necessary modifications to the models and interventions supported by MCSP, including their mode of delivery, if future replication is considered? Final thoughts and conclusion How will you recommend the project to provide support differently in the future? KII with Donor: Staff from USAID Introduction: a. Why did the Mission select MCSP to implement activities in Myanmar? b. What was your roles and responsibilities on the MCSP program? (Probe: how did you & in what ways were you involved? How would you describe your experience & knowledge working with the MCSP program as –administrative or technical or external? c. What would you say are the strengths and what are the weaknesses of this program (in general)? d. What worked well in this project? And what did not work very well? e. How does implementation of MCSP in Myanmar compared to other countries or global MCSP? EQ1: To what extent did MCSP assistance influence in-service training practices and related systems to improve maternal, neonatal and child health? Probes: a. How successful and influential, would you say, this program in improving capacity of the in-service training practices and related systems? Why and why not? Please explain. b. How does increased in the capacity of staff and institutional contribute for better maternal, neonatal and child health in Myanmar? In what way? c. Any significant change you observed as donor perspectives? EQ1a: To what extent have the MCSP’s in-service capacity building activities, supported policies, practices and the enabling environment for in-service training at different levels? Probes: 59 a. Any significant change you observed as donor perspectives? EQ1b: To what extent MCSP programs contribute address the key barriers for strengthening in￾service training at State/Region and Township levels (S/R/T)? Probes: a. Any significant change you observed as donor perspectives? EQ1c: To what extent were MCSP’s interventions and program design aligned to address drivers of maternal and child mortality and morbidity? Probes: a. Any significant change you observed as donor perspectives? EQ2: How have MCSP approaches contributed to the potential sustainability of project results? a. How did the project’s design contribute to country engagement and ownership? b. What interventions will likely be/not be sustained or scaled up by the government of Myanmar? c. What re the key factors/evidence that support such conclusions? EQ3: What are the specific lessons that can be learned to inform future programs that aim to strengthen systems for capacity building related to MCH, particularly at the township level? Probes: a. Any similar approaches/interventions that should/should not be supported/replicated through future assistance? Why/why not? b. Any challenges in the health system that MCSP did not address, which would need to be addressed for future programs to be successful, particularly toward affecting improvements at the township level and below? c. Any interventions/supports that should be removed or modified to better adjust interventions to health system realities? d. Any necessary modifications to the models and interventions supported by MCSP, including their mode of delivery, if future replication is considered? e. What are the greatest strengths of MCSP’s approach towards achieving expected project results? f. What are the greatest challenges associated with MCSP’s approach? How is MCSP responding to these challenges? g. What are the missed opportunities or gaps associated with the MCSP’s approach? h. Tell me about future strategy, approach and funding mechanism to improve MNCH status in Myanmar? Final thoughts and conclusion How will you recommend the project to provide support differently in the future? 60 Focus Group Discussion (FGD): State and Regional Training Team (State/Regional/District Health Department and implementation site-Hospital) Informed Consent and Confidentiality: Hi, my name is X, and I work for Social Impact, which is an independent research company based in the Washington, DC area. We are collecting data about the Maternal and Child Survival Program (MCSP) program began in 2014 in Myanmar. The overall goal of the program is to reduce maternal and newborn mortality and morbidity in Myanmar and MCSP works with the MOHS and key partners to create enabling policy environment for the inclusion of MNCH best practices, improve quality and effectiveness of in-service midwifery training, assessment and performance and capacity-building interventions.. MCSP program works with the MOHS to ensure that activities are in line with national priorities of improving health worker capacity to deliver the high-quality life-saving care included in the Essential Package of Health Services (EPHS) and thereby improved health outcomes. Our evaluation is intended to inform the relevant departments in the Ministry of Health (MOHS) as well as the U.S. Government’s design of future technical assistance and institutional strengthening activities in Myanmar. We selected you and other respondents to interview because we understand that you may have perspective on the MCSP program and/or on relevant subject matter. We expect the duration of this focus group discussion (FGD) to be 6. We plan to ask you about the MCSP implementation, its achievement, lesson learned, constraint and your opinion on sustainability or future scaling up. There are no known risks or direct benefits related to your participation; however, your inputs may lead to recommendations that benefit actors engaged in MCSP in Myanmar—and, thereby, the general public. All information that you share will be kept confidential. We try to make sure that no one knows about your participation in this discussion and especially your response in the FGD. We are asking that you keep this discussion confidential and that you do not discuss the information with others outside of this group. However, we understand that someone might still give out information to someone else, and so we cannot guarantee that everything said here will remain confidential. Again, you do not need to participate if you are not comfortable with this. We will not disclose any of your information to others or in the report and more importantly. We will not note down or record your name in FGD note and record as participant number. We will aggregate and present our findings to USAID in a way that cannot be attributed to any individual or organization. Therefore, please feel free to speak openly and candidly with us. Your participation is voluntary. Please feel free to ask to skip any question that you do not feel comfortable answering, end this interview at any point, or withdraw your responses after the interview. Do you confirm your consent to participate in this interview? ☐ Yes ☐ No To guarantee accuracy, we find it useful to keep an audio record of the conversation. If you prefer, however, we will not use recording devices. Do you confirm your consent for us to record this interview? ☐Yes ☐No Interview Place and Date: Interviewer(s): Interviewee Name & Title: Sex: ☐ Female ☐ Male 61 Introduction: Cover logistics and ground rules, for the FGD: a. Explain the study and what are going to discuss, explain ethics of the research. b. Everyone is encouraged to share their ideas, and the FGI is strengthened if everyone participates. c. There are no wrong answers, and everyone’s perspective is equally valued. d. The ideas shared during the FGI should not be shared outside the FGI with non-participants in order to respect participants’ privacy. e. Disagreements about ideas can be valuable and productive, but personal attacks will not be tolerated. 1. General a. What is your knowledge and understanding of the USAID funded MCSP program in Myanmar? b. What was your role on the project? (probe: how involved where you with the program and in what ways were you involved?) c. What would you say are the strengths and what are the weaknesses of this program? d. What worked well in this project? And what did not work very well? 2. Training related: (mainly at S/R level but probe at NPT level briefly) Probes: based on the various activities depend on the roles and responsibilities. a. What kind of training did you receive from the MCSP program? How long and how many times and content received? b. Could you provide feedback on the training received? i. What did you like about it? ii. What did you not like about it? iii. What are the main barriers that prevent you or others from attending the trainings? iv. What would you change in the future? v. Why are you saying it? c. Has there been overlap of training content or time from MCSP and others (e.g.- MOHS? NGO?) d. In what ways has the MCSP related training impacted your capacity to do your job? i. What are the significant achievements based on your role as a trainer? ii. How many times and to whom do you share your knowledge at other trainings? iii. How do you apply your learning from MCSP? iv. How does increasing in your capacity contribute for better maternal, neonatal and child health in Myanmar? In what way? e. What new information was learned? Have you applied the knowledge and skills learned and how? f. How did the MCSP program reinforce the skills you already had? g. How have the trainings affected the health status of women, newborns and children in the area you cover? i. What significant changes have you noticed? ii. Any success stories to share in managing MNCH health care in your area (your own or other’s experiences) because of the MCSP activities? EQ1: To what extent did MCSP assistance influence in-service training practices and related systems to improve maternal, neonatal and child health? a. EQ1a: How successful and influential, would you say, this program has been in improving the capacity of the in-service training practices and related systems? Why and why not? Please explain. i. Influencing policies… ii. Influencing practices… iii. Influencing an enabling environment for in-service training a all levels iv. To what extent have you been able to apply and replicate the interventions introduced by MCSP? 62 b. EQ1b: To what extent MCSP programs contribute address the key barriers for strengthening in￾service training at State/Region and Township levels (S/R/T)? c. EQ1c: To what extent were MCSP’s interventions and program design aligned to address drivers of maternal and child mortality and morbidity in your area? EQ2: How have MCSP’s approaches contributed to the potential sustainability of project results? a. What interventions will likely be/not be sustained or scaled up by the GoM? b. What can be done to improve sustainability of the training team? c. What can be done to better sustain these kinds of activities? d. What are the key factors that support your conclusions about sustainability? EQ3: What are some specific lessons that can be learned, to inform future programs, that aim to strengthen systems for capacity building related to MCH, particularly at the township level? a. Any similar approaches/interventions that should/should not be supported/replicated through future assistance (in terms of trainings)? Why/why not? b. Any challenges in the capacity building aspect of health system that MCSP did not address, which would need to be addressed for future programs to be successful, particularly toward affecting improvements at the township level and below? c. Any interventions/supports that should be removed or modified to better adjust interventions to health system realities? d. Any necessary modifications to the model and interventions supported by MCSP, including their mode of delivery, if future replication is considered? How will you rate your stratification of the trainings you received using a 5 level Likert scales (1 being low and 5 being the highest)? 1 2 3 4 5 How satisfied were you with the skill of the trainers? How satisfied were you with the content of the training? How relevant were the trainings to your job responsibilities? How likely are you to support these trainings in the future? (Note Taker, estimate the average score from respondents and mark it on the chart above.) 63 Focus Group Discussion (FGD): Mothers benefited from Kangaroo Mother Care Program ) from Thanlyin General Hospital, Taunggyi Women and Children Hospital, Department of Public Health and Medical Services, MOHS Informed Consent and Confidentiality: Hi, my name is X, and I work for Social Impact, which is an independent research company based in the Washington, DC area. We are collecting data about the Maternal and Child Survival Program (MCSP) program began in 2014 in Myanmar. The overall goal of the program is to reduce maternal and newborn mortality and morbidity in Myanmar and as part of this MCSP program, a model of Kangroo Mother care demonstration site has been established in Thanlyin General Hospital and Taunggyi Women and Children Hospital. Our evaluation is intended to inform the relevant departments in the Ministry of Health (MOHS) as well as the U.S. Government’s design of future technical assistance and institutional strengthening activities in Myanmar. We selected you and other mothers for FGD because we understand that you have received KMC service for your baby in some way. We expect the duration of this focus group discussion (FGD) to be 6. We plan to ask you about the MCSP implementation, its achievement, lesson learned, constraint and your opinion on sustainability or future scaling up. There are no known risks or direct benefits related to your participation; however, your inputs may lead to recommendations that benefit actors engaged in MCSP in Myanmar—and, thereby, the general public. All information that you share will be kept confidential. We try to make sure that no one knows about your participation in this discussion and especially your response in the FGD. We are asking that you keep this discussion confidential and that you do not discuss the information with others outside of this group. However, we understand that someone might still give out information to someone else, and so we cannot guarantee that everything said here will remain confidential. Again, you do not need to participate if you are not comfortable with this. We will not disclose any of your information to others or in the report and more importantly. We will not note down or record your name in FGD note and record as participant number. We will aggregate and present our findings to USAID in a way that cannot be attributed to any individual or organization. Therefore, please feel free to speak openly and candidly with us. Your participation is voluntary. Please feel free to ask to skip any question that you do not feel comfortable answering, end this interview at any point, or withdraw your responses after the interview. Do you confirm your consent to participate in this interview? ☐ Yes ☐ No To guarantee accuracy, we find it useful to keep an audio record of the conversation. If you prefer, however, we will not use recording devices. Do you confirm your consent for us to record this interview? ☐Yes ☐No Interview Place and Date: Interviewer(s): Interviewee Name & Title: Sex: ☐ Female ☐ Male Introduction: Cover logistics and ground rules, for the FGD: f. Explain the study and what are going to discuss, explain ethics of the research. g. Everyone is encouraged to share their ideas, and the FGI is strengthened if everyone participates. 64 h. There are no wrong answers, and everyone’s perspective is equally valued. i. The ideas shared during the FGI should not be shared outside the FGI with non-participants in order to respect participants’ privacy. j. Disagreements about ideas can be valuable and productive, but personal attacks will not be tolerated. (Prefer to have 6-10 participants, and ideal will be 6-8). KMC related e. Did you have any experience in delivering low-birth weight or pre-term infants in your last pregnancy? Tell me about it? Did you happen to aware about it before delivery? f. Which kind of service your child (low-birth weight or pre-term infant’s infants) received after delivery? Tell me about your experience in receiving KMC services. Did you receive KMC care as out-patient or in-patient? Please explain based on your understanding? (Facilitator explains about what is KMC and if respondent answers are in line with KMC service, take it as KMC even they did not able to state KMC term correctly). g. Can you share your experiences or feedbacks on the KMC related support via MCSP program? What aspects did you like the most and did you do not like the most? How do you find about health staff service at that time (probe: doctor, mid wives, admin staffed)? Give reasons. h. Do you think that KMC received is benefited for your child health, does it meet your needs? Why and why not? If yes, how do you think that KMC helped you to improve your child health? (Usefulness) i. During receiving KMC, how many hours do you practice KMC per days and is it easy to difficult to follow health staff suggestions? Why? j. Any constraints, barriers and challenges in accessing or receiving KMC service? Any barriers in terms of socio-cultural, financial and others barriers? Any experiential barrier which is any barrier / enabler directly related to the experience of practicing KMC specific to the individual (e.g.; individual barrier in practicing KMC? ) k. Any support or barriers from men or family members or community in receiving KMC? Explain.(Probe) l. Any success story you have to share? m. What kind of service will you recommend for future KMC activities if you have to hospitalize again (just for example)? n. Do you happen to share about KMC service you receive to other family members, friends or mothers? What are their response? (Value). How do you find about community acceptance for KMC? o. Will you recommend other mothers in your situation (low birth weight or pre-term) to seek health assistance from KMC affiliated hospital? (Value) p. Do you think KMC approach could be benefited to other pregnant women and mothers and should replicate in other hospital? Why and why not? q. Any suggestions to improve KMC service? What should health staff or government do in future to deliver similar program in better way? r. How will you rate your stratification of the service you received using a 5 level Likert scales (1 being low and 5 being the highest)? (Facilitator, probe each participants for their opinion and note down as a group opinion). Final thoughts and conclusion How will you recommend the project to provide support differently in the future? 65 66 ANNEX D. EVALUATION TEAM MEMBERS (COMPONENT A) Component A: Team Leader, M&E Specialist/MCH Expert, Dr. Myat (Crystal) Pan Hmone is a senior public health and development exert with specialized expertise in MCH and nutrition among others. She has conducted and is currently involved in various evaluations, research, assessments and consultancy projects, using diverse approaches in quantitative and qualitative data collection and analysis. Dr. Hmone brings demonstrated experience on evaluations in Myanmar and the MCH sector. She is currently the lead for an ongoing evaluation on the Maternal Neonatal and Child Health program (MNCH) with International Organization for Migration and previously held roles as an evaluation consultant including on a Maternal and Child Cash Transfer (MCCT) Program in Myanmar with UNOPS/LIFT/MSWRR. Some of the tasks she undertakes include conducting desk research and literature reviews; developing quantitative and qualitative research methods, sampling and tools; administering surveys, KIIs and FGDs with communities, service providers and different stakeholders; overseeing data management and analysis using statistical software; writing reports; and providing recommendations. Dr. Hmone is also extensively published in high impact, peer-reviewed journals as a first author. She holds a PhD from the University of Sydney and has double Masters of Public Health from National University of Singapore and the Institute of Medicine in Yangon, and is fluent in English and Myanmar language. Component A: Research Specialist, Dr. May Thet Kyaw is a native Burmese and Maternal Child Health (MCH) research, training, and M&E specialist with extensive knowledge of the operating environment in Myanmar. She has more than six years of experience working on MCH projects funded by governmental and non-governmental agencies in Myanmar to include USAID. Dr. Kyaw also brings solid experience evaluating MCH, in-service capacity building for health strengthening; and has experience with gender integration in evaluation/assessment design. Recently with Voluntary Services Overseas in Myanmar, Dr. Kyaw provided participatory needs research in Shan State on MCH, gender￾based violence and primary education, using participatory tools to assess community needs. With Pact Myanmar, she provided research and monitoring and evaluation services for USAID's Shae Thot MCH Project. In this capacity, she drafted and submitted monthly and quarterly project evaluation reports, analyzed project data, conducted training needs assessments for project staff and capacity development focal conflict sensitivity research. She speaks, reads and writes Myanmar language and English. 67 ANNEX E. EVALUATION DESIGN MATRIX Data Collection and Analysis Matrix, Component A Evaluation Questions Data Source Data Collection Methods Data Analysis Methods EQ1: To what extent did MCSP assistance influence in-service training practices and related systems to improve maternal, neonatal and child health? Document: MCSP work plans and reports, other MCH projects reports in country, relevant reports (NHP Operational Plan, implementation reports, routine facility data) Qualitative: KIIs with IPs (Jhpiego/John Snow/SC), MOHS representatives (NPT), State/Regional Health Departments and Training Teams, Dept. of Public Health, Township Health Department and Medical Officers from selected hospitals, Ethnic Health Organizations (EHOs) (KDHW), civil society partners, 3MDG, MNMC, MNMA and, FGDs with State/Regional Health Departments and Training Teams, trainers, mentors, etc. Qualitative- KII, FGD, Desk review, secondary data analysis ● Content analysis for identifying project successes and challenges ● Thematic organization for qualitative analysis ● Summary statistics used to assess progress against program indicators EQ 2: How have MCSP’s approaches contributed to the potential sustainability of project results? Document: Same as EQ1 Qualitative: Same as EQ1 Same as EQ1 ● Same as EQ1 EQ 3: What are the specific lessons learned to inform future programs that aim to strengthen systems for capacity building and in-service training related to MCH particularly at the township level? Document: Same as EQ1 Qualitative: Same as EQ1 Same as EQ1 ● Content analysis for identifying project successes and challenges ● Thematic organization or qualitative analysis 68 ANNEX F. INTERVIEWEE LIST STAKEHOLDER GROUP RESPONDENT NUMBER (KII) ORGANIZATIONAL AFFILIATION SEX Donor 1. USAID M 2. USAID F 3. USAID F Implementing Partner 4. John Snow Inc. F 5. Jhpiego M 6. Jhpiego F 7. Jhpiego F 8. Save the Children M 9. Jhpiego M Government (MOHS-Central) 10. DMRH F 11. NHPIU M 12. DCH F Government Staff (MOHS- State and Regional) 13. Ayeyarwady-DPH M 14. Shan-DPH M 15. Shan-DPH F 16. Shan-DPH F 17. Shan-DPH F 18. Shan-DPH F 19. Shan-DPH M 20. CWH, Taunggyi F 21. CWH, Taunggyi M 22. Thanlyin General Hospital F 69 Professional Body 23. MNMC F 24. MNMA F Ethnic Health Organization 25. KDHW F 26. KDHW M 27. KDHW F STAKEHOLDER GROUP RESPONDENT NUMBER ORGANIZATIONAL AFFILIATION SEX FGD1: Professional Body 1. MNMC F 2. MNMC F 3. MNMC F 4. MNMC F FGD 2: Professional Body 5. MNMA F 6. MNMA F 7. MNMA F 8. MNMA F 9. MNMA F 10. MNMA F FGD 3: Government Staff (MOHS- State and Regional) 11. Shan State-DPH F 12. Shan State-DPH F 13. Shan State-DPH F FGD 4: Government Staff (MOHS- State and Regional) 14. Shan State- Township DPH F 15. Shan State- Township DPH F 16. Shan State- Township DPH F 17. Shan State- Township DPH F 18. Shan State- Township DPH F 70 FGD 5: Government Staff (MOHS- State and Regional) 19. Shan State- Township DPH F 20. Shan State- Township DPH F 21. Shan State- Township DPH F 22. Shan State- Township DPH F 23. Shan State- Township DPH F FGD 6: Government Staff (MOHS- State and Regional) 24. WCH, Taunggyi F 25. WCH, Taunggyi F 26. WCH, Taunggyi F FGD 7: Government Staff (MOHS- State and Regional) 27. Thanlyin General Hospital F 28. Thanlyin General Hospital F 29. Thanlyin General Hospital F 30. Thanlyin General Hospital F 31. Thanlyin General Hospital F 71 ANNEX G. DISCLOSURE OF CONFLICT OF INTEREST 72 73 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523