USAID/GLOBAL HEALTH EVALUATION AND LEARNING SUPPORT ACTIVITY (GH EvaLS) Evaluation Report: 024 MOMENTUM MID-TERM PROCESS EVALUATION, 2022 November 1, 2022 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by ME&A, Inc., its subcontractor Dexis Consulting Group, and the evaluation team comprised of Tim A. Clary, Pamela Putney, and Bambang Heryanto. USAID Contract No. GS-10F-154BA/7200AA20M00003 Period of Performance: March–October 2022 Submitted by: Andrea Camoens, Chief of Party GH EvaLS 4350 East-West Highway, Suite 210 Bethesda, MD 20814-4410 Tel: (301) 652-4334 Email: acamoens@engl.com Cover Photo: MOMENTUM Integrated Health Resilience South Sudan. Wilma Keji, 24-years-old, holds baby Susan, 5 months old, while arranging the dress of her child on the 7th of June 2021, in Gurei PHCC supported by MOMENTUM Integrated Health Resilience, near Juba, South Sudan. IMA World Health / Adrienne Surprenant. DISCLAIMER The views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document is available online. Online documents can be located in the GH EvaLS website at https://ghevals.meandahq.com. Documents are also made available through the Development Experience Clearinghouse (http://dec.usaid.gov). Additional information can be obtained from: Global Health EvaLS 4350 East-West Highway, Suite 210 Bethesda, MD 20814-4410 Tel: (301) 652-4334 Website: https://ghevals.meandahq.com GH EvaLS MOMENTUM Mid-Term Process Evaluation / i ABSTRACT This mid-term process evaluation was commissioned by the U.S. Agency for International Development (USAID) Bureau for Global Health’s Office of Maternal and Child Health and Nutrition and the Office of Population and Reproductive Health. Its goal was to investigate the costs and benefits of the Moving Integrated, Quality Maternal, Newborn, and Child Health and Family Planning and Reproductive Health Services to Scale (MOMENTUM) project’s processes, working groups, and systems, and how those affect program implementation. It reviews MOMENTUM’s design hypotheses and assumptions in order to document how they have evolved due to the realities of the program environment, and it identifies potential risks to achieving MOMENTUM’s vision. This review does not focus on assessing the impact of implementation on health outcomes or on specific technical interventions. The MOMENTUM Suite of Awards consists of six flagship service delivery cooperative agreements awarded between December 2019 and September 2020. Through MOMENTUM, USAID seeks to accelerate reductions in maternal, newborn, and child mortality and morbidity in high-burden USAID partner countries. Each MOMENTUM award offers a deep understanding of different settings and approaches and is intended to work in a coordinated way across the Suite to provide specialized technical and regional expertise. The evaluation utilized an extensive desk review, key informant interviews with 89 MOMENTUM stakeholders, and an online survey as sources of evidence. While recognizing the limitations of a primarily qualitative review, the evidence indicates that MOMENTUM has succeeded in being responsive to partner needs, providing technical leadership, and adapting to the COVID-19 pandemic. Its overall approach and the idea of a Suite of awards premised on coordination, collaboration, and shared learning, though, is still a work in progress. It should be noted that the USAID Missions, which MOMENTUM is intended to serve, do not consider the functioning of MOMENTUM as a Suite one of their higher priorities. Rather, they have a greater interest in ensuring specific activities are undertaken and targets are met. Designing and implementing a mechanism such as MOMENTUM entailed some risks and missteps, especially regarding the management burden, partnering arrangements, and expectations for how MOMENTUM could clearly communicate its story as a project. In addition, although MOMENTUM quickly adjusted its activities in response, the COVID-19 pandemic undermined the sense of cohesion and collaboration among the multitude of partners at both the headquarters and the country levels. Nevertheless, there is still time for MOMENTUM to solidify this interconnected approach, as most of its awards are at or near the mid-point of implementation. Thus, recommendations are offered to strengthen how MOMENTUM can enhance its partnering opportunities; streamline its strategic communications; and ensure that monitoring, evaluation, and learning reinforce MOMENTUM’s coordinated approach. GH EvaLS MOMENTUM Mid-Term Process Evaluation / ii ACKNOWLEDGEMENTS This evaluation benefited greatly from the contributions of dozens of individuals who provided documents and perspectives on the MOMENTUM Suite of Awards and the six activities as part of the Suite. The evaluation team is grateful for the participation of the teams from the offices of Population and Reproductive Health (PRH) and Maternal and Child Health and Nutrition (MCHN) — particularly for the consistent leadership and timely assistance of Benjamin Isquith, Rebecca Levine, Shawn Malarcher, and Barbara Rawlins, whose knowledge and management support have been essential to this endeavor. The evaluation was conducted while the global COVID-19 pandemic was ongoing. The authors are especially indebted to the 89 USAID, implementing partner, and global technical Agency staff in the United States and multiple countries who, under challenging circumstances, gave their time to participate in the key informant interviews and group discussions that constituted a principal data source for this evaluation. We hope that it does justice to all the stakeholders from the PRH and MCHN health teams which undertook both an innovative and a challenging approach to improving global health. GH EvaLS MOMENTUM Mid-Term Process Evaluation / iii TABLE OF CONTENTS ABSTRACT .................................................................................................................................... I ACKNOWLEDGEMENTS ...........................................................................................................II ACRONYMS AND ABBREVIATIONS ....................................................................................... V EXECUTIVE SUMMARY.............................................................................................................. 7 1. EVALUATION PURPOSE AND EVALUATION QUESTIONS........................................ 14 2. BACKGROUND...................................................................................................................... 17 3. ASSIGNMENT METHODS AND LIMITATIONS ............................................................... 23 4. FINDINGS ............................................................................................................................... 32 5. CONCLUSIONS ..................................................................................................................... 54 6. RECOMMENDATIONS.......................................................................................................... 57 ANNEX 1: SCOPE OF WORK.................................................................................................. 60 ANNEX 2: ASSIGNMENT DESIGN MATRIX....................................................................... 104 ANNEX 3: INDONESIA COUNTRY CASE STUDY ............................................................ 107 3A: Indonesia Case Study — List of Documents Reviewed............................................... 116 3B: Indonesia Case Study — Profile of Respondents ......................................................... 117 3C: Indonesia Case Study — Translation of Roles and Functions..................................... 118 3D: Indonesia Case Study — Health Facility Participation ............................................... 119 ANNEX 4: DATA COLLECTION TOOLS ............................................................................ 121 ANNEX 5: SOURCES OF INFORMATION........................................................................... 152 5A: List of Persons Interviewed ........................................................................................... 152 5B: Bibliography of Documents Reviewed .......................................................................... 155 ANNEX 6: SURVEY GRAPHICS............................................................................................. 158 ANNEX 7: DISCLOSURE OF ANY CONFLICTS OF INTEREST ...................................... 166 ANNEX 8: EVALUATION TEAM MEMBERS ....................................................................... 169 GH EvaLS MOMENTUM Mid-Term Process Evaluation / iv LIST OF FIGURES Figure 1: Locations of MOMENTUM Implementation............................................................ 18 Figure 2: MOMENTUM’s Theory of Change............................................................................ 22 Figure 3: Survey Respondents’ Locations ................................................................................. 26 Figure 4: Survey Respondents’ Affiliation with MOMENTUM ............................................... 26 Figure 5: Level of Effort Needed to Manage and Access the MOMENTUM Awards ........... 43 Figure 6: The Utility of Grouping MOMENTUM Under One Mechanism ............................ 47 Figure 7: Unmet Technical Needs Within MOMENTUM....................................................... 48 Figure 8: Survey Respondents’ Satisfaction with MOMENTUM ............................................ 49 Figure 9: MOMENTUM’s Responsiveness by Survey Respondents........................................ 49 Figure 10: MOMENTUM’s Contribution to USAID’s Global Technical Leadership Role.... 51 Figure 11: Global Partners’ Responses to MOMENTUM’s Technical Breadth..................... 52 LIST OF TABLES Table 1: Evaluation Questions for MOMENTUM .................................................................... 14 Table 2: Overview of the MOMENTUM Suite of Awards ....................................................... 17 Table 3: List of all MOMENTUM Awards, IPs, and Sub-IPs.................................................... 19 Table 4: MOMENTUM Coordination Mechanisms.................................................................. 21 Table 5: Unique Number of KIIs and Their Stakeholder Affiliations..................................... 25 Table 6: Initial Criteria for Country Selection (KIIs and Survey)........................................... 27 Table 7: ADS 201 Guidance on Learning.................................................................................. 39 GH EvaLS MOMENTUM Mid-Term Process Evaluation / v ACRONYMS AND ABBREVIATIONS Acronym Definition ADS Automated Directives System AMPSR Audit Maternal Perinatal Surveillance and Response Section, Indonesia Ministry of Health AOR Agreement Officer Representative APS Annual Program Statement BGH Bureau for Global Health BMGF Bill &and Melinda Gates Foundation BPJS Social Security Administrator for Health (Indonesia) DHO District Health Office DTO Digital Transformation Office EMAS Expanding Maternal and Newborn Survival Project EQ Evaluation question FBO Faith-based organization FCR Findings, Conclusions, and Recommendations FP Family Planning GH EvaLS Global Health Evaluation and Learning Support Activity GKIA Directorate of Nutrition, and Maternal and Child Health, Indonesia Ministry of Health GOI Government of Indonesia HQ Headquarters IP Implementing Partner IPC Infection Prevention and Control KI Key Informant KII Key Informant Interview KM Knowledge Management LOE Level of effort M&E Monitoring and Evaluation MCGL MOMENTUM Country and Global Leadership MCH Maternal and Child Health MCHN Maternal and Child Health and Nutrition MEL Monitoring, Evaluation, and Learning MIHR MOMENTUM Integrated Health Resilience MKA MOMENTUM Knowledge Accelerator MMR Maternal Mortality Ratio MNCH Maternal, Newborn, and Child Health MNH Maternal and Newborn Health GH EvaLS MOMENTUM Mid-Term Process Evaluation / vi Acronym Definition MOH Ministry of Health MOMENTUM Moving Integrated, Quality Maternal, Newborn, and Child Health and Family Planning and Reproductive Health Services to Scale Project MPDN Maternal Perinatal Death Notification MPHD MOMENTUM Private Health Care Delivery MPK Directorate of Quality of Health Services Indonesia Ministry of Health M-RITE MOMENTUM Routine Immunization Transformation and Equity MSSFPO MOMENTUM Safe Surgery in Family Planning and Obstetrics NMR Neonatal Mortality Rate NTT Nusa Tenggara Timur Province, Indonesia PHC Primary Health Care PMB Private Midwife Practice POCQI Point of Care Quality Improvement, Indonesia Ministry of Health PRH Population and Reproductive Health PUSDATIN Centre of Data and Information (Indonesia) RH Reproductive Health SC Strategic communications SIMATNEO Hospital Self-Assessment Instrument SISRUTE Indonesian Integrated Referral System SISRUTE Integrated Referral System, Indonesia Ministry of Health SOW Scope of Work TKPK Governance of Health Services, Indonesia Ministry of Health USAID United States Agency for International Development WASH Water, Sanitation, and Hygiene WHO World Health Organization GH EvaLS MOMENTUM Mid-Term Process Evaluation / 7 EXECUTIVE SUMMARY EVALUATION BACKGROUND This mid-term process evaluation was commissioned by USAID’s Bureau for Global Health (GH) Office of Maternal and Child Health and Nutrition (MCHN) and the Office of Population and Reproductive Health (PRH). Its purpose was to examine the costs and benefits of the processes, working groups, and systems of the Moving Integrated, Quality Maternal, Newborn, and Child Health and Family Planning and Reproductive Health Services to Scale (MOMENTUM) Suite of Awards and how these affected program implementation. It reviewed MOMENTUM’s design hypothesis and assumptions to document how they have evolved due to new realities of the program environment (e.g., the COVID-19 pandemic) and identify potential risks for achieving MOMENTUM’s vision. Recommendations from this evaluation are to inform adaptations to MOMENTUM’s processes and structures, as well as future management decisions and designs. USAID’s vision for MOMENTUM was that it would be a group of distinct but interrelated awards, all of which collectively contribute to a common purpose to accelerate reductions in maternal, newborn, and child mortality and morbidity by increasing and leveraging existing capacity in USAID partner countries. Each of the six awards was designed to focus on supporting countries in a particular phase of the development continuum or in priority technical areas that address emerging global priorities or tackle entrenched obstacles. Further, it was expected that the awards would create opportunities to collaborate on synergistic activities for greater impact and the ability to leverage the expertise, experience, and knowledge of the full Suite of consortium partners. The vision for MOMENTUM was predicated on several assumptions, including: • Awards with specific expertise and experience can provide countries with more tailored and targeted support, resulting in more effective programming; • An explicit mandate to coordinate, collaborate, and share learning across awards will help foster cross-fertilization of ideas and amplify the breadth of experiences and best practices in the field; • A suite of awards will support the ability to tell a holistic and persuasive story to various stakeholders including Congress, about USAID’s investment in preventing child and maternal deaths; • Distinct awards that all work toward the same ultimate results and purpose will have greater overall impact — in essence, the whole is greater than the sum of its parts; • Integrated programming across the PRH and MCHN offices, with adequate coordination and collaboration, can reduce duplication of effort and increase complementarity in program implementation; and • Structuring GH programs similar to Mission programs will allow central mechanisms to be more responsive to Mission needs. GH EvaLS MOMENTUM Mid-Term Process Evaluation / 8 EVALUATION QUESTIONS The following four evaluation questions, most of which included several sub-questions, defined and structured the evaluation. They focus on the thematic areas of design, adaptation, stakeholder satisfaction, and future trends. 1. Have the awards been able to leverage their relationships with other MOMENTUM awards to benefit their work, e.g., to improve the efficiency, effectiveness, and/or scale of programming? If so, how? 2. What operational shocks and stressors are driving programmatic adaptations? 3. Does the MOMENTUM Suite address the needs and expectations of implementing partner (IP) individual awards, USAID Missions, MOMENTUM Agreement Officer Representative (AOR) management teams, Technical Offices, and other Agency leadership? 4. What are the emerging trends and needs in family planning (FP)/reproductive health (RH)/maternal, newborn, and child health that might be addressed in the current or future awards? These questions reflect the evaluation’s focus on MOMENTUM’s processes rather than its performance in order to derive findings, conclusions, and recommendations. In other words, this evaluation does not examine whether MOMENTUM or any of the individual awards have or have not reached specific targets. EVALUATION METHODS AND LIMITATIONS As the initial Annual Program Statement (APS) for MOMENTUM was issued in May 2019 — more than three years ago — and the design of MOMENTUM occurred even earlier, the evaluation relied on both current and retrospective perspectives to elicit and analyze the reasons for and consequences of the choices and processes observed. The evaluation team reviewed MOMENTUM’s Theory of Change, its vision and assumptions, and the Evaluation Questions to develop the evaluation’s approach, specific methods, and evidence collection tools. The evaluation methods were primarily qualitative, although one method, an electronic survey, collected some limited quantitative evidence. The methods included a review of MOMENTUM documentation, key informant interviews (KIIs) or group interviews (GIs) with a purposive sample of individuals knowledgeable about MOMENTUM at the global and country levels, an online survey, and a country case study (Indonesia). Data collection instruments included tailored KII guides and data extraction templates for document and KII review. These were shared among the evaluation team. Through an iterative review and discussion, and by triangulating findings across sources, the team arrived at the assessment’s findings, conclusions, and recommendations. It should be noted, though, that because of low sample sizes in response to some of the survey questions, as well as greater depth of information provided through the KIIs, the evaluation team gave more weight to the information obtained via interviews. GH EvaLS MOMENTUM Mid-Term Process Evaluation / 9 A number of limitations were inherent in the assessment. These included the overall qualitative approach; potential bias in the sample of participants; response, recall, and feedback biases; and the small samples for some survey questions. With these factors in mind, the evaluation team used the various data sources to mitigate the limitations and created a system to rate the strength of evidence (green, yellow/orange, and red) for each finding.1 FINDINGS The findings presented below follow on and correspond directly with the four major evaluation questions and their sub-questions. Evaluation Question Findings Strength of Evidence 1 The main contributing factors for effective leveraging of relationships at the country level are the formal division and clarity of roles. There were few examples cited by the key informants (KIs) of leveraging relationships to improve the scale of programming thus far at either the headquarters or country level. In general, as mentioned by KIs working at the global level, the awards are not functioning as a Suite, and collaboration occurs primarily on an as￾needed basis based on varying perceptions regarding when that collaboration is needed. At the country level, many in-country KIs do not perceive MOMENTUM as a singular project. The most notable achievements of the MOMENTUM Knowledge Accelerator (MKA) have been the MOMENTUM HUB (its internal knowledge management/KM platform) and the Share Fairs. Perceived benefits of MKA’s strategic communications engagement vary based on awards’ communications capacity. A major barrier that many KIs cited is that MKA presents an additional layer of bureaucracy for approvals. The major benefits have been around particular technical areas (e.g., immunizations, public/private sector linkages) when there is a clear scope and mandate to interact, specifically at the country level. 1 Ratings of strength of evidence are as follows. Green indicates that the finding is supported by multiple data sources and little to no contradictory evidence was found. Yellow indicates that the finding is supported by multiple data sources, although not all information may be completely aligned. Red indicates that the finding is worth mentioning (insightful, useful, etc.) but is supported by few data sources and/or may have strong contradictory responses. GH EvaLS MOMENTUM Mid-Term Process Evaluation / 10 Evaluation Question Findings Strength of Evidence The major cost, as described by KIs, is the level of coordination. Local technical leadership at the subnational level has been enhanced and strengthened in some countries. Feedback was mixed regarding whether the MOMENTUM design has been responsive to the needs of health ministries and/or partners. There is limited evidence to date that the awards have created new partnerships and ways of working within the consortium. Feedback from most KIs focused on USAID needing first to structure itself internally to provide guidance and instruction as to how coordination should take place. 2 The key shock or stressor has been the COVID-19 pandemic. The MOMENTUM Integrated Health Resilience (MIHR) award, by its nature, needs to be discussed separately from the other MOMENTUM awards when examining shocks and stressors. The MOMENTUM Routine Immunization Transformation and Equity (M￾RITE) award faced the unexpected task of contributing to mitigating the impact of COVID-19. The individual awards and the MOMENTUM Suite collectively have made an outstanding response to the COVID-19 pandemic. There also have been some challenges to the MOMENTUM Suite due to adaptations to the COVID-19 pandemic. At the headquarters level, overstretched IP staff do not have sufficient time to devote to learning activities. At the country level, there is limited evidence of shared learning. Potential solutions for increasing shared learning were offered, and the COVID-19 pandemic has presented both challenges and opportunities for learning. Most of the technical components that comprise MOMENTUM’s Theory of Change do not need adjustment. No generalizable statements can be made for adaptations of the Theory of Change at the country/Mission level. Per KIs, a stronger vision and guidance are needed from USAID in terms of coordination and collaboration along with additional light “touchpoints.” GH EvaLS MOMENTUM Mid-Term Process Evaluation / 11 Evaluation Question Findings Strength of Evidence Because MIHR works in fragile settings and emphasizes humanitarian￾development interventions, its ability to align with other awards has been especially limited. In general, the staggered timing of the awards and their start-ups significantly and negatively affected the MOMENTUM Suite. Missions were largely unaffected by the staggered timing (of the awards). 3 Technically, MOMENTUM generally meets expectations. Strategically, MOMENTUM generally does not meet expectations. There are, perhaps, some gaps in technical and functional skills within the MOMENTUM Suite; however, those gaps may be country- and stakeholder-specific. There was a mixed level of knowledge and understanding among KIs as to where certain technical and functional skills are within the Suite and how to access those skills. MOMENTUM neither inhibits nor facilitates meeting USAID (USAID/W and Missions) and individual award needs, as it is not functioning as a Suite. The overlaps in MOMENTUM award functions require clarification. It is too early to determine whether MOMENTUM addresses the needs of global partners. To date, global partners have had limited engagement with MOMENTUM. Per global partners, there are both technical and learning gaps within MOMENTUM. 4 There was no clear consensus regarding emerging trends and needs in FP/RH/MNCH that current or future awards might address. CONCLUSIONS The evidence gathered by the evaluation team indicated that MOMENTUM has succeeded in ensuring that USAID Missions are technically supported by the six awards. Further, MOMENTUM has shown its adaptability both as a Suite and individual awards by responding to the COVID-19 pandemic by modifying its internal processes and how activities were implemented. It has brought new ideas forth in how technical assistance can be provided, and it is well-aligned with the needs of its stakeholders. What is less certain is whether MOMENTUM is truly functioning as an interrelated Suite of awards that contribute collectively to a common purpose through a coordinated and collaborative approach with the whole being greater than the sum of its parts. GH EvaLS MOMENTUM Mid-Term Process Evaluation / 12 RECOMMENDATIONS The evaluation team offers four high-priority recommendations derived from the findings that should receive immediate consideration. They are (in brief): 1. Develop explicit internal and external guidance for coordination, collaboration, and partnering. 2. Review and revise the MOMENTUM monitoring, evaluation, and learning (MEL) framework(s) specifically to provide measurable incentives for coordination and collaboration. 3. Streamline strategic communications, especially the guidance document and approval process. 4. Provide additional learning opportunities, including those outside of MKA’s responsibilities. Six additional recommendations are presented for consideration, based on both USAID and IP interest and bandwidth to implement them. They are (in brief): 1. Increase publicity and dissemination opportunities for COVID-19 lessons learned. 2. Name a full-time USAID MOMENTUM coordinator who would be appointed to neither the MCHN nor PRH offices. 3. Consider MIHR- and M-RITE-specific adjustments to the Theory of Change and MEL Framework. Thematic Conclusions Design. MOMENTUM is not currently functioning as a Suite as the MOMENTUM Theory of Change envisioned, and there is a lack of clarity regarding how the awards are supposed to work together. However, there are some examples, primarily at the country level, around specific technical issues that can provide some guidance as to how better coordination and collaboration can occur. Adaptation. MOMENTUM has, for the most part, adapted exceptionally well to the COVID-19 pandemic during its implementation, which was the primary shock and stressor that it faced. There are some inherent limitations to MOMENTUM’s adaptability, though, such as working in countries that are very diverse on the development continuum, unclear perceptions of in-country stakeholders about MOMENTUM’s vision, and how well USAID Missions are aligning their individual MOMENTUM theories of change within the greater framework. Stakeholder satisfaction. Technically, for the most part, MOMENTUM is satisfying USAID Mission needs, and Missions are the primary users of its services via buy-ins. Strategically, MOMENTUM is not meeting its initial expectations to function as a Suite. However, given that this approach is of less importance to USAID Missions, it will most likely affect neither MOMENTUM’s overall performance nor the end users’ satisfaction with the mechanism. Future trends. No clear consensus emerged as to what future priority trends USAID should address in future FP/RH/MNCH designs. Some recurrent ideas put forward included a focus on PHC, mitigating climate change impacts, global health security, and effectively utilizing digital tools to enhance partner capacity. GH EvaLS MOMENTUM Mid-Term Process Evaluation / 13 4. Determine whether having Mission-level theories of change is in MOMENTUM’s strategic interests. 5. Continue outreach to Missions and partners to ensure clarity on the MOMENTUM Suite and individual awards. 6. Prioritize primary health care (PHC) and climate change for possible future investments/awards for PRH and MCHN. GH EvaLS MOMENTUM Mid-Term Process Evaluation / 14 1. EVALUATION PURPOSE AND EVALUATION QUESTIONS 1.1. EVALUATION PURPOSE The purpose of this evaluation is to investigate the costs and benefits2 of MOMENTUM’s3 processes, working groups, and systems, and how these affect programs implementation. It reviews MOMENTUM’s design hypotheses and assumptions in order to document how they have evolved due to new realities of the program environment, including the COVID-19 pandemic, and it identifies potential risks to achieving MOMENTUM’s vision. Recommendations from this evaluation will inform adaptations to MOMENTUM’s processes and structures, as well as future program designs. This is a process evaluation rather than a performance evaluation; therefore, this review does not focus on assessing the impact of implementation on health outcomes to date or on specific technical interventions. Rather, the focus is on assessing how program operations, management, and systems across the MOMENTUM Suite affect resources and outputs. 1.2. EVALUATION QUESTIONS4 The evaluation questions address four main themes: design hypothesis, adaptation, stakeholder satisfaction, and future trends (see Table 1: Evaluation Questions for MOMENTUM). Where needed, the evaluation team has modified and/or interpreted the evaluation questions to ensure a shared understanding of terminology and definitions (original Evaluation Questions are given in Annex 1: Scope of Work). Table 1: Evaluation Questions for MOMENTUM 2 Costs and benefits are limited here to administrative and managerial measures (level of effort, communications, meetings, documents developed, etc.). 3 MOMENTUM, per the original 2019 APS, is the Moving Integrated, Quality Maternal, Newborn, and Child Health and Family Planning and Reproductive Health Services to Scale project. For the sake of brevity, the term MOMENTUM is used throughout the report. 4 The evaluation team modified and expanded the Evaluation Questions included in the original Scope of Work (SOW) based on consultations with USAID (the SOW is provided in Annex 1). Main Question Sub-Questions Design 1. Have the awards been able to leverage their relationships with other MOMENTUM awards to benefit their work, e.g., to improve the efficiency, effectiveness, and/or scale of programming? If so, how? 1.A. What has been the cost and benefit (to USAID and IPs) of coordinating with the MOMENTUM MKA and across the Suite of awards? What have been the barriers and facilitators to achieving this? 1.B. What has been the cost and benefit of awards coordinating with each other (exclusive of MKA)? 1.C. What has been the cost and benefit of in-country coordination (to USAID, to IPs, to country partners)? GH EvaLS MOMENTUM Mid-Term Process Evaluation / 15 Main Question Sub-Questions 1.D. Have awards been able to create new partnerships and ways of working within the consortium? 1.E. What adaptations (e.g., changes in systems/processes) would lead to more efficient and/or more effective coordination? Adaptation 2. What operational shocks and stressors are driving programmatic adaptations? 2.A. How has the Suite collectively, and as individual awards, adapted to the realities of working in dynamic country contexts, particularly the ongoing effects of the COVID-19 pandemic? 2.B. What adaptations (e.g., changes in systems/processes) would lead to more efficient and/or more effective shared learning? 2.C. Do any of these adaptations require adjustments to the development hypothesis and/or key assumptions of the MOMENTUM Award Suite and/or implementation at the country or global level? 2.D. Are additional adjustments needed to achieve the vision for this Suite of Awards? If so, what? 2.E. How has the staggered timing of the awards start-up affected coordination and collaboration across the awards, with MKA and with Missions? Stakeholder Satisfaction 3. Does the MOMENTUM Suite address the needs and expectations of IP individual awards, USAID Missions, MOMENTUM AOR management teams, Technical Offices, and other Agency leadership? 3.A. Are there technical or functional skills not available or accessible in the current configuration? 3.B. Does the Suite design inhibit or facilitate meeting USAID’s and individual award needs? 3.C. Does the MOMENTUM Suite address key needs of global partners (e.g., Bill & Melinda Gates Foundation/BMGF, UNICEF, World Health Organization/WHO) through its role in global forums and partnerships? How has the design of the Suite helped or hindered this? Future Trends 4. What are the emerging trends and needs in FP/RH/ MNCH that might be addressed in the current or future awards? GH EvaLS MOMENTUM Mid-Term Process Evaluation / 16 1.3. EVALUATION AUDIENCES There are multiple audiences for this evaluation. The primary audiences for the evaluation are the USAID MOMENTUM AOR teams and MOMENTUM IPs. Additionally, it is anticipated that this evaluation will be of interest to USAID Mission-based activity managers and ministries of health (MOH) in countries with MOMENTUM field support buy-ins and/or countries receiving core support, including, but not limited to, the proposed country case study (Indonesia). Global partners/thought leaders in PRH and MCHN outside of USAID may also be interested in the evaluation’s outputs. Finally, participating stakeholders, either KIs or survey participants, may be interested in receiving the evaluation, as their participation demonstrates specific interest in MOMENTUM and its achievement and challenges. The evaluation team anticipates that recommendations will be of primary benefit to the offices of PRH and MCHN to inform current award implementation and future designs. GH EvaLS MOMENTUM Mid-Term Process Evaluation / 17 2. BACKGROUND The PRH and MCHN offices’ flagship service delivery projects, under the MOMENTUM Suite of Awards, were awarded between December 2019 and September 2020 as a set of six five-year cooperative agreements. Through MOMENTUM, USAID seeks to accelerate reductions in maternal, newborn, and child mortality and morbidity in high-burden USAID partner countries. The project represents a paradigm shift in how USAID carries out its work, with a greater focus on strengthening the capacity, sustainability, and resilience of local institutions. Table 2: Overview of the MOMENTUM Suite of Awards is an overview of the six awards. Table 2: Overview of the MOMENTUM Suite of Awards MOMENTUM Award Focus Funding Ceiling Start Date / End Date Country and Global Leadership (MCGL) Providing targeted technical and capacity development assistance to Missions, partner countries, and local organizations; contributing to global technical leadership and policy dialogue for maternal, newborn, and child health (MNCH) voluntary FP, and reproductive outcomes. $200,000,000 December 27, 2019– December 26, 2024 MIHR Addressing the impacts of fragility on the health and well-being of women, newborns, and children, whether triggered by political instability, conflict, violence, or natural disasters such as floods and earthquakes. $199,999,472 May 22, 2020–May 21, 2025 MKA Supports monitoring and evaluation (M&E), KM, strategic communications, and the translation of learning across the Suite of Awards. $29,670,166 December 31, 2019– December 30, 2024 MOMENTUM Private Health Care Delivery (MPHD) Ensuring increased availability of and demand for evidence-based, quality voluntary FP care and MNCH information, services, and supplies through private providers and outlets; and expanding public and private partnerships to ensure a total market approach for improved health care coverage. $75,000,000 June 22, 2020–June 21, 2025 M-RITE Sustainable strengthening of routine immunization programs to overcome entrenched obstacles contributing to declining immunization rates, and addressing barriers to reaching zero-dose and under-immunized children with life￾saving vaccines and other health services. $120,000,000 July 27, 2020– July 26, 2025 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 18 MOMENTUM Award Focus Funding Ceiling Start Date / End Date Safe Surgery in Family Planning and Obstetrics (MSSFPO) Promoting awareness of and equitable access to high-quality of care for voluntary and consented safer surgeries within maternal health and FP programs. $40,000,000 September 21, 2020– September 20, 2025 As shown on its website,5 MOMENTUM works in the following countries (see Figure 1: Locations of MOMENTUM Implementation): Bangladesh, Benin, Bosnia and Herzegovina, Burkina Faso, Burundi, Cameroon, Côte d’Ivoire, the Democratic Republic of the Congo, Ghana, Guinea, Haiti, India, Indonesia, Kenya, Kyrgyzstan, Madagascar, Malawi, Mali, Moldova, Mozambique, Nepal, Niger, Nigeria, Pakistan, the Philippines, Rwanda, Senegal, Serbia, Sierra Leone, Sudan, South Sudan, Tanzania, Togo, Uganda, Vietnam, and Zambia. Figure 1: Locations of MOMENTUM Implementation6 Number of MOMENTUM Projects = 1 =2 =3 Each MOMENTUM award offers a deep understanding of different settings and approaches but is intended to work in a coordinated way across the Suite to provide specialized technical and regional expertise to address challenges to MNCH/FP/ RH care in varied contexts. Table 3: List of all 5 https://usaidmomentum.org/where-we-work/ 6 Map is from MOMENTUM’s website as of October 1, 2022. GH EvaLS MOMENTUM Mid-Term Process Evaluation / 19 MOMENTUM Awards, IPs, and Sub-IPs provides a full list of IPs and sub-IPs. USAID designed and further refined this Suite of Awards during implementation under the hypotheses that: ● An explicit mandate to coordinate, collaborate, and share learning across awards will help foster cross-fertilization of ideas and amplify the breadth of experiences and best practices in the field. ● A Suite of Awards will enable the ability to tell a holistic and persuasive story about USAID’s investment in preventing child and maternal deaths to various stakeholders, including Congress. ● Distinct awards that are all working toward the same ultimate results and purpose will have greater overall impact — in essence, the whole is greater than the sum of its parts. ● Integrated programming across the PRH and MCHN offices, with adequate coordination and collaboration, can reduce duplication of effort and increase complementarity in program implementation. ● Aligning GH programs to be structured similar to Mission programs will allow central mechanisms to be more responsive to Mission needs. ● Awards with specific expertise and experience can provide countries with more tailored and targeted support resulting in more effective programming. ● Each award is designed to focus on supporting countries in a particular phase of the development continuum or on priority technical areas which address emerging global priorities or tackle entrenched obstacles. Table 3: List of all MOMENTUM Awards, IPs, and Sub-IPs Name of Award Prime IP Sub-IPs MIHR IMA World Health 1. JSI Research & Training Institute, Inc. 2. Pathfinder International 3. Africa Christian Health Associations Platform 4. GOAL 5. CARE GH EvaLS MOMENTUM Mid-Term Process Evaluation / 20 Name of Award Prime IP Sub-IPs MCGL Jhpiego 1. Save the Children 2. Johns Hopkins University International Vaccine Access Center 3. The Manoff Group 4. Quicksand 5. Matchboxology 6. BAO Systems 7. Avenir Health 8. McKinsey & Company 9. PACT 10. Institute for Health Care Improvement 11. Christian Connections for International Health 12. Ubora Quality Institute MKA Population Reference Bureau 1. JSI Research & Training Institute, Inc. 2. Ariadne Labs 3. GEMNet MPHD Population Services International 1. Jhpiego 2. Avenir Health 3. FHI 360 4. ThinkWell MSSFPO EngenderHealth 1. IntraHealth International 2. London School of Hygiene and Tropical Medicine 3. Johns Hopkins Center for Communications Programs GH EvaLS MOMENTUM Mid-Term Process Evaluation / 21 Name of Award Prime IP Sub-IPs M-RITE JSI Research & Training Institute, Inc. 1. PATH 2. Accenture Development Partnerships 3. Results for Development 4. Gobee Group 5. CORE Group 6. The Manoff Group Table 4: MOMENTUM Coordination Mechanisms is an illustrative list of mechanisms for coordination across the awards include formal agreements, professional exchanges, and technology. Table 4: MOMENTUM Coordination Mechanisms Formal Agreements Professional Exchanges Technology • Letters of collaboration (e.g., between MKA and other awards) • Guidance on MOMENTUM research transparency and data sharing • Working groups o MEL o KM o Strategic communications o Digital health o Youth • Project-to-project meetings • Internal KM platform (HUB) • Data sharing platform • Public website • Social media accounts At the March 23, 2021, MOMENTUM All Hands Meeting,7 guidance was provided as to when and how coordination was to take place across the MOMENTUM Suite. Specifically: • Coordination mechanisms should be co-created and agreed upon by awards; • Coordination should be efficient and on an as-needed basis (no coordination for coordination’s sake); • Coordination should add value (result in better ideas, more data, more learning, and better programs, and reduce duplication of efforts); and • Coordination is part and parcel of each award’s mandate and budget (not considered as an extra burden or additional request). 7 All the awards, by this point, had been in implementation for several months and, for two awards, for more than one year. GH EvaLS MOMENTUM Mid-Term Process Evaluation / 22 Following approximately the first year of implementation for some of the projects, MKA conducted a review on behalf of USAID to gauge the understanding of “One MOMENTUM” and its implementation among the different MOMENTUM award holders and their USAID teams. Representatives from all six MOMENTUM awards and members of their USAID AOR and extended technical teams participated in focus group discussions carried out by MKA from January 11–22, 2021. The results of this review were presented in a two-day leadership meeting on February 16–17, during which collaborative solutions to improving technical and country coordination among the awards were decided. The MOMENTUM Suite of Awards is now approaching the mid-point of its current five-year award. MCGL and MKA are about to start their Project Year 4 work year, while MPHD, MIHR, MSSFPO, and M-RITE are all either beginning or about to begin their Project Year 3. Thus, the timing of this evaluation should, to the extent possible, feed into work plans for the subsequent years. Further, the evaluation results may affect the overall approach to MOMENTUM’s Theory of Change (Figure 2: MOMENTUM’s Theory of Change). Figure 2: MOMENTUM’s Theory of Change GH EvaLS MOMENTUM Mid-Term Process Evaluation / 23 3. ASSIGNMENT METHODS AND LIMITATIONS 3.1. ASSIGNMENT METHODOLOGY The MOMENTUM mid-term process evaluation commenced in March 2022 with an inception phase that lasted through the beginning of May. The next step was data collection (additional details follow below), which occurred from May until the end of July. A series of preliminary finding debriefs held during August allowed the team to receive feedback from various stakeholders, including USAID/W, IPs, and other in-country stakeholders. Drafting of this report occurred during August, with an internal review by GH EvaLS prior to its submission in September 2022. A non-experimental design with a mixed-methods approach utilizing quantitative and qualitative data was used to gather and analyze the needed information to produce findings against each Evaluation Question and Sub-question (the MOMENTUM articulated development hypothesis, Theory of Change, and Results Framework formed the fundamental evaluation elements). To answer the evaluation questions, the evaluation team used: • A desk review of documents and data; • KIIs and/or GIs; • An online survey; and • A country case study (Indonesia). An introductory email explaining the purpose of the evaluation was sent to stakeholders prior to requesting stakeholder participation in KIIs and the survey, and USAID/Indonesia sent an introductory email to inform local stakeholders about the Indonesia country case study. Because of the COVID-19 pandemic, virtual approaches to data collection were used, including virtual stakeholder meetings and key informant and group interviews. However, for the Indonesia case study, the local expert was able to travel within country to interview selected KIs as necessary. Protocols to ensure strict privacy and confidentiality, based on the GH EvaLS data management plan, were part of the data collection process. This included seeking informed oral consent, a statement that all information provided would be confidential and not connected to the individual, the right to refuse to answer any question, the right to refuse participation in the assignment, and the right to refuse to have the KII recorded (see Annex 4). 3.2. DATA SOURCES 3.2.1. Desk Review The first step in data collection was a desk review of approximately 60 MOMENTUM-related documents. The purpose was to obtain background information and data for the evaluation team and for use as part of the analysis of the MOMENTUM Suite. Documents and data reviewed included the initial APS and subsequent awards, results reporting, other MOMENTUM technical documents (briefs, case studies, etc.), meeting notes, data sharing agreements, workplans, and performance monitoring plans Annex 5B is a bibliography of documents reviewed. Documents were categorized as either “core” (i.e., all team members reviewed them) or “non-core” (only some team members reviewed the documents depending on the relevancy to assigned Evaluation Questions). GH EvaLS MOMENTUM Mid-Term Process Evaluation / 24 GH EvaLS assembled a passworded electronic folder of background documents from USAID and the MOMENTUM HUB, to which the evaluation team added materials it identified independently. To enhance the systematic analysis of the documentation it obtained, the team summarized documents using a standard template that was shared with team members. Any documents that contained sensitive information (proprietary technology, budgetary, procurement, etc.) were redacted before being shared with the team. 3.2.2. Key Informant and Group Interviews KIIs were conducted with selected informants whom the team identified through document review and verified with USAID and MOMENTUM IPs. KIs included representatives from: • Project AORs; • MCHN Office representation; • PRH Office representation; • BGH (specific to MOMENTUM interactions); • USAID Mission-based Activity Managers and/or Health Officers; • MOMENTUM IP staff (primes and a select group of sub-partners); • MOH officials (when available); • Global partners outside of USAID (WHO, UNICEF, GAVI, FP2020, etc.); and • Additional stakeholders identified during data collection. For stakeholders at the country level, the evaluation team contacted only individuals within a select sub￾group of countries where MOMENTUM is being implemented (see Section 3.3, Country Selection, below). In some cases, KIs were interviewed in small groups of similar respondents, if all participants felt free to express their opinions in a group. Interviews followed interview guides tailored to specific stakeholder groups (see Annex 4). The guides included opportunities for interviewees to provide comments at the end of the interview on topics not covered by the questions in the guide. Interview notes and recordings were kept in a password-protected electronic folder accessible only to the evaluation team and the GH EvaLS team. Information from each interview was logged using a data extraction log for interviews to enhance systematic analysis. All interviews were conducted in English. Table 5: Number of KIIs and Their Stakeholder Affiliations summarizes the interview participants (a full list is provided in Annexes 3B and 5A). GH EvaLS MOMENTUM Mid-Term Process Evaluation / 25 Table 5: Unique Number of KIIs and Their Stakeholder Affiliations 3.2.3. Online Survey An online survey was sent to targeted stakeholders including USAID Missions, MOMENTUM AOR management teams, USAID technical offices and other Agency leadership, MOMENTUM staff, and other stakeholders. This list of survey respondents was developed from inputs provided by USAID and the prime IPs. The initial list was then narrowed to 180 potential respondents, ensuring that there would be no duplication among those individuals participating in the interviews (either individually or within groups) and those responding to the survey, a rating by USAID and IPs (“High,” “Medium,” and “Low”) for potential information yield, and a review of the potential respondents’ positions within the MOMENTUM structure. The survey was sent to 11 countries based on the criteria listed in Table 6: Initial Criteria for Country Selection (KIIs and Survey) below, focusing on countries scoring 3+ (except Indonesia, the country case study). The full draft survey, included in Annex 4, was beta-tested internally several times prior to distribution. It was translated and distributed in English, French, and Portuguese and was open for responses for three weeks; two reminders were sent in addition to an introductory email. A total of 88 responses were collected (approximately 48% response rate) across the three languages (56 English, 25 French, six Portuguese). Some basic demographic information, presented in Figure 3 and Figure 4, shows that the majority of respondents were based at the country level and that a large percentage of feedback came from IPs. Not all survey respondents received the same set of questions, based on their MOMENTUM affiliation (see Figure 4). Skips were inserted into the survey based on affiliation to better target questions; however, as will be seen in the Findings (Section 4), this resulted in several questions having a low number of respondents/sample size. 8 From the six MOMENTUM awards, the total distribution of KIs was MCGL = 9; MIHR = 9; MKA = 5; MPHD = 3; M-RITE = 7; and MSSFPO = 5. The five IP country staff interviewed were drawn from four countries. Stakeholder Number USAID/W 13 USAID Mission staff 9 IP headquarters staff 338 IP country staff 56 External partner organizations (international technical agencies, foundations, etc.) 5 Indonesia (country case study) 24 Total 89 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 26 Figure 3: Survey Respondents’ Locations N=88 Figure 4: Survey Respondents’ Affiliation with MOMENTUM 60% 40% Where are you located? In-country Headquarters 6% 75% 7% 4% 8% MOMENTUM Affiliation Global partner (e.g., WHO, UNICEF, GAVI, FP2020, etc.) Implementing partner Ministry of Health / in-country government counterpart USAID Mission Staff USAID/W staff N=88 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 27 3.2.4. Country Case Study (Indonesia) The purpose of the case study was to provide an in-depth, specific example of how MOMENTUM IPs are collaborating, learning, and adapting. This would enable the evaluation team to generate additional insights. While the case study focused on one country (Indonesia), it is situated within the overall evaluation framework and the corresponding Evaluation Questions. The evaluation team triangulated data and findings from the case study with data and findings from the KIIs, survey, GIs, and desk review. USAID’s criteria for selecting the country case study included significant investment; multiple MOMENTUM IPs in-country; USAID Mission interest and support; use of in-country expertise; and no international travel. Based on an iterative process, including discussions with USAID Missions, Indonesia was selected as the case study country. After reviewing USAID/Indonesia’s initial feedback, the local consultant liaised and consulted with Mission staff to ensure that their concerns and perspectives were incorporated into the data collection tools, with guidance from the evaluation team, while adhering to the fundamental themes and questions of the evaluation. Then, with USAID/Indonesia’s introduction, he began contacting, scheduling, and conducting KIIs. Quality assurance was provided via email and check-in calls to ensure any issues with data collection were mitigated. The full case study is provided in Annex 3. 3.3. SAMPLE/COUNTRY SELECTION (KIIs AND SURVEY) Not all countries in which MOMENTUM is working were involved in the KIIs and the survey. In Table 6: Initial Criteria for Country Selection (KIIs and Survey) below, the evaluation team lists the initial criteria against applicable countries. Based on this table and discussions with USAID, the initial MOMENTUM countries were narrowed down to 119 to participate either in the interviews or survey. Indonesia, although it had a high score, was excluded from the survey as it had already been chosen for the country case study, which would include in-country KIIs. Table 6: Initial Criteria for Country Selection (KIIs and Survey) MOMENTUM Presence10 Has at least 2+ MOMENTUM Awards11 USAID Priority Countries for Maternal and Child Health (MCH)12 Priority PRH Countries Total Score (1 Point per X) Bangladesh X X 2 Benin X 1 Burkina Faso X 1 Cameroon X 1 9 Democratic Republic of the Congo, Ghana, India, Kenya, Mali, Mozambique, Nepal, Nigeria, Rwanda, Tanzania, Uganda. 10 Per Where We Work - USAID MOMENTUM, April 28, 2020. 11 Ibid. 12 Per https://www.usaid.gov/global-health/health-areas/maternal-and-child-health/priority-countries as of April 28, 2022. GH EvaLS MOMENTUM Mid-Term Process Evaluation / 28 MOMENTUM Presence10 Has at least 2+ MOMENTUM Awards11 USAID Priority Countries for Maternal and Child Health (MCH)12 Priority PRH Countries Total Score (1 Point per X) Côte d’Ivoire X 1 Democratic Republic of the Congo X X X 3 Ghana X X X 3 India X X X 3 Indonesia X X X 3 Kenya X X X 3 Malawi X X 2 Mali X X X 3 Mauritania X 1 Mozambique X X X 3 Nepal X X X 3 Niger X X 2 Nigeria X X X 3 Pakistan X X 2 Philippines X 1 Rwanda X X X 3 Senegal X X 2 Sierra Leone X 1 South Sudan X X 2 Tanzania X X X 3 Togo X 1 Uganda X X X 3 Zambia X X 2 Number of countries scoring 2+ 20 Number of countries scoring 3+ 12 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 29 3.4. DATA MANAGEMENT AND ANALYSIS All information and data collected were analyzed and triangulated across the data sources, focused on answering the evaluation questions, and fitting within the time and resources allotted for the evaluation. The use of quantitative and qualitative methods, as well as existing data, enabled the team to triangulate findings to produce more robust results. Data and findings from the Indonesia case study were triangulated with data and findings from the evaluation KIIs, survey, GIs, and desk review to derive overall conclusions, supplemented by specific examples from the case study. All data sources were reviewed holistically to determine corroboration or identify specific themes. The team analyzed the material identified in the Document and Interview Extraction forms by Evaluation Question and Sub-question. During data reduction and analysis, the team developed a provisional list of emerging themes related to the questions and elements of the Theory of Change. These themes provided the basis for thematic content analysis of the reviewed documents and interviews to identify preliminary findings in relation to the Evaluation Questions and Sub-questions. When different methods presented conflicting evidence, the team examined the degree to which the findings diverged to determine why the data conflicted, and presented this information as part of the strength of evidence within the findings. It should be noted that, in general, though, more weight was given to the information derived from the KIIs in comparison to the other lines of evidence (e.g., survey, desk review, country case study) because they provided a deeper information and, specifically in reference to the survey, a greater number of responses to specific issues, even though there were 89 KIs and 88 survey participants. As noted previously, not all survey respondents received the same set of questions. Thus, in some cases, the corresponding sample sizes were small. Through triangulation of the qualitative and quantitative evidence gathered and analyzed, the evaluation team produced a final Findings, Conclusions, and Recommendations (FCR) Matrix that provided a clear line of sight from Evaluation Questions and sub-questions to FCR. 3.4.1. Quantitative Data Analysis The evaluation team analyzed quantitative data primarily using descriptive statistics from survey responses; when possible, the team also attempted to quantify data generated from the key informant and group interviews. Data were primarily stratified by location (e.g., in-country or headquarters) and stakeholder affiliation (e.g., IP, USAID, in-country government, etc.). After the data were reviewed and cleaned, a number of graphics were generated by combining data across the three survey languages and, when applicable, across stakeholder groups. Some of the graphics are presented as part of the findings; the full disaggregated information can be found in Annex 6. 3.4.2. Qualitative Data Analysis The evaluation team used qualitative data to substantiate quantitative findings, provide more insights into quantitative data alone, and answer questions where other data did not exist. The evaluation team analyzed KIIs, GIs, and qualitative responses within the survey using content and thematic analysis techniques aligned with the Evaluation Questions and Sub-questions. Key themes and their definitions were iteratively developed and discussed among team members and documented in the FCR Matrix. The team compared evidence and perceptions from the perspectives of the various stakeholders and, through that process, accumulated evidence and insights. The thematic review of the qualitative data GH EvaLS MOMENTUM Mid-Term Process Evaluation / 30 sought to establish relationships, context, interpretation, nuances, homogeneity, and outliers within and among data points to better explain the overall perception of the various aspects of MOMENTUM. 3.5. ETHICAL CONSIDERATIONS Stakeholders’ informed oral consent was sought as a condition of participating as key informants. The documentation (see Annex 4) contained:13 ● An introduction of the interviewer; ● A review of the purpose of the evaluation; ● A brief discussion of the purpose of the interview; ● A statement that all information provided is confidential and information provided would not be connected to the individual; ● The right to refuse to answer questions or participate in interview/discussion and right to stop interview at any time; and ● A request for consent prior to initiating data collection (i.e., interview)/discussion. Information obtained from KIIs was secured in a GH EvaLS password-protected folder accessible only to the team. The data were reviewed and discussed only within the team. The confidentiality of respondents’ contributions was maintained by pooling and citing all input by stakeholder group and by anonymizing any quotations that were extracted to illustrate key points. Similar measures were put in place to ensure the confidentiality of survey results. The survey questionnaire also included an informed consent statement as part of the introduction (see Annex 4). The evaluation team did not interview any project beneficiaries. Further, no one under 18 years of age was interviewed or participated in the online survey. 3.6. LIMITATIONS The evaluation team anticipated that several possible biases and other data limitations would need to be mitigated through methodological and/or analytical means. These included: 1. Qualitative approach: The primary approach for this evaluation was qualitative data collection via KIIs and possibly GIs and, to a lesser degree, qualitative and quantitative questions within the survey. The opinions of stakeholders are, by their nature, subjective, and may also reflect certain vested interests. Further, team members may not have accurately recorded or correctly transcribed important data for a variety of reasons (such as fatigue and difference in understanding). Due to the COVID-19 pandemic, all interviews were virtual. Virtual interviews may limit the ability of evaluation team members to observe non-verbal communication expressed by informants. 2. Selection and representation biases: Key stakeholders may have been inadvertently excluded, and those persons who participated in a KII or GI may have introduced self-selection bias (either beneficial or detrimental) into the results. Persons with stronger vested interests in the results of the assessment (either negatively or positively) may have been willing to spend 13 KIIs were developed in alignment with the Common Federal Policy for Protection of Human Subjects in Research (the Common Rule) adopted by USAID. GH EvaLS MOMENTUM Mid-Term Process Evaluation / 31 more time with the interviewers. However, the evaluation team identified numerous stakeholders at the global and country levels and, while the team experienced some difficulties in scheduling interviews with stakeholders (e.g., because of other demands on their time or personal leave), a large and diverse group was interviewed. 3. Response bias: Response bias is a common problem for assessments, evaluations, and reviews and takes several forms. For instance, respondents may have given the interviewers positive remarks about MOMENTUM because they have a vested interest in seeing the project succeed and continue. Respondents may have been inclined to shape their responses according to gender or other social norms, or to say what they think the interviewer wants to hear. 4. Recall and recency biases: Respondents’ experience with the MOMENTUM Suite may have varied by the length of time involved, which would positively or negatively influence the opinions of stakeholders regarding MOMENTUM processes. Respondents may have given more weight to recent events and understated the significance of conditions and events that occurred earlier. 5. Inherent competition/feedback bias: Most of the organizations involved in MOMENTUM and that participated in the KIIs are inherently competitors for USG awards and funding. Thus, they may have incentives to provide positive or negative feedback on the MOMENTUM Suite of Awards that would distort the accuracy of the findings. This could be to improve USAID’s current perception of their performance and negatively impact a competitor organization or to better position themselves for future awards. The evaluation team, however, drew from a wide range of stakeholders from all awards (see Table 5 above), as well as additional partners, and gave the awardees the opportunity to submit self-developed lists of individuals to be interviewed. The evaluation team believes, therefore, that this bias was substantially mitigated. 6. Survey conversion: One unexpected challenge in the implementation of the evaluation that caused a delay in collecting data was the instruction to the evaluation team that the survey could not be completed in Survey Monkey but rather had to be entered and distributed via Google Forms. Unfortunately, this instruction came after the survey had already been finalized and beta￾tested in Survey Monkey and necessitated its re-entry/conversion. Further, because none of the team members had previously conducted a survey using Google Forms, a substantial learning curve involved further delayed the survey’s distribution. The evaluation team believes that these limitations were mitigated by the number and scope of methods utilized and that the robustness of the methods enabled the team to develop valid findings, estimate the strength of the underlying evidence, synthesize conclusions, and produce specific and actionable recommendations. GH EvaLS MOMENTUM Mid-Term Process Evaluation / 32 4. FINDINGS Findings are presented below by evaluation question and sub-question and, as mentioned previously, draw upon four lines of evidence (documentation, KIIs and GIs, survey results, and the Indonesia case study). The strength of the evidence (High or Medium) is noted as well as when, for example, the data triangulates against multiple lines of evidence and is consistent throughout those data sources. Evaluation Question 1: Have the awards been able to leverage their relationships with other MOMENTUM awards to benefit their work, e.g., to improve the efficiency, effectiveness, and/or scale of programming? If so, how? Finding 1.1 (High): The main contributing factors for effective leveraging of relationships at the country level are the formal division and clarity of roles in cases when one award focuses on the private or public sector; division of geographic intervention areas; division of focus of engagement at different levels of the health care system, such as a national MOH versus the district level; and active Mission commitment to and involvement in planned collaboration. Examples of countries where MOMENTUM awards have leveraged their relationships to improve efficiency and effectiveness at the country level include Mali, Nigeria, Niger, the Democratic Republic of the Congo, Kenya, and Indonesia. Further discussion and examples are outlined in the findings for Evaluation Sub-question IC. STRENGTH OF EVIDENCE: HIGH Finding 1.2 (High): KIs cited few examples of leveraging relationships to improve the scale of programming thus far at either the headquarters or country level. Informants cited a loss of efficiency of scale due to less core funding, lack of time, and fewer staff to foster collaborative relationships. The relationship between MIHR and M-RITE in the Democratic Republic of the Congo potentially improves the scale of programming through a formal agreement to work in different geographic areas and at different levels of the health system. M-RITE has taken the lead in coordinating with the MOH at the national level, while MIHR focuses its interventions at the district level. STRENGTH OF EVIDENCE: HIGH Finding 1.3 (High): In general, as mentioned by KIs working at the global level, the awards are not functioning as a suite, and collaboration occurs primarily on an as-needed basis based on varying perceptions of when that collaboration is needed. Informants stated that staff do not consistently reach out to other awards, the vision of MOMENTUM as a Suite has not worked as it was intended, and the distinct realms are not working together on implementation. Few USAID and IP staff see much benefit to coordination, while citing costs in time and duplication of efforts. In some cases, the awards have had to create dual tracking and reporting systems. “There is no scale of programming, although there has been some leveraging of technical expertise.” — Key informant interview “Approaching MOMENTUM as one entity is a challenge. There are too many partners in the mega awards and each organization has its own branding.” — Key informant interview GH EvaLS MOMENTUM Mid-Term Process Evaluation / 33 STRENGTH OF EVIDENCE: HIGH Finding 1.4 (Medium): At the country level, many in-country KIs do not perceive MOMENTUM as a singular project. Many KI IPs described a lack of understanding at Missions regarding what the Suite is intended to do and how the Suite of Awards should work together to meet the needs of the Mission. This was reinforced by USAID Mission KIs who, likewise, noted that when they bought into multiple MOMENTUM awards it was not because of any overall coordinated approach by the Suite but rather a need to respond to specific in-country technical needs. Their expectations for any collaborative efforts were limited. Further, KIs mentioned a lack of clarity about whether coordination was required and the potential for cross-learning and -fertilization. However, the few (n=6) survey respondents who answered this question generally had a different view. When asked whether they perceived MOMENTUM as “one” project, five of the six individuals responded that they did. STRENGTH OF EVIDENCE: MEDIUM Evaluation Sub-question 1A: What has been the cost and benefit (to USAID and IPs) of coordinating with MKA and across the Suite of awards? What have been the barriers and facilitators to achieving this? Finding 1A.1 (High): The most notable achievements of MKA have been the MOMENTUM HUB and the Share Fairs. Most IPs, both at the country and headquarters levels, are appreciative of the MOMENTUM HUB and the Share Fairs, although many informants noted that the level of effort required to prepare for the Share Fairs was a challenge. The Share Fair is considered a useful tool for shared knowledge and learning and for increasing engagement and collaboration. The HUB is seen as a valuable resource for accessing information, tools, and resources and for standardization, although the reported use varied due to time constraints. The training for adaptive management and MAKLab were also cited as achievements of MKA. There were only seven survey respondents to the question “How would you rate the benefit of coordinating with MKA to USAID and Implementing Partners?” Three respondents believed that the benefits of coordinating with MKA across the Suite were High,” three rated them as Medium, and one rated them as Low. STRENGTH OF EVIDENCE: HIGH Finding 1A.2 (High): Perceived benefits of MKA’s strategic communications engagement varied based on awards’ communications capacity. Many headquarters-based KIs stated that the MKA approval process has slowed and has been a barrier to both messaging and implementation. However, this is award-specific, based on organizational capacity (e.g., high￾capacity awards saw it as burden, and low-capacity awards did not). Awards led by organizations with longstanding communications departments generally perceive collaboration with MKA as a barrier to sharing knowledge; producing messages, learning materials, and publications; and telling the story of MOMENTUM. However, most informants commented positively on the dedication, professionalism, and exceptional skills of MKA staff (e.g., they respected and thought highly of them as people). KIs from awards with little strategic communications capacity or experience stated that MKA saved them time and money. This was stated by KIs at two non-MKA projects. and MKA staff also noted that a subset of the MOMENTUM awards reached out to MKA more than others to assist with strategic communications. “There is a bottleneck if we want to prepare a blog. It takes months and then it is rejected.” — Key informant interview GH EvaLS MOMENTUM Mid-Term Process Evaluation / 34 STRENGTH OF EVIDENCE: HIGH Finding 1A.3 (High): A major barrier that many key informants cited is that MKA presents an additional layer of bureaucracy for approvals. For M&E frameworks, numerous headquarters-based informants stated that the MOMENTUM Data Platform is too macro, many of the indicators are not relevant to their projects, the lead time for requests from MKA is too short, approval processes take too long, and MKA lacks authority to enforce its coordinating efforts. According to some KIs, some projects have had to set up parallel data systems because the MOMENTUM Data Platform is not appropriate for their needs, and they have had to pay for data entry at a high cost. STRENGTH OF EVIDENCE: HIGH Evaluation Sub-question 1B: What has been the cost and benefit of awards coordinating with each other (exclusive of MKA)? Finding 1B.1 (High): The major benefits have been around particular technical areas (e.g., immunizations, public/private sector linkages) when there is a clear scope and mandate to interact, specifically at the country level. This has enabled knowledge sharing and some joint work planning and activities, primarily at the country level. In some countries (for example, Nigeria), the awards are sharing tools; this saves time and contributes to standardization of care. In countries where there have been military coups, MPHD liaised with the private sector to work with MCGL, MIHR, and M-RITE, enabling the MOMENTUM programs to continue.14 However, the workplans needed to be totally revised. Informants cited M-RITE’s role as the technical lead in the response to the COVID-19 pandemic to support other awards as an example of effective collaboration. STRENGTH OF EVIDENCE: HIGH Finding 1B.2 (High): The major cost, as described by KIs, is the level of coordination — both internally (i.e., within the awards) and externally (i.e., with USAID or other partners). According to both headquarters and country-level IP KIs, the extra time and effort required to coordinate as a Suite is burdensome for relatively modest gains. Other costs cited included challenges and confusion with the project’s identity subsuming the organization’s identity, as well as branding and rebranding issues. STRENGTH OF EVIDENCE: HIGH Evaluation Sub-question 1C: What has been the cost and benefit of in-country coordination (to USAID, to IPs, to country partners)? 14 Specific country examples of this include Mali and Niger. “The document produced by MKA outlining the rules is 19 pages long and the rules are confusing and contradictory.” — Key informant interview “We have had to set up two separate systems for everything. It is manageable but there is a duplication of efforts and there is a cost to that.” — Key informant interview GH EvaLS MOMENTUM Mid-Term Process Evaluation / 35 Finding 1C.1 (High): Local technical leadership at the subnational level has been enhanced and strengthened in some countries. According to most in-country informants, local expertise has been strengthened and cross-pollinated best practices. One informant stated that the multi-country design has been a virtual platform for learning and that it increased districts’ ability to interact with experts in other countries and at the global program level. STRENGTH OF EVIDENCE: HIGH Finding 1C.2 (High): Feedback was mixed regarding whether the MOMENTUM design has been responsive to the needs of health ministries and/or partners. The benefits have been modest, with some joint programming done. For many partners, MOMENTUM, through the depth of experience and extensiveness of the participating IPs (including both primes and subs) has avoided gaps in technical skills. In some countries, IPs have been able to coordinate and ensure limited overlap by dividing their efforts either geographically or functionally. MOMENTUM’s support for the response to the COVID-19 pandemic was appreciated by many MOHs and increased engagement with local NGOs and districts at the central and subnational levels, according to one informant. Other country informants stated that the MOMENTUM design is not aligned with the priorities of the MOH. Initially, both Missions and government counterparts experienced quite a bit of confusion at the country level regarding the structure of MOMENTUM. However, most of those issues have been resolved, and any outstanding issues are dependent on Mission bandwidth (and interest) to clarify with counterparts MOMENTUM’s wider-ranging approach. Most country counterparts see little benefit in the structure of the award; without USAID staff consistently pushing for collaboration and coordination, it occurs infrequently. STRENGTH OF EVIDENCE: HIGH Evaluation Sub-question 1D: Have awards been able to create new partnerships and ways of working within the consortium? Finding 1D.1 (Medium): There is limited evidence to date that the awards have created new partnerships and ways of working within the consortium. In August 2021, MKA published Approaches to Partnership Measurement: A Landscape Review to examine existing measurement approaches, frameworks, and metrics utilized to measure partnerships. The review aimed to inform the selection of evidence￾informed, appropriate, and feasible measurement indicators to monitor partnerships within the MOMENTUM consortium. Similarly, MCGL published a policy brief, Promising Practices for Engaging Local Faith Actors to Promote Uptake of COVID-19 Vaccination: Lessons Learned from Four Countries (Ghana, Indonesia, Sierra Leone, and Uganda) in 2022 to advance learning and evidence regarding how programs could work with faith-based organizations and religious leaders to accelerate the uptake of vaccines and counter vaccine misinformation. Although both documents were intended to provide guidance and support for forming new partnerships, KIs commented that stakeholders prefer to continue to interact with “known quantities,” and this may have inhibited their willingness to seek out new partnerships. MOMENTUM Uganda’s response to COVID-19 is an example of new partnerships and ways of working. There were huge gaps in water, sanitation, and hygiene (WASH) at the local level, and MOMENTUM, “The health system was challenged and engagement (by MOMENTUM) with the local NGOs and the MOH at the district and subnational level was very responsive to MOH needs.” — Key informant interview GH EvaLS MOMENTUM Mid-Term Process Evaluation / 36 primarily through MCGL, partnered with UNICEF, Makerere University, and the Catholic Medical Bureau to improve infection prevention care to decrease neonatal and maternal infections. Virtual platforms of the Digital Data Collection were downloaded on phones, and the facilities uploaded data to feed into the MOH. Dashboards were developed to use the data and inform procurements and trainings. The data were shared with local leaders and partners. Local leaders noticed that 60 percent of staff were not washing their hands, and infection prevention improved after dissemination of information. The use of personal protective equipment was low, and supplies were lacking; the data helped local leaders advocate to procure more supplies and to help with vaccines. A USAID informant commented that there is a need to figure out a way to incentivize cross￾collaboration and make it easier for awards; there is currently little incentive to collaborate; and it is even more difficult for field support, which is primarily focused on meeting the needs of the Mission rather than serving USAID’s agenda for MOMENTUM. Per a few KIs, core funds are low compared to buy-ins, so the amount of funding that awards are able and willing to use for collaboration around the MOMENTUM Theory of Change is unclear. STRENGTH OF EVIDENCE: MEDIUM Evaluation Sub-question 1E: What adaptations (e.g., changes in systems/processes) would lead to more efficient and/or more effective coordination? Finding 1E.1 (High): Feedback from most KIs focused on USAID needing to first structure itself internally to provide guidance and instruction as to how coordination should take place. Many stakeholders noted that turnover in AORs had hindered their ability to coordinate due to different management styles and requests, and that the AORs themselves were not well coordinated. Informants cited the need for USAID to encourage and elevate collaboration and coordination and to promote “positive deviance.” Other suggestions include decreasing complexity at the USAID level and determining who really needs to give the go-ahead for approvals to improve efficiency and save time. STRENGTH OF EVIDENCE: HIGH Evaluation Question 2: What operational shocks and stressors are driving programmatic adaptations? Finding 2.1: The key shock or stressor has been the COVID-19 pandemic (discussed further in the findings for Evaluation Sub-question 2A). Finding 2.1.1 (High): MIHR, by its nature, needs to be discussed separately from the other MOMENTUM awards when examining shocks and stressors. MIHR is focused on strengthening resilience in fragile settings (i.e., countries affected by conflict, disasters, weak governance and institutions, population displacements, and other acute and chronic crises); therefore, its activities will face a substantial number of challenges by definition. Since the award was made in May 2020, MIHR has received funding to implement interventions in, for example, Burkina Faso, which is in the midst of a seven-year insurgency; Mali, which underwent a coup in August 2020; and the Democratic Republic of the Congo, which has hosted a U.N. peacekeeping mission for more than 20 years. These conflicts “We are trying to weave one MOMENTUM together without things really being coordinated.” — Key informant interview GH EvaLS MOMENTUM Mid-Term Process Evaluation / 37 occur within the greater context of climate change and related environmental emergencies, loosely controlled borders and migration, and periodic public health crises (e.g., ongoing Ebola outbreaks). While the four other awards within the MOMENTUM Suite that have country presence15 may be facing their own challenges (e.g., the COVID-19 pandemic), MIHR has borne the greatest burden, per KIs, and has had to develop innovative solutions to continue with activities when, for example, it is unable to interact with government counterparts. Because of these ongoing difficulties, implementation has been delayed in MIHR countries. These delays, say KIs, further complicate MIHR’s ability to coordinate, collaborate, and align itself within the greater MOMENTUM Suite (see Finding 2D.2 for additional details). STRENGTH OF EVIDENCE: HIGH Finding 2.1.2 (High): The M-RITE award faced the unexpected task of contributing to mitigating the impact of COVID-19. M-RITE, which was awarded in July 2020, was originally intended to focus its efforts on applying best practices and innovations to increase equitable immunization coverage in USAID-supported countries, specifically to overcome barriers to reaching zero-dose and under￾immunized children and older populations with vaccines and other integrated health services. However, the COVID-19 pandemic significantly changed that focus. M-RITE’s Year 2 Core Workplan lists the first technical priority as “Support COVID-19 vaccine roll out” which, while perhaps at a future date would be considered a routine immunization, is not currently. As in the workplan, M-RITE is devoting substantial technical resources to supporting COVID-19 vaccine rollout in low- and middle-income countries. At the global level, M-RITE has provided technical support in helping countries develop national deployment and vaccine plans and facilitated their review by the COVID-19 Vaccines Advance Market Commitment equitable access vaccine initiative. KIs from M-RITE and other IPs noted that M-RITE has responded positively to these challenges; however, its COVID-19 response has stretched M-RITE’s bandwidth, pulled it away from routine immunizations, and placed greater constraints on its ability to coordinate and collaborate with other awards within the MOMENTUM Suite. STRENGTH OF EVIDENCE: HIGH Evaluation Sub-question 2A: How has the Suite collectively, and as individual awards, adapted to the realities of working in dynamic country contexts, particularly the ongoing effects of the COVID-19 pandemic? Finding 2A.1 (High): The individual awards and the MOMENTUM Suite collectively have made an outstanding response to the COVID-19 pandemic. KIs were nearly unanimous in the observation that MOMENTUM had not only adapted well to COVID-19 but had successfully innovated methods and activities to significantly mitigate the pandemic’s impact. KIs provided numerous examples, such as the 15 MKA has no ongoing country presence. “M-RITE was supposed to focus on routine immunization, but COVID changed the focus overnight. We received a lot of emergency funding and there is a much shorter time frame to spend the money and for staff to get the work done.” — Key informant interview GH EvaLS MOMENTUM Mid-Term Process Evaluation / 38 move toward virtual trainings, implementation of COVID-19 prevention measures for in-person activities, the modeling behavior of staff in receiving vaccinations, development of templates for monitoring how COVID-19 affected activities each month, and the general flexibility of adapting workplans in response to changing or easing in-country government guidance and restrictions. Indeed, a few KIs praised MOMENTUM clinical teams for their courage in continuing in-facility activities during the pandemic. From this experience, MOMENTUM developed numerous documents (technical guidance, research articles, distance learning resources, etc.) that can provide a plethora of information and lessons. Some of those documents are listed in Annex 5B. STRENGTH OF EVIDENCE: HIGH Finding 2A.2 (High): There also have been some challenges to the MOMENTUM Suite due to adaptations to the COVID-19 pandemic. Per KIs (primarily IPs), the move to virtual trainings, for example, limited the number of individuals able to participate because of either lack of online access or the natural limits to how many persons can be hosted without lowering the quality of the instruction. On a similar note, KIs noted that it was difficult to build teams at all levels. This included at headquarters, both within individual awards and across the Suite; between headquarters and in-country offices for individual awards; and within countries. Additionally, outreach efforts to counterparts (government, technical agencies, bilateral awards, etc.) at the country level were impeded because of the inability to meet. Other COVID-19-related barriers need to be considered on a case-by-case basis; however, in general, the awards have adapted, although sometimes at a cost of slower-than-expected implementation. While slower implementation was referenced by all interviewed USAID Mission staff, they also noted that it was not a substantial impediment to achieving Mission-level goals and, given the circumstances, the awards were, for the most part, meeting Mission expectations. STRENGTH OF EVIDENCE: HIGH Evaluation Sub-question 2B: What adaptations (e.g., changes in systems/processes) would lead to more efficient and/or more effective shared learning? Finding 2B.1 (High): At the headquarters level, overstretched IP staff do not have sufficient time to devote to learning activities. The design of MOMENTUM, as given in its original APS, had learning as part of its mandate via Results 3: Adaptive learning and use of evidence in MNCH/FP/RH programming through sustained USAID and host country technical leadership increased. USAID, in Automated Directives System (ADS) Chapter 201, provides definitions and guidance for how learning should be undertaken, and its importance to the agency and the development process. Thus, the findings for the Evaluation Sub-question need to be examined based on that understanding. Table 7 summarizes some of the ADS 201 learning information. “We have to figure out a way to incentivize the cross-collaboration and to make it easier for them and it’s even more difficult when we’re really focused on serving the Mission’s needs.” — Key informant interview GH EvaLS MOMENTUM Mid-Term Process Evaluation / 39 Table 7: ADS 201 Guidance on Learning Definition of learning. A continuous organizational process throughout the Program Cycle of analyzing a wide variety of information sources and knowledge, including evaluation findings, monitoring data, research, analyses conducted by USAID or others, and experiential knowledge of staff and development actors. Importance. A learning organization supports actively questioning assumptions, seeking evidence, reflecting, and exploring a range of solutions to development problems. Methods. Learning can take place through a range of processes and use a variety of sources, such as monitoring data, evaluation findings, research findings, analyses, lessons from implementation, and observation. Roles of USAID and IPs. IPs and local and regional actors play a central role in USAID’s efforts to be a learning organization. Knowledge and learning should be documented, disseminated, and used to help spread effective practices widely for improved development. How and when. Pausing periodically to reflect on new learning and knowledge and adapting accordingly. To date, based on the criteria described above, the MOMENTUM Suite has had limited learning opportunities, specifically in being able to pause and reflect16 on new learning and knowledge and to adapt accordingly.17 However, several resources for Adaptive Learning/Adaptive Management have either been developed or were in the process of being finalized during the review period. There have been several Share Fairs, topic-based learning exchanges, and technical forums, and learning agendas and tools have been developed. However, IP KIs noted that the “strategic” element of learning is not occurring because there is limited time for reflection. Further, because of the lack of clear guidance on whether and how coordination for learning should take place, opportunities for cross-learning and cross-fertilization have been missed — both for adaptative management purposes and technical issues (local capacity building, immunizations, private sector engagement, etc.). Approximately 20 percent of IP KIs who were asked about learning noted that adaptive learning, while a substantial focus for the MOMENTUM Suite, takes place at the individual award level and that there should be a greater effort to unite these undertakings. 16 Per stakeholders, MIHR did engage in a pause and reflect activity after Project Year 1. 17 MOMENTUM as a project does have a shared learning agenda, as described in the MOMENTUM MEL Framework. Individual MOMENTUM awards have specific learning agendas. In addition to exploring whether MOMENTUM goals were achieved, the learning agenda explores how MOMENTUM partners achieved success, capturing information on the contexts, strategies, approaches, and adaptations shaping MOMENTUM’s work. “I know there is so much we could be learning from the other awards. We are, as a community, very siloed.” — Key informant interview GH EvaLS MOMENTUM Mid-Term Process Evaluation / 40 STRENGTH OF EVIDENCE: HIGH Finding 2B.2 (High): At the country level, there is limited evidence of shared learning. Some KIs (primarily IPs at the country level and USAID Mission staff) cited instances of participating in learning events and trainings, and some mentioned specific technical knowledge generated by MOMENTUM activities. More generally, however, those respondents were only somewhat aware of the Suite’s learning agenda, knowledge products, and more global activities. Some reasons cited for the suboptimal learning were the lag time between the start of implementation and the development and rollout of the Learning Agenda, lack of capacity at the local level to contribute to learning, and the centralization of learning into MKA,18 which created a gatekeeper situation of countries being limited in determining their learning priorities. Similar to headquarters staff, in-country counterparts noted missed opportunities for expanded learning. STRENGTH OF EVIDENCE: HIGH Finding 2B.3 (Medium): Potential solutions for increasing shared learning were offered, and the COVID-19 pandemic has presented both challenges and opportunities for learning. While the pandemic limited travel and in-person knowledge sharing, it also forced the MOMENTUM Suite to adapt the way it conceives of and implements learning. Activities such as trainings, workshops, and Share Fairs were moved online with few negative effects other than the limited number of participants. Some of the 60 respondents to the survey question “What adaptations (changes in systems/processes) would lead to more efficient and/or more effective coordination, help foster cross-fertilization of ideas and best practices, and reduce duplication of efforts?” offered several possible solutions focused on increasing learning. Suggestions included: • More in-person regional knowledge fairs; • South-South missions; • Cross-border/regional technical assistance opportunities; • Appointing one of the awards at the country level as the “convener”; • Monthly highlight meetings; • More thematic groups and coordination platforms; and • Joint field visits. A general theme in several responses was that sharing and learning need to be more regular and systematic, and that they should be based on a jointly prepared timetable. However, there was no consistent response as to who or which institution should be responsible for ensuring that this occurs at the country and regional levels. 18 It should be noted that the MKA’s learning agenda was originally developed based on the other awardees’ global learning agendas. “Cross award coordination, learning etc. are non-existent in my setting. This would be useful if made available.” — Survey respondent GH EvaLS MOMENTUM Mid-Term Process Evaluation / 41 STRENGTH OF EVIDENCE: MEDIUM Sub-question 2C: Do any of these adaptations require adjustments to the development hypothesis and/or key assumptions of the MOMENTUM award Suite and/or implementation at the country or global level? Finding 2C.1 (High): Most of the technical components that comprise MOMENTUM’s Theory of Change do not need adjustment. However, several of its underlying assumptions, as well as Result 4, need to be reconsidered as there has been limited progress in those areas. MOMENTUM, as per its Theory of Change and the original APS,19 has as its goal contributing to USAID’s priority of preventing maternal and child deaths by assisting USAID-supported FP and MCH priority countries in meeting global goals in MNCH and voluntary FP by 2030. That Theory of Change, as shown and discussed in the Background section, has four results, as well as several underlying assumptions around collaboration, learning, adaptation, partnerships, cross-fertilization, and holistic storytelling. Further, Result 4, as given in the APS, states: “Cross-sectoral collaboration and innovative partnerships between MNCH/FP/RH and non￾MNCH/FP/RH organizations increased. MOMENTUM will advance a holistic, multi-sectoral approach to meet the health needs of women and children across the MNCH/FP/RH continuum, exploring strategies to support cross-sectoral linkages and priorities where possible with both traditional and non-traditional organizations.” While KIs were able to provide some examples of how some of the Intermediate Results under Result 4 had been achieved (MPHD’s increasing of public-private partnerships, the inclusion of academic institutions within the original Suite and outreach during implementation, etc.), the underlying theme is that, within the Suite, there had not been sufficient collaboration and partnership, and this resulted in less-than-optimal results for external partnering as well. As discussed in the findings for the other evaluation questions and sub-questions, the challenges around collaboration, learning, partnership, and holistic storytelling are recurrent. They were mentioned not only by survey respondents, but by all KIs. Only two of seven survey respondents stated that the benefits of coordinating across the Suite were High. STRENGTH OF EVIDENCE: HIGH Finding 2C.2 (Medium): No generalizable statements can be made for adaptations of the Theory of Change at the country/Mission level. At the country level, even though field support buy-ins are within the global Theory of Change, Missions may develop their own theories of change. Thus, without a review of all country-level theories of change, there are no generalizable statements about a need for adaptation which can be derived from this review. However, several KIs noted that it is challenging to 19 MOMENTUM (Moving Integrated, Quality Maternal, Newborn, and Child Health and Family Planning and Reproductive Health Services to Scale), Annual Program Statement, 2019. “I understand the rationale behind the Suite of awards given the optics of the previous 20–25 years. However, I find that splitting up the award has created inefficiencies that cannot be overcome because of the structure.” GH EvaLS MOMENTUM Mid-Term Process Evaluation / 42 reconcile the Theory of Change at the global level with different theories of change at the country level, even though guidance was provided on how to ensure integration. The same limitations for the global Theory of Change, though, apply at the country level in that partnering, coordination, and collaboration are labor-intensive. For example, as shown in the Indonesia country case study which has MCGL and MPHD implementing activities, the combination of involving the public and private health sectors has been complex. However, USAID/Indonesia believed having Jhpiego as part of each award would lead to easier coordination of their activities and this has been demonstrated during implementation. STRENGTH OF EVIDENCE: MEDIUM Sub-question 2D: Are additional adjustments needed to achieve the vision for this Suite of awards? If so, what? Finding 2D.1 (High): Per KIs, a stronger vision and guidance are needed from USAID in terms of coordination and collaboration, along with additional light “touchpoints.” Key IP informants were nearly unanimous (greater than 80%) in stating that USAID/W had not sufficiently communicated with them the expectations for how the six awards were to function as a Suite. This issue was compounded by the perceived lack of a unified vision for partnering among the six AORs and the fact that some awards have had multiple AORs since their inception. Further, KIs stated that, because of unclear and evolving partnering expectations for many of the awards it was very challenging to determine the right level of effort to dedicate to interacting both within the Suite and with external stakeholders (USAID/W, USAID Missions, bilateral awards, etc.). A high level of effort for coordination was also noted by survey respondents as shown in Figure 5: Level of Effort Needed to Manage and Access the MOMENTUM Awards below; however, the number of respondents was fairly low (n = 10). “The project was designed well but the Missions have not necessarily followed the design. Many of the things Missions are asking for are not six sub-projects or a theory driven project.” — Key informant interview GH EvaLS MOMENTUM Mid-Term Process Evaluation / 43 Figure 5: Level of Effort Needed to Manage and Access the MOMENTUM Awards N=10 Many stakeholders suggested that USAID first needs to clarify its expectations and develop clearer guidance. Additionally, several IP KIs noted that there had been few opportunities, other than Share Fairs, for the various awards to interact, and that they need a systematic method for touching base with each other (although, as shown in Table 4, there are several types of coordination mechanisms). 10% 50% 40% How would you rate the level of effort required to manage and access the various awards under the MOMENTUM Suite? Low Medium High Not Applicable GH EvaLS MOMENTUM Mid-Term Process Evaluation / 44 Selected KI feedback to the question “Are there additional adjustments needed to achieve the vision for this Suite of awards? If so, what?” “It is hard to figure out ‘where the buck stops’ because all the AORs are peers. We don’t know who is in charge. It would be helpful to have someone in charge overall who provides leadership in where MOMENTUM is going, what the key messages are, etc. For something as resource intensive as MOMENTUM we need more directive leadership from USAID.” “We need to invest in clarity and shared expectations across the USAID AOR teams and how they should function as a Suite and how is that different from having six separate awards? Are they six awards or one award that has the same results?” “The relationships to other MOMENTUM awards should be clarified at the outset by D.C. and the Missions and sit together to define the relationship. It is not clear how to meet the expectations for coordination.” “I wonder if ‘we are all singing from the same Theory of Change.’ I know there is so much we could be learning from the other awards. Coordination across the awards needs to be pushed by USAID.” “I’m not sure the individual awards see that it is important to collaborate, and they need to understand the value of having ‘One MOMENTUM.’ We need more clarity on the value of One MOMENTUM.” “It would be helpful for USAID to look at how they could work together with IPs to strategize and plan how to implement the vision.” “There is a flaw in the design that the project directors don’t talk to one another often enough.” “Six months into the project we had a large MOMENTUM meeting to coordinate but we haven’t had another one since.” “More coordination mechanisms with space for a quarterly Project Directors’ meeting with AORs. There was no movement on actions recommended during the February AOR meeting-it seems like the AORs decided not to. We function in an ad hoc way to coordinate and ensure collaboration across work plans. There are unspoken expectations about coordination, but we have to plan this ourselves.” STRENGTH OF EVIDENCE: HIGH FINDING 2D.2 (High): Because MIHR works in fragile settings and emphasizes humanitarian development interventions, its ability to align with other awards has been especially limited. As noted in some other findings, MIHR works in operating environments where there are both human-made and environmental challenges. Most of the countries in which it works are not as far along the development spectrum as those where the other four awards with in-country presence are implementing activities. KIs observed that it is difficult for MIHR to align with the other awards, as its ability to align and partner within the suite is “When you have awards that do such different things in places like Mali, does it make sense to link them together? I don’t know.” — Key informant interview GH EvaLS MOMENTUM Mid-Term Process Evaluation / 45 naturally constrained — especially because it has faced several challenges that have delayed implementation (MIHR was awarded in May 2020). Indeed, some KIs questioned whether MIHR should belong within the Suite, given its inherently different goals of addressing the humanitarian-development nexus and working in areas with prolonged crises and inherent fragilities versus the goals of localization and sustainability. STRENGTH OF EVIDENCE: HIGH Evaluation Sub-question 2E: How has the staggered timing of the awards start-up affected coordination and collaboration across the awards, with MKA and with Missions? Finding 2E.1 (High): In general, the staggered timing of the awards and their start-ups significantly and negatively affected the MOMENTUM Suite. The MOMENTUM awards were made in the following order: 1. MCGL December 20, 2019 2. MKA December 26, 2019 3. MIHR May 22, 2020 4. MPHD June 17, 2020 5. M-RITE July 23, 2020 6. MSSFPO September 21, 2020 In other words, there was a nine-month span of time between the first and last award and, further, MCGL was awarded slightly before MKA (which was awarded during the end-of-year holiday season). For MCGL, per KIs, the timing of the awards was particularly difficult as it had already developed several core documents (a learning agenda, a MEL plan, etc.) and then had to retrofit those documents to the overarching frameworks that MKA was tasked with developing. Of the 11 KIs (USAID/W and IP headquarters staff) who commented on the staggering of the awards, 10 said that the effects were largely negative. Some of those effects included difficulties and delays in implementation, fragmentation of coordination, possible confusion amongst Missions and in￾country counterparts, and alignment of technical activities. STRENGTH OF EVIDENCE: HIGH Finding 2E.2 (Medium): Missions were largely unaffected by the staggered timing. All Mission KIs were asked to comment on the staggered timing of the MOMENTUM awards. They noted that they were generally unaware of any timing issues and instead focused on determining how to proceed with their specific procurements (e.g., choosing between a bilateral award or a buy-in). The only challenges related to some initial confusion about which award would address which issues, and whether and how to proceed with FP/RH and MCHN procurements. In other words, respondents were unclear about whether to buy into a current award, wait for a possible upcoming award, or design a bilateral award. However, respondents also noted that, as MOMENTUM awards continued to be made and rolled out, USAID/W made several efforts to clarify any issues, which they appreciated. “The consensus was that the first year of the program suffered because the awards were staggered.” — Key informant interview GH EvaLS MOMENTUM Mid-Term Process Evaluation / 46 STRENGTH OF EVIDENCE: MEDIUM Evaluation Question 3: Does the MOMENTUM Suite address the needs and expectations of IP individual awards, USAID Missions, MOMENTUM AOR management teams, Technical Offices, and other Agency leadership? Finding 3.1 (Medium): Technically, MOMENTUM generally meets expectations (see Finding 3A.1 for further details). KIs were in general agreement that the MOMENTUM Suite meets their technical needs due mainly to the breadth and depth of IP participants. However, survey respondents and the country case study results require some caveats. Finding 3A.1 provides additional details. STRENGTH OF EVIDENCE: MEDIUM Finding 3.2 (High): Strategically, MOMENTUM does not generally meet expectations. Many of the evaluation questions and sub-questions touch upon some of the underlying premises and assumptions for the strategy behind the MOMENTUM Suite. As demonstrated in multiple findings, strategically MOMENTUM is not generally meeting expectations around coordination, collaboration, partnerships, holistic approaches and storytelling, and others. While the findings for those questions and sub-questions provide specific details, one starting point is the responses shown in Figure 6: The Utility of Grouping MOMENTUM Under One Mechanism. Only 40 percent of the 69 survey respondents found the grouping useful (16 percent found it somewhat useful, 22 percent had no opinion, and 22 percent had a less favorable view). These results are actually better than those provided by IP and USAID Mission KIs, who did not see much cohesion among the awards and provided limited examples of MOMENTUM functioning as a cohesive Suite without substantial involvement by USAID. Most partnering examples, including those provided as part of the Indonesia country case study, involve specific technical discussions or tasks done at the country level, as time zone differences made the involvement of headquarters staff challenging. GH EvaLS MOMENTUM Mid-Term Process Evaluation / 47 Figure 6: The Utility of Grouping MOMENTUM Under One Mechanism N=69 STRENGTH OF EVIDENCE: HIGH Sub-question 3A: Are there technical or functional skills not available or accessible in the current configuration? Finding 3A.1 (Medium): There are, perhaps, some gaps in technical and functional skills within the MOMENTUM Suite; however, those gaps may be country- and stakeholder-specific. The MOMENTUM Suite of six awards includes more than 30 IPs (a few IPs are on multiple awards) at both the prime and sub level. The IPs range from academic institutions (Johns Hopkins Bloomberg School of Public Health), global management consulting firms (McKinsey & Company), large international non-profits (IntraHealth International), technology companies (Premise Data), and local organizations (Ubora Quality Institute) to smaller organizations (CORE Group). As such, it would be unlikely that there would be technical and/or functional gaps within the Suite, and this was reflected in responses by KIs. More than 60 percent of KIs across all stakeholder groups who were asked if there were any gaps in the MOMENTUM Suite responded that, in general, most areas were well covered; however, they did have recommendations for future PRH and/or MCHN awards (see EQ 4). This finding among KIs was not quite matched by the survey respondents or the findings of the Indonesia country case study. As shown in Figure 7: Unmet Technical Needs Within MOMENTUM, respondents (N=70) were nearly evenly divided as to whether they believed there were unmet technical needs within the MOMENTUM Suite. Similarly, in the country case study, KIs noted some gaps in the technical and/or functional skills within the two MOMENTUM awards (MCGL and MPHD) implementing within the country. For example, respondents stated that international technical expertise is needed to instill comprehensive understanding of concepts and practical business processes for private health providers, 3% 9% 10% 16% 40% 22% How useful has it been to have the Suite of awards grouped under one mechanism? Useless Somewhat useless Neutral Somewhat useful Useful No Opinion GH EvaLS MOMENTUM Mid-Term Process Evaluation / 48 as well as a need for a local expert to formulate concepts and explore potential instruments of credit incentives/subsidy and tax exemptions for purchasing medical equipment. Thus, there may be some perceived gaps for some stakeholder groups and, perhaps, within some country contexts (e.g., where a limited number of MOMENTUM awards are being implemented). Figure 7: Unmet Technical Needs Within MOMENTUM N=70 STRENGTH OF EVIDENCE: MEDIUM Finding 3A.2 (Medium): There was a mixed level of knowledge and understanding among KIs as to where certain technical and functional skills are available within the Suite and how to access those skills. This finding is primarily derived from KIIs with IPs (at both the headquarters and country levels) and with USAID Mission staff. A sizeable minority of IP staff (approximately 33 percent) who were asked about the skills available within MOMENTUM noted that they were unfamiliar with all of the available capacities of the various organizations at the award level, but especially in regard to the other IPs. It was unclear whether this had any discernible effect on their ability to implement, though. All USAID Mission staff noted that, at the beginning of the rollout of MOMENTUM, there was substantial confusion about which awards would provide which skills and/or whether future awards would provide needed technical and functional skills; however, they also noted that much of that confusion had been mitigated through outreach from USAID/W to provide guidance regarding the Suite. STRENGTH OF EVIDENCE: MEDIUM Sub-question 3B: Does the Suite design inhibit or facilitate meeting USAID’s and individual award needs? Finding 3B.1 (High): The Suite neither inhibits nor facilitates meeting USAID’s (USAID/W and Missions) and individual award needs as it is not functioning as a Suite. As noted in other findings, MOMENTUM currently functions only nominally as a Suite. While there are some examples of coordination and collaboration, they have been limited. As noted by the majority of KIs across all stakeholder groups, whether MOMENTUM can be considered a holistic approach is still to be 34% 36% 30% Are there technical needs that are not met by MOMENTUM? Yes No No Opinion GH EvaLS MOMENTUM Mid-Term Process Evaluation / 49 determined. For USAID Missions, the issues around MOMENTUM functioning as a Suite appear secondary, as most Mission-level KIs noted their focus is on how easily they can place buy-ins into the field support mechanisms. These observations by KIs are supported by the survey findings (see Figure 8 and Figure 9), although the number of respondents for each question was fairly low. As shown in Figure 8: Survey Respondents’ Satisfaction with MOMENTUM, half of survey respondents were somewhat satisfied with MOMENTUM, with nearly equal numbers highly satisfied and highly dissatisfied. And, as shown in Figure 9: MOMENTUM’s Responsiveness by Survey Respondents, most survey respondents (67 percent) believed that MOMENTUM has been somewhat responsive to their needs. Figure 8: Survey Respondents’ Satisfaction with MOMENTUM N=12 Figure 9: MOMENTUM’s Responsiveness by Survey Respondents 9% 8% 25% 50% 8% Satisfaction with MOMENTUM Highly Dissatisfied Somewhat dissatisfied Neutral Somewhat satisfied Highly satisfied 16% 67% 17% How responsive is the MOMENTUM Suite to your needs and goals? Very responsive Somewhat responsive Not so responsive Not at all responsive Don't know N=6 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 50 STRENGTH OF EVIDENCE: HIGH Finding 3B.2 (Medium): The overlaps in MOMENTUM award functions require clarification. A limited number of stakeholders, primarily among USAID Mission staff and global partners, are confused about the different technical and functional areas of each award. For example, a few stakeholders were confused about what falls under “humanitarian assistance” as part of MIHR, as well as the different responsibilities for communication between MKA and other awards. This lack of clarity among partner organizations and USAID Missions stands in direct contrast to responses from KIs from USAID/W and IPs, almost all of whom noted distinct roles and responsibilities between awards and little overlap. STRENGTH OF EVIDENCE: MEDIUM Evaluation Sub-question 3C: Does the MOMENTUM Suite address key needs of global partners (e.g., BMGF, UNICEF, WHO) through its role in global forums and partnerships? How has the design of the Suite helped or hindered this? Only five global partners participated in KIIs, and only five global partners of the potential 23 survey respondents (8.9 percent of all respondents, or a 22 percent response rate for global partners) completed the survey. Therefore, all findings related to global partners should be qualified by the strength of the underlying evidence, which is rated medium at best. Finding 3C.1 (Medium): It is too early to determine whether MOMENTUM addresses the needs of global partners. All global partner KIs agreed that MOMENTUM is noted for bringing certain technical issues (maternal mental health, safe surgery, private sector engagement, etc.) to the forefront of the global health agenda. However, none could provide a conclusive answer regarding the definitive contributions that MOMENTUM has provided via global forums and partnerships. All stated that, while they anticipated that MOMENTUM might serve a catalytic role in the future, it is currently too early to tell whether this will occur. However, for the most part, survey respondents noted that MOMENTUM has contributed to USAID’s leadership role in advancing MNCH and FP/RH (see Figure 10: MOMENTUM’s Contribution to USAID’s Global Technical Leadership Role). “MOMENTUM confuses the local stakeholders. They have all these activities and the stakeholders overlap, but some stakeholders are different. Even within the Mission people are confused about the awards and why they have similar names, but are doing different things.” — Key informant interview GH EvaLS MOMENTUM Mid-Term Process Evaluation / 51 Figure 10: MOMENTUM’s Contribution to USAID’s Global Technical Leadership Role N=6 STRENGTH OF EVIDENCE: MEDIUM Finding 3C.2 (Medium): To date, global partners have had limited engagement with MOMENTUM. Three of the five KIs (60 percent) confirmed that they had engaged with some MOMENTUM awards (primarily MCGL and MKA), and they provided examples of that engagement (technical working groups, conferences, and webinars). All acknowledged, though, that their engagement had been fairly limited. Most attributed the limited interactions to the newness of the awards, frequency of engagement forums, and IPs’ need to seek USAID approval prior to engaging. STRENGTH OF EVIDENCE: MEDIUM Finding 3C.3 (Medium): Per global partners, there are both technical and learning gaps within MOMENTUM. As shown in Figure 11: Global Partners’ Responses to MOMENTUM’s Technical Breadth, three of the five survey respondents (60 percent) mentioned technical needs that the MOMENTUM Suite is not meeting; however, they provided few details regarding those gaps. Regarding future MNCH and FP/RH issues to be addressed, respondents (survey participants and KIs) cited several potential technical areas without achieving a clear consensus. 50% 17% 33% Has the design of MOMENTUM enhanced/amplified local technical leadership? Yes No Don't know GH EvaLS MOMENTUM Mid-Term Process Evaluation / 52 Figure 11: Global Partners’ Responses to MOMENTUM’s Technical Breadth Additionally, a theme that emerged in qualitative survey comments from global partners is MOMENTUM’s approach to learning. One respondent noted, “From engaging with MOMENTUM grantees in various contexts, we found that the learning and knowledge frameworks in particular remain based on a ‘container’ view of knowledge, with learning efforts primarily transmissive.” The same respondent commented that this approach was “… disempowering for local practitioners and implementers” and that more modern approaches and frameworks for learning are needed to adhere more closely to the principles of MOMENTUM. Another respondent stated that MOMENTUM is missing mechanisms for: 1. Dissemination and knowledge sharing that reach large numbers of subnational staff; 2. Bottom-up generation of new knowledge and expertise grounded in local practice; and 3. Supporting local practitioners in implementing corrective actions to tackle root causes of their priority challenges using only locally available resources. STRENGTH OF EVIDENCE: MEDIUM Evaluation Question 4: What are the emerging trends and needs in FP/RH/MNCH that might be addressed in the current or future awards? Finding 4 (Medium): There was no clear consensus among IPs, USAID staff, or country or global partners (survey respondents and KIs) regarding emerging trends and needs in FP/RH/MNCH that current or future awards might address. The trends mentioned included the impact of climate change, PHC, integration of services, maternal mental health, localization, gender, youth, self-care, nutrition, urbanization, global health security, universal health coverage, health care financing, health systems strengthening, adaptive learning, a post–Sustainable Development Goal agenda, engaging the private sector, how to move humanitarian efforts into the development space, and digital transformation technologies. One informant stated, “COVID has helped leapfrog digital tools and we need a digital transformation.” GH EvaLS MOMENTUM Mid-Term Process Evaluation / 53 KIs’ Responses Regarding Potential Future FP/RH/MNCH Needs to Be Addressed “Rapid urbanization and the expansion of slum areas increases the need for MNCHRH services.” “PHC systems are the backbone of the pandemic response. Are our workplans reflective of that?” “The money is in global health security and the only priority for this administration. The next MOMENTUM needs to focus on global health security.” “Maternal mental health is an emerging and important field and a neglected area.” “Maternal neonatal health is being pushed to the margins in the movement towards universal health coverage. What role can MOMENTUM play in the conversation?” “What is the innovation ecosystem for technology and immunizations?” STRENGTH OF EVIDENCE: MEDIUM GH EvaLS MOMENTUM Mid-Term Process Evaluation / 54 5. CONCLUSIONS Evaluation Question 1: DESIGN The six diverse MOMENTUM awards are not currently functioning as a Suite as the MOMENTUM Theory of Change envisioned, and there is a lack of clarity regarding how the awards are supposed to work together. The benefits of attempting to leverage relationships to improve the efficiency, effectiveness, and/or scale of programming across the Suite of Awards, with 37 IPs, in 34 countries in varying stages along the development continuum, in seven regions of the world have yet to be realized. While there are examples of successful collaboration and coordination among two or three awards in select countries, the level of effort and project management time required at the headquarters and country levels for cross-MOMENTUM collaboration, in addition to fulfilling workplan deliverables, is significant and costly. Collaboration across the suite requires already limited core funding to be spent on coordination rather than implementation. The burden is especially felt by awards with smaller budgets and staff. The sheer number of IPs, technical advisors, and inputs contributes to delays in implementation and slows decision-making. The key factors to effective leveraging of relationships in countries where more than one MOMENTUM award is active are 1) ongoing Mission involvement in planning, collaboration, and coordination and a 2) clear definition of roles that outline implementation at 3) what level(s) of the health system and technical and geographic areas of focus. Collaboration between awards does not happen naturally. MKA has produced and is responsible for some valuable tools and initiatives such as MAKLab, the Share Fair, the HUB, digital health expertise, adaptive management training, and the MOMENTUM working groups. However, the full extent of level of effort required to collaborate with MKA and respond to their requests was not anticipated by the awards in their proposals and workplans. The review processes for KM and communications products are confusing and lengthy. The additional layers of coordination and review of communications is a disincentive to producing communications at the global and country levels to tell the story of MOMENTUM. Evaluation Question 2: ADAPTATIONS The MOMENTUM Suite of Awards, for the most part, adapted exceptionally well to the COVID-19 pandemic during its implementation. As noted, all of the awards were made either slightly before the pandemic began or after it had been recognized. That MOMENTUM has been able to operate during those challenging times with only nominal delays in its implementation attests to the flexibility and commitment of its IPs. Indeed, as noted in the findings, many lessons learned and best practices have enabled MOMENTUM to ensure its continuity during COVID-19. These should be shared widely, not just within MOMENTUM but externally as well. Further, that M-RITE was able to absorb COVID-19 funds and respond to the pandemic is notable; however, that response has come at a cost to its original core mandate and ability to collaborate within the Suite. Similarly, MIHR has faced challenges in its ability to coordinate and collaborate; however, those issues are particular to an award focused on a humanitarian-development nexus versus the other MOMENTUM awards, which work with countries farther along the development continuum. This raises the issue of whether it makes sense to have MIHR within the Suite, or how the Theory of Change needs to be viewed in light of MIHR’s unique and challenging operating environments. At the very least, this GH EvaLS MOMENTUM Mid-Term Process Evaluation / 55 issue (a health development award working in fragile settings within the humanitarian-development nexus versus progress toward localization and sustainability) should be considered prior to any future designs. Overall, though, MOMENTUM’s Theory of Change, excluding the issues with MIHR, seems to serve the Suite fairly well technically at the headquarters level. However, there are several caveats to that statement. First, as noted in the findings, USAID Missions are able to develop their own theories of change within the overall MOMENTUM Theory of Change; whether those are aligned and functioning needs further examination. Second, several of the foundational assumptions, such as collaboration, coordination, holistic storytelling, partnering, and learning for MOMENTUM to function as a Suite have yet to be fulfilled. For instance, there are limited examples of continuous learning within the Suite that happen outside of the periodic and labor-intensive Share Fairs. This issue seems to be particularly acute at the country level. Finally, the staggered timing of the awards had a substantial and negative impact on the ability of the Suite to function cohesively, although this had less of an impact at the Mission/country level. While a few specific examples were given of problems with alignment and coordination (the development of an M&E framework, knowledge products, etc.), more generally, the six awards have not yet realized the goal of functioning as a Suite. Rather, they are perceived and operate as six individual awards that occasionally work together on specific technical issues. KIs and survey respondents made several suggestions as to how to improve this situation, and those ideas are further expanded upon in the recommendations (Section 6). Evaluation Question 3: STAKEHOLDER SATISFACTION Stakeholder satisfaction needs to be seen through multiple lenses as there are multiple stakeholders involved in the MOMENTUM Suite, as well as two primary parameters — technical and strategic — that formed the basis for determining satisfaction. From a technical viewpoint, MOMENTUM in general satisfies stakeholder needs; however, the few survey respondents from partner organizations believed there was room for improvement. The response from the partner organizations may be due to their limited engagement to date with MOMENTUM, as noted by the KIs, as well as the evaluation coming too early in implementation to form a definitive opinion. The overall level of satisfaction from most stakeholders can be attributed to the breadth and depth of the IPs within the MOMENTUM Suite. Although there may be some additional need for clarification among Missions and partner organizations about which award is responsible for which area, the technical gaps within MOMENTUM are few, and Missions are appreciative of their ability to buy into the awards fairly easily. From a strategic point of view, MOMENTUM is not meeting expectations for functioning as a holistic Suite. This was revealed across multiple findings. As noted, the IPs within MOMENTUM, whether at the prime or sub level, are fundamentally competitors, and the additional and more significant factors of a lack of clear guidance for coordination and limited bandwidth to do so have substantially undermined those efforts. The idea underpinning MOMENTUM — that IPs would organically (or with some slight motivation) coordinate and collaborate their efforts — was naïve. Without a strong inducement, either positive or negative, there has not been a clear incentive for the awards to function synergistically. Compounding this have been unclear expectations across the awards as to what the coordination and collaboration entails and the subsequent inability to plan for varying efforts. In both within the survey and the interviews, IPs strongly requested that USAID develop some standard internal and external guidance. This is further discussed in the recommendations. GH EvaLS MOMENTUM Mid-Term Process Evaluation / 56 Evaluation Question 4: FUTURE TRENDS As stated in the findings (Section 4), there was no clear consensus among the IPs, USAID staff, or country or global partners regarding emerging trends and needs in FP/RH/MNCH that might be addressed in current or future awards. Among the plethora of emerging trends mentioned by some informants was PHC. A tenet of health care financing and sustainability is that essential basic health services are delivered at the lowest unit cost at the lowest levels of the health system. PHC can be delivered in all settings — urban, rural, and fragile — in both public and private facilities. PHC centers provide most services currently prioritized in the MOMENTUM Suite of Awards and include FP, RH, MNCH, immunizations, nutrition, and others. PHC centers can be formal parts of a referral system to transfer women, children, and their families to the next level of care when needed. They can also include or be important links to community outreach and social and behavior change interventions. Many PHC centers have birthing centers that offer 24/7 obstetric care by safe birth attendants, contributing to reductions in maternal and neonatal mortality and morbidity. A focus on PHC would also contribute to improving global health security by offering services to detect, manage, and refer care for infectious diseases such as COVID-19 at the basic community level — services that are accessible to vulnerable populations. Finally, the impact of climate change and ways to mitigate its adverse impact on MNCH, how to effectively utilize digital tools to enhance the capacity of the public and private sectors to deliver essential MNCH care and monitor and improve outcomes, and global health security are also critical emerging trends and needs. GH EvaLS MOMENTUM Mid-Term Process Evaluation / 57 6. RECOMMENDATIONS Priority Recommendations No. Recommendations Description Responsible Parties Related Finding(s) 1 Develop explicit internal and external guidance for coordination, collaboration, and partnering This guidance would be aimed at: 1) USAID/W AORs, 2) USAID Activity Managers, and 3) IPs. It should define what are considered collaborative efforts, 4) align and/or set the level of efforts expected based on corresponding budgets and staff, and 5) delineate the overall coordinative and learning efforts versus those covered by MKA. At the country level, it should provide criteria for how to define the “convening” role and clarify the specific coordination mechanism(s). Given that most previous examples of well￾coordinated efforts have occurred at the country level around specific technical issues, targeting this guidance first at countries with multiple MOMENTUM awards, where possible technical linkages could be encouraged, should be given priority. USAID/W 1E.1, 2B.3, 2D.1, 3A.2, Annex 3 (Indonesia case study) 2 Review and revise the MOMENTUM MEL framework(s) The current overarching MOMENTUM MEL framework section on collaboration and coordination needs to be further developed or revised, with better indicators to provide greater incentive for the Suite of Awards to work together. This may need to be applied at the individual award level. USAID/W and MKA (primary) Other IPs (secondary) As above (No. 1) and 1A.3, 1B.2, and 1D.1 3 Streamline strategic communications The strategic communications guidance document and approval process needs to reviewed and revised to ensure that information dissemination does not become a bottleneck to implementation. USAID/W and MKA 1A.2, 1A.3, 3B.2 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 58 Priority Recommendations No. Recommendations Description Responsible Parties Related Finding(s) 4 Provide additional learning opportunities, including those outside of MKA’s responsibilities USAID needs to collectively negotiate with all IPs the budgets and workplans (and level of effort) for additional learning opportunities and corresponding activities to ensure that there is sufficient time and resources for IPs to learn from each other. Further, MKA’s role should be clarified with all IPs such that it is not put in the position of serving as a gatekeeper to cross-IP learning. USAID/W (AORs) and all IPs 1.4, 1A.2, 2B.2, 2B.2, 2B.3, 3C.3 For Consideration No. Recommendations Description Responsible Parties Related Finding(s) 5 Increase publicity and dissemination for COVID-19 lessons learned Given MOMENTUM’s outstanding adaptation and response to COVID-19, lessons learned need to be shared widely, both internally to USAID and with external partners. MKA and USAID/W should develop COVID-19– specific events (conferences, workshops, publication in peer￾reviewed journals) to ensure information generated from MOMENTUM is acknowledged and used. MKA and USAID/W 1D.1, 2A.1, Annex 3 (Indonesia case study) 6 Appoint a full-time USAID MOMENTUM coordinator Rather than the two MOMENTUM coordinators from the PRH and MCHN offices who have other demanding job duties, USAID/W should consider appointing one individual whose sole responsibility would be to ensure and oversee both internal (AORs and Activity Managers) and external (IP) coordination efforts. This person, ideally, would not sit within either the PRH or MCHN offices to ensure technical and managerial neutrality. USAID/W 1A.3, 1B.2, 1D.1, 1E.1, 2B.3, 2D.1, 3A.2 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 59 For Consideration No. Recommendations Description Responsible Parties Related Finding(s) 7 Consider MIHR and M-RITE specific adjustments to the Theory of Change and MEL framework Because of their respective special circumstances, USAID/W should consider whether both the Theory of Change and MEL framework need to be revised to account for MIHR’s and M￾RITE’s challenges. USAID/W 2.1.1, 2.1.2, 2D.2, 3B.2 8 Determine whether having Mission-level theories of change is in MOMENTUM’s strategic interests Reviewing every Mission-level theory of change for MOMENTUM was outside the scope of this evaluation. However, as noted in the findings, stakeholders were uncertain whether having these country-specific theories of change was beneficial to MOMENTUM’s strategic goals. This question needs further examination. USAID/W 1D.1, 2C.1, 2C.2, 2D.1 9 Continue outreach to Missions and partners to ensure clarity on the MOMENTUM Suite and individual awards These efforts are already underway. However, based on stakeholder feedback, periodic refresher outreach is needed. USAID/W (and IPs, if appropriate) 1.1, 1.4, 2D.1, 2E.2, 3B.2 10 Prioritize PHC and climate change for possible future investments/awards for PRH and MCHN Although there was no overwhelming consensus on PRH and MCHN emerging and/or future needs, several KIs mentioned two issues: PHC and climate change. PHC could address a multitude of additional emerging trends mentioned by stakeholders, such as integration, urbanization, rural health care, localization, health system strengthening, resilience, self-care, youth services, and nutrition. Other critical emerging trends and needs include the impact of climate change and ways to mitigate its adverse impact on MNCH; how to effectively utilize digital tools to enhance the capacity of the public and private sectors to deliver essential MNCH care and monitor and improve outcomes; and global health security. USAID/W 4 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 60 ANNEX 1: SCOPE OF WORK GH EvaLS MOMENTUM Mid-Term Process Evaluation / 61 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 62 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 63 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 64 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 65 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 66 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 67 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 68 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 69 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 70 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 71 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 72 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 73 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 74 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 75 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 76 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 77 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 78 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 79 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 80 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 81 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 82 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 83 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 84 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 85 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 86 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 87 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 88 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 89 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 90 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 91 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 92 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 93 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 94 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 95 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 96 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 97 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 98 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 99 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 100 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 101 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 102 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 103 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 104 ANNEX 2: ASSIGNMENT DESIGN MATRIX Assignment Question Data Sources Data Collection Methods Data Analysis Methods 1 Design Hypothesis: Have the awards been able to leverage their relationships with other MOMENTUM awards to benefit their work, e.g., to improve the efficiency, effectiveness, and/or scale of programming? If so, how? A. What has been the cost and benefit (to USAID and IPs) of coordinating with MKA and across the Suite of awards? What have been the barriers and facilitators to achieving this? B. What has been the cost and benefit of awards coordinating with each other (exclusive of MKA)? C. What has been the cost and benefit of in-country coordination (to USAID, to IPs, to country partners)? D. Have awards been able to create new partnerships and ways of working within the consortium? E. What adaptations (e.g., changes in systems/processes) would lead to more efficient and/or more effective coordination? Award reporting documents Key informants (Interviewed individually or in small groups) Desk review Key informant interviews (KIIs) Group interviews (GIs) Synthesis of background documents Analysis of relevant secondary data Content/thematic analysis of KII/GI notes and transcripts Triangulation of data/information from all sources GH EvaLS MOMENTUM Mid-Term Process Evaluation / 105 Assignment Question Data Sources Data Collection Methods Data Analysis Methods 2 Adaptation: What operational shocks and stressors are driving programmatic adaptations? A. How has the Suite collectively, and as individual awards adapted to the realities of working in dynamic country contexts, particularly the ongoing effects of the COVID-19 pandemic? B. What adaptations (e.g., changes in systems/processes) would lead to more efficient and/or more effective shared learning? C. Do any of these adaptations require adjustments to the development hypothesis and/or key assumptions of the MOMENTUM award Suite and/or implementation at the country or global level? D. Are additional adjustments needed to achieve the vision for this Suite of awards? If so, what? E. How has the staggered timing of the awards startup affected coordination and collaboration across the awards, with MKA and with Missions? Award reporting documents Key informants (Interviewed individually or in small groups) Desk review KIIs GIs Synthesis of background documents Analysis of relevant secondary data Content/thematic analysis of KII/GI notes and transcripts Triangulation of data/information from all sources GH EvaLS MOMENTUM Mid-Term Process Evaluation / 106 Assignment Question Data Sources Data Collection Methods Data Analysis Methods 3 Stakeholder Satisfaction: Does the MOMENTUM Suite address the needs and expectations of IP individual awards, USAID Missions, MOMENTUM AOR management teams, Technical Offices, and other Agency leadership? A. Are there technical or functional skills not available or accessible in the current configuration? B. Does the Suite design inhibit or facilitate meeting USAID’s and individual award needs? C. Does the MOMENTUM Suite address key needs of global partners (e.g., BMGF, UNICEF, WHO) through its role in global forums and partnerships? How has the design of the Suite helped or hindered this? Award reporting documents Key informants (Interviewed individually or in small groups) Survey respondents (stakeholders) Online survey Desk review KIIs GI Synthesis of background documents Analysis of relevant secondary data Content/thematic analysis of KII/GI notes and transcripts Quantitative analysis of the online survey responses and qualitative analysis of narratives, as applicable. Triangulation of data from all sources 4 Future Trends: What are the emerging trends and needs in FP/RH/MNCH that might be addressed in the current or future awards? Literature review Key informants (Interviewed individually or in small groups) Desk review KIIs GIs Synthesis of literature review Content/thematic analysis of KII/GI notes and transcripts Triangulation of data/information from all sources GH EvaLS MOMENTUM Mid-Term Process Evaluation / 107 ANNEX 3: INDONESIA COUNTRY CASE STUDY ABSTRACT Two of the MOMENTUM Suite of Awards in Indonesia, MOMENTUM Country Global and Leadership (MCGL) and MOMENTUM Private Healthcare Delivery (MPHD), coordinate to support the Government of Indonesia (GOI) in achieving as a priority a reduction in maternal mortality by strengthening the maternal and neonatal care systems that involve public and private health facilities. This case study, part of the mid-term process evaluation for the MOMENTUM Suite of Awards, assesses how both programs are collaborating, learning, and adapting in Indonesia to generate insights on lessons learned. The study used a mixed methods approach to answer the Evaluation Questions. Key informant interviews (KIIs) were conducted with 24 respondents from MCGL, MPHD, USAID, and national and subnational stakeholders. A desk review gathered data and triangulated findings. This study’s limitations included some respondents’ limited knowledge of the MOMENTUM Suite of Awards and/or previous USAID maternal, newborn, and child health (MNCH) projects in Indonesia. Therefore, the evaluator focused on key MOMENTUM Indonesia staff with in-depth understanding of the previous and current implementation of USAID maternal and child health (MCH) programs, including USAID’s partnership with the Ministry of Health (MOH). The national and subnational stakeholders acknowledged that the MCGL and MPHD programs provide benefits for Indonesia in achieving its national priorities. There has been close internal coordination between MCGL and MPHD, as well as with other stakeholders. As MOMENTUM focuses on the continuum of the development of local partner institutions to achieve this goal, it is recommended that MCGL and MPHD 1) establish a secretariat embedded in the MOH to create intensive relationships with related directorates and centers in the MOH as core activities grow; 2) hire experts to clarify business processes in private health facilities and explore potential low-rate credit instruments, subsidies, and tax exemptions for public health facilities; and 3) plan joint monitoring visit(s) for key stakeholders at the central level to improve their understanding of implementing programs in the field. BACKGROUND MCGL Indonesia focuses on improving the quality of primary and referral services for maternal and newborn health (MNH). This four-year USAID/Indonesia buy-in builds on the work of previous USAID￾supported MNCH programs, including Jalin and Expanding Maternal and Newborn Survival (EMAS), that supported the GOI priority of strengthening maternal and neonatal care at the national and subnational levels. Additionally, MPHD, a four and one-half-year USAID/Indonesia buy-in, works to enhance the engagement and effectiveness of the private health care sector as a sustainable partner to provide quality maternal and newborn services and contribute to reductions in the maternal mortality rate (MMR) and neonatal mortality rate (NMR) in Indonesia. The MCGL rapid assessment in the province of Nusa Tenggara Timur (NTT) found large gaps in the capacity of human resources for health, including in the event distribution of health workers and limited health facilities supported for MNH services in public and private health facilities. MCGL supports 300 primary health care centers (puskesmas) and district hospitals across all 22 districts of the province, as well as the provincial hospital. GH EvaLS MOMENTUM Mid-Term Process Evaluation / 108 MPHD Indonesia builds on previous and current efforts under the USAID-supported Jalin project, EMAS Program, Health Financing Activity, Tuberculosis Private Sector, and Health Policy Plus projects. It also collaborates with MCGL to support the GOI priority of strengthening MNH care at the national and subnational levels. The MPHD Program aims to reduce disparities in maternal and newborn mortality in 25 districts and cities in North Sumatra, DKI Jakarta, Banten, South Sulawesi, and East Java by modeling and leveraging private sector stakeholder resources and then extending the successful approaches to other areas. In Year 2, in addition to covering all 22 districts in NTT province, MCGL added targeted technical assistance in two districts co-located with MPHD to support the modeling of cohesive coordination between the projects to support to the GOI in achieving the ultimate goals of reducing MMR and NMR. OBJECTIVES AND EVALUATION QUESTIONS As part of the Mid-Term Process Evaluation for the MOMENTUM Suite of Awards, this case study presents an in-depth example of how the MOMENTUM MCGL and MPHD programs are collaborating, learning, and adapting to generate insightful lessons to contribute to the global MOMENTUM learning agenda and to synthesize the learning experience for technical and non-technical audiences. This case study also provides recommendations to accelerate the implementation of MOMENTUM programs in Indonesia. The main Evaluation Questions are: 1. Have the awards been able to leverage their relationships with other MOMENTUM awards to benefit their work, e.g., to improve the efficiency, effectiveness, and/or scale of programming? If so, how? 2. What operational shocks and stressors are driving programmatic adaptations? 3. Does the MOMENTUM Suite address the needs and expectations of IP individual awards, USAID Missions, MOMENTUM AOR management teams, Technical Offices, and other Agency leadership? 4. What are the emerging trends and needs in FP/RH/MNCH that might be addressed in the current or future awards? METHODS This case study used a mixed methods approach to answer the four main questions. It conducted in￾person key informant interviews (KII) and a desk review to gather data and triangulate findings. Informed consent was obtained for all KIIs, which were conducted in Bahasa Indonesia. Desk Review A desk review was completed with available documents such as the MCGL and MPHD workplans and annual reports, and other relevant documents including MOH regulations related to implementation of the MCGL and MPHD programs. A full list of documents reviewed is provided in Annex 3A. Key Informant Interviews The evaluator conducted 24 KIIs. The KIIs consisted of open-ended questions and focused on experiences from implementation of the MCGL and MPHD programs, perceptions of challenges, and areas for improvement. Respondents were selected from key personnel of the MCGL and MPHD GH EvaLS MOMENTUM Mid-Term Process Evaluation / 109 programs, the USAID/Indonesia Health Office, program management cluster of MCGL NTT, program management of MPHD Regional Banten, and national and subnational stakeholders associated with the implementation of MCGL and MPHD in Indonesia. Annex 3B provides a profile of the respondents. Qualitative Data Analysis This study developed key themes and findings. The qualitative data were analyzed using tally sheets. To ensure reliability, the qualitative data were triangulated among respondents and refer to related documents. When needed, the interviewer contacted respondents for clarification. Limitations A major limitation of this case study was that some respondents were newly involved in MCGL or MPHD and may have had limited information about the MOMENTUM Suite and/or the previous USAID EMAS and Jalin projects. To mitigate the limitation, the evaluator conferred with key persons in MOMENTUM Indonesia who have in-depth understanding of current and previous USAID MNCH projects and the current situation in the MOH. The evaluator coordinated with USAID/Indonesia in listing key stakeholder respondents to obtain more comprehensive and balanced responses. FINDINGS 1. Profile of respondents. This study recruited 24 respondents from USAID/Indonesia, MCGL, MPHD, and national and subnational stakeholders (see Annex 3B, which also describes their involvement in the MCGL/MPHD programs and experiences with MCH). 2. The costs and benefits of coordinating awards. The design of the MOMENTUM program in Indonesia was based on pragmatic considerations. Combining activities involving the public and private health sectors is complex. Therefore, the MOMENTUM program in Indonesia is divided among the interconnected activities of MCGL and MPHD. Although the programs focus on the public and private health sectors, respectively, USAID Indonesia believed having JHPIEGO lead both would enable easier coordination of their activities. “…with one implementing partner (JHPIEGO), it is easier to coordinate, further, it is also possible to have some opportunities such as having offices in the same place allow them for daily interaction and facilitating regular meetings between team of MCGL and MPHD.” — Key informant The MCGL program focused on improving MNH services, closely relating them to the roles and functions of the Directorate of Nutrition and Maternal and Child Health (GKIA). The MPHD program for improving MNH services in private health facilities needs to coordinate with the Directorate of Quality of Health Services (MPK). Annex 3C describes the organization of the MOH. MCGL and MPHD need to establish substantial coordination with both directorates, as both projects focus on improving the quality of MNH health services in public and private health facilities. MCGL and MPHD have developed joint guidelines for hospital mentoring and confer with representatives from the GKIA. They have conducted joint trainings that consider the needs of each program — including intersecting ones, such as the Audit Maternal Perinatal Surveillance and Response (AMPSR) and the Point of Care Quality Improvement sections. MCGL and MPHD meet regularly with Monitoring, Evaluation and Learning (MEL) regional clusters to discuss their achievements and provide feedback so they can bridge gaps between regions. GH EvaLS MOMENTUM Mid-Term Process Evaluation / 110 On coordination in the field between MPHD and MCGL, MPHD Regional Banten has shared the data. This included data on infants receiving resuscitation from both public and private health facilities to MCGL. However, the MPHD Regional Banten note: “What a different from the previous Jalin or EMAS program is that the MOMENTUM MCGL and MPHD programs must involve the public and private health sectors, so it takes time to involve and coordinate to these parties.” — Key informant The MCGL and MPHD programs, in supporting the MOH’s establishment of the Integrated Referral System (SISRUTE), should coordinate with broad MOH institutions: the Directorates of GKIA; MPK; Referral Health Services; Governance of Health Services (TKPK); Centre of Data and Information (PUSDATIN); and Digital Transformation Office (DTO). MOMENTUM initiated the development of a referral system for the accommodated competency of public and private health facilities that provide MNH services, including for maternal and neonatal emergencies. A trial of the SISRUTE in Jakarta and NTT involved public hospitals, puskesmas, district health offices (DHOs), private hospitals, and private midwife practices. Participants from public and private health facilities (see Annex 3D) reported 100 percent usability. The challenges are to synchronize competency-based referral systems and class-based referrals. The Social Security Administrator for Health (BPJS-Kesehatan) applies class-based referrals; however, for maternal and newborn care, elective maternal comorbidity requires competency-based referral services. It will be easier to adapt competency-based referral services in NTT, which has a limited number of public and private hospitals, than in Jakarta, which has many public and private hospitals. 3. The costs and benefits of in-country coordination with USAID, IPs, and country partners. At the start, both MOMENTUM awards experienced challenges in building relationships with the MPK. This was the first time USAID/Indonesia had engaged with the MPK in the MOH as the main host for MPHD. Previously, many USAID programs involved the GKIA and the Directorate of Prevention and Control of Infectious Disease for MCH and infectious diseases projects, respectively. Rapid changes in MPK leadership, including the recent MOH restructuring, affected project cooperation and internal coordination with the MOH. Most MCGL and MPHD key personnel (83 percent, n=12) stated that time and effort were needed to build understanding with MPK in order to gain support for MOMENTUM program implementation. Intensive coordination by MOMENTUM has resulted in programmatic support from the MPK. “…coordination with MOMENTUM and GKIA is considered to have gone well and in the meeting at the Grand Melia Hotel, it was also discussed adjusting the workplan with current MOH strategic plan, which will assess patient satisfaction at the referral facility.” — Key informant In the field (for example, NTT province), although MGCL’s main mandate is strengthening capacity for MNH services in public health facilities, the program needs to identify the strengths and contributions of private hospitals. The district has a public hospital with limited facilities for MNH services and a private hospital with the capacity for mentoring. This private hospital has been identified as a learning location where staff and health workers from the public hospital can build their skills. Some key stakeholders at the national and subnational levels stated that there is a benefit to having MCGL and MPHD operate in Indonesia. MOMENTUM support for MNH services involving the public and private sectors aligns with the MOH and DHO legal requirements and GH EvaLS MOMENTUM Mid-Term Process Evaluation / 111 policies.20,21,22 MCGL and MPHD complement each other, as seen in NTT. Respondents from stakeholders at the subnational level, such as the Provincial Health Office, DHOs, directors of public and private hospitals in areas covered by this study, and the provinces of NTT and Banten all acknowledged MOMENTUM’s important technical assistance. This includes assistance in hospital mentoring and utilization of the SISRUTE, using reporting system digital applications such as E-Kohort, maternal perinatal death notification (MPDN), assisting DHO in implementing facilitative supervision, and establishing a working group (POKJA). Concrete local government commitment to supporting the hospital mentoring program was demonstrated in the district of Kupang, NTT, through the allocation of IDR 1 billion ($68,966) to purchase equipment for comprehensive emergency obstetric and newborn care in Naibonat Hospital. The size of the expenditure is considered significant, as the district has limited financial resources and numerous priorities for the poor. However, as the life of MOMENTUM is limited, building resilience in the host country must be initiated earlier. Further, the MCGL and MPHD teams need to ensure that the central and local government can accommodate their support/cost and including it in their budgets. Engagement with private health facilities (hospitals, clinics, or private midwife practices) to improve MNH service care delivery requires greater understanding of business processes. A fundamental issue raised by the MPHD Regional Banten is that some private hospitals still do not provide services for BPJS patients. Therefore, hospitals that provide mentoring have not been able to fully follow the referral system link involving BPJS. “It needs to mapping out what aspects of finance and management are needed and which institutions have potency to assist their needs.” — Key informant In addition to exploring potential incentives for mentor hospitals on mentoring mentee hospitals, the programs also should explore the potential incentives for private health facilities for their supporting the national program in reducing MMR and NMR. The involvement of the Ministry of Finance for low credit rate instruments or subsidies to finance private health facilities, and tax exemptions for the purchase of medical equipment as incentives, should be explored because reducing MMR and NMR is a national priority.23 4. Create partnerships within the consortium. USAID has a strategy of involving national organizations under the assumption that doing so will create a multi-player effect by making sub￾awards to small local organizations. In other words, to enhance local capacity, USAID has 20 Circular Letter of Secretary General of MOH No. HK/02.02/DIII/548 Year 2020 Re: Strengthening Roles of Hospital on Acceleration Reducing MMR and NMR. 21 MOH Decree No 21 Year 2021 Re: Maintenance of Health Services Pre-Pregnancy, During Pregnancy, Labour, and Post Labour, Contraception Services and Reproductive Health Services. 22 Regulation of Head of District of Tangerang No: 128 Year 2015 Re: Guidance for Referral System of MNH Emergencies. 23 Guideline Book: Tax Exemption on Imported Goods for Health Services, Directorate General Customs. Ministry of Finance, 2018. “It needs the right timing and knowing which local government budgets can support MOMENTUM activities. In addition to building ownership, we also need to take into account the efforts that they may be able to make, not too big beyond their capabilities.” — Key informant GH EvaLS MOMENTUM Mid-Term Process Evaluation / 112 channeled funds to local IPs that are expected to have capacity and networks at the national scale and to work in intervention areas such as tuberculosis and related areas. Private hospitals have made significant contributions to MNH services in Tangerang. However, the Tangerang DHO has limited manpower to capitalize on those contributions. The MPHD Regional Banten engages with private hospitals that have national networks, such as Siloam Hospital, Ciputra Hospital, and Eka Hospital. A KI from a public hospital acknowledged the benefit of hospital mentoring for maternal and newborn care and committed the institution’s support as a mentoring hospital. However, the KI also addressed potential financial limitations in strengthening maternal and newborn care, especially for small public hospitals. Because MNH issues are related to other issues such as nutrition and stunting, a KI noted that the GKIA initiated a forum with development and U.N. agencies to learn who is doing what and what can be done together. In Banten, USAID-funded projects such as MADANI/FHI 360 and the Health Financing Activity/Think-Well also cover the health sector. These have potential as platforms for collaboration, adaptation, and sharing lessons learned. 5. Adaptations such as changes in systems or processes are required for more efficient and effective coordination. One advantage of MOMENTUM implementation in Indonesia is that a majority of the respondents from MCGL, MPHD, and USAID (62 percent) have been involved with the program for at least a year. Eighty-seven percent are experienced in MCH (see Annex 3B). The MCGL and MPHD chiefs of party are both from the area and are known to key stakeholders for their work in MCH. The MCGL and MPHD teams’ experience in MCH benefit program implementation. “… for NTT it was relatively easy because MCGL have recruited a team in NTT who have experiences and have many relationships with local governments, so the local team plays a very important role to communicate directly effectively.” — Key informant MCGL and MPHD established a working group (POKJA) embedded in stunting programs, and they appointed champions to build local government and private sector commitment. The group’s agenda, initiated by the NTT cluster province, includes gradually transitioning from full support by MOMENTUM to local government support. MPHD has developed a concept of Lead Quality Assurance, Quality Improvement with the aim of improving quality of care by encouraging hospital management and leadership to adhere to clinical standards. This initiative will be steered by leading private hospitals (such as Hermina or Siloam), involving other private hospitals that seek quality standard improvements. Most directorates or centers in the MOH interact with many development agencies. MCGL and MPHD have engaged many directorates and centers; this may create some complexity in coordination. Exclusively formal communication with the directorates may delay responses or rely on out-of-date information. However, unlike Jalin, MOMENTUM has not embedded a secretariat in the MOH office to interact with a point of contact. In some situations, under decentralized government systems in Indonesia, local governments can implement central government policies and strategies. One KI at the central level expressed interest in a field visit to better understand how central policies are implemented in the field. Insight, understanding, and new ideas can be gained through joint monitoring visits with representatives from related MOH directorates and institutions, the National Development and Planning Agency (which has a role in national priorities that reduce stunting, MMR, and NMR) and the Ministry of Finance (which has roles in budget allocation, credit instruments, and tax exemption). GH EvaLS MOMENTUM Mid-Term Process Evaluation / 113 6. The MOMENTUM Suite, both collectively and as individual awards, adapted to the realities of working in dynamic country contexts — particularly responding to the ongoing effects of the COVID-19 pandemic. The pandemic has affected program implementation, limited baseline data collection from health facilities, and reduced productive interaction by using online meetings to engage partners such as private health providers. The pandemic also eliminated travel for several international short-term technical advisors, limiting opportunities for MOMENTUM Indonesia to develop ideas or strategic approaches within the local context. Limitations in the implementation of new ideas and concepts have led to some MOMENTUM activities conducting programs in line with MOH activities. In NTT, which has a scattered population and few positive COVID-19 cases, the MCGL team conducted activities in-person with recommended health protocol. 7. Additional adjustments are needed to achieve the vision for this Suite of awards. Most members of the MCGL and MPHD teams (77 percent) stated that the structure under MOMENTUM Knowledge Acceleration (MKA) focused on KM; strategic communications; and monitoring, evaluation, innovation, and learning — and that they benefitted from that structure. However, online technical support from the MOMENTUM Suite across Indonesia’s multiple time zones limited their participation. Both regional teams preferred in-person training, especially for topics requiring intensive interaction and discussion. Innovations from a developed country, such as a referral system or cross-subsidy for the poor from premium health facilities, may provide inspiration for both MOMENTUM and stakeholders. Indonesia also may share current activities, such as the application of MPDN or a digital platform for hospital self-assessment instrument to other developing countries. 8. Technical or functional skills are not available or accessible in the current configuration. KIs noted some gaps in technical and/or functional skills that MOMENTUM awards provide. For example, respondents stated that technical expertise is needed to ensure comprehensive understanding of concepts and practical business processes for private health providers. Additionally, MOMENTUM needs to explore potential incentives such as tax exemptions for purchasing medical equipment needed by private health facilities for providing MNH services. The regional team has limited time to develop policy briefs or other advocacy materials. Regular updates are important to inform stakeholders at the national and subnational levels, whose assignments may change frequently. With unique programmatic partner portfolios that involve public and private entities in MOMENTUM, MCGL and MPHD need more substantial internal coordination, along with current technical coordination and regular meetings. Finally, KIs commented that MCGL and MPHD need a more comprehensive understanding of the vision for the continuum of development toward the self-reliance of local partner institutions and responding to challenges they face in the field. 9. Emerging trends and needs in FP/RH/MNCH might be addressed in the current or future awards. KIs commented that, in developing public-private partnership activities in co￾districts, MOMENTUM must provide clear guidance for its teams, including field teams. For example, MOMENTUM should consider a tailored program based on regional situations (for example, the provincial hospital in NTT can be a mentor hospital for the Banten provincial hospital, which has some limitations). Pending issues include the finalization of hospital mentoring guidelines and AMPSR guidelines, and identification of where capacity needs to be built. Capacity building takes time, whether the focus is on the health services, health facilities, or professional staff, and it should be clearly defined from the start to make best use of MOMENTUM’s limited resources. GH EvaLS MOMENTUM Mid-Term Process Evaluation / 114 As part of a discussion involving GKIA, MPK, USAID, MCGL and MPHD, the MOH expects MOMENTUM’s support in drafting government regulations that discuss factors contributing to maternal and infant mortality, such as access to referral health services, transportation, and others, to be supported by local governments. The MOH also needs support to obtain patient satisfaction data on health services provided by private hospitals through studies from independent parties. In relation to FP, decreasing MMR and NMR requires increasing the implementation of family planning programs in terms of preparing mothers to have good health for pregnancy. With the start of the Country Health Information Systems and Data Use project and potential USAID support for stunting in Indonesia, it will be important to define a platform for coordinating these projects with MOMENTUM and MOH in addition to current, close coordination with current USAID partners, such as FHI 360, and projects, such as the Health Financing Activity, in co-located areas. CONCLUSIONS AND RECOMMENDATIONS Staff and leadership from Indonesia’s public and private sectors acknowledged that the MCGL and MPHD programs have benefited the host country. MCGL and MPHD have established internal and external coordination with stakeholders. There is promising evidence that the MCGL and MPHD have begun and already established good coordination for public-private engagement in their fields. Strengthening coordination with existing USAID projects and designing a platform for new USAID projects, such as the Country Health Information Systems and Data Use and Stunting projects are also on their agendas. This will expand the impact of the comprehensive USAID and GOI partnership. The recommendations for further improvements for MOMENTUM in the near future at the global, national and project level are: Global 1. The MOMENTUM Suite of Awards needs to activate international short-term technical assistance corresponding to technical assistance required by both MCGL and MPHD, which has been postponed during the COVID-19 pandemic. 2. For global learning, MOMENTUM needs to accommodate the situations and conditions in Indonesia regarding time differences, updating materials for advocacy in the country context, and providing some models from more developed countries in using technology and management on public private partnership for MNH services. National 3. As the number of directorates and centers involved in MOMENTUM increase, there is a need to establish a secretariat embedded in the MOH to build more substantial and intensive relationships with these institutions ensuring both MOMENTUM and MOH have updates and effective information exchanges 4. MOMENTUM needs to plan joint monitoring visit(s) involving key stakeholders at the central level to better understand the situation in the field and provide adjustments and adaptations for their priorities when needed. GH EvaLS MOMENTUM Mid-Term Process Evaluation / 115 MCGL and MPHD 5. MCGL and MPHD, in coordination with respective directorates in the MOH for MNH Services, need to provide clear guidance for their field teams to develop activities in the co-location areas involving public and private health providers. 6. There is a need for a more comprehensive understanding of business processes and to explore potential incentives for the private health providers on MNH services that ensure effective and resilient engagement with private health providers. MPHD needs to hire expert(s) to examine these issues further. GH EvaLS MOMENTUM Mid-Term Process Evaluation / 116 3A: Indonesia Case Study — List of Documents Reviewed 1. Circular Letter of Secretary General of MOH No. HK/02.02/DIII/548 Year 2020 Re: Strengthening Roles of Hospital on Acceleration Reducing MMR and NMR 2. Document of Budget Implementation District of Health Office Kupang, 2021 3. Final Draft, Hospital Mentoring Program, Directorate General Public Health, MOH, 2020 4. Final Report Assessment of Hospital Mentoring, Development Evaluation USAID Project, 2021 5. Guidance Book: Tax Exemption on Imported Goods for Health Services, Directorate General Customs, Ministry of Finance, 2018. 6. MCGL Indonesia Annual Report, October 1, 2020–September 30, 2021 7. MCGL Indonesia Fact-Sheet: Saving Moms and Babies, 2021 8. MCGL Indonesia SAR Country Summary, 2021 9. MCGL Indonesia Year 1 Workplan, February–December 2021 10. MCGL Indonesia, Semi Annual Report, October 30, 2021–March 31, 2022 11. MCGL Monitoring, Evaluation, and Learning Plan, March 2022 12. Ministry of Health (MOH) Decree No. 5 Year 2022 Re; Organization and Working Procedure in MOH 13. MOH Decree No 13 Year 2022 Re: Strategic Plan of MOH 2020-2024 14. MOH Decree No 21 Year 2021 Re: Maintenance of Health Services Pre-Pregnancy, During Pregnancy, Labour, and Post Labour, Contraception Services and Reproductive Health Services. 15. MOMENTUM Policy Brief: Promising Practices for Engaging Local Faith Actors to Promote Uptake of COVID-19 Vaccination, 2021 16. MPHD Indonesia Annual Report, October 1, 2020–September 30, 2021 17. MPHD Indonesia Country Work Plan Year 1 April–December, 2021 18. MPHD Indonesia Country Work Plan Year 2, March 1–December 31, 2022 19. Regulation of Head of District of Tangerang No: 128 Year 2015 Re: Guidance for Referral System of MNH Emergencies 20. System of Hospital Clinical Performance Tools (Alat Pantau Sistem Kinerja Klinik di Rumah Sakit), USAID Expanding Maternal & Neonatal Survival Project, 2014 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 117 3B: Indonesia Case Study — Profile of Respondents GH EvaLS MOMENTUM Mid-Term Process Evaluation / 118 3C: Indonesia Case Study — Translation of Roles and Functions Translation of Roles and Function Each related Directorate and Centre from Ministry of Health Regulation No. 5 Year 2022 Directorate of Nutrition, Maternal and Child Health, (GKIA)*}, Article 54: The Directorate of GKIA has the tasks of carrying out the formulation and implementation of policies, drafting norms, standards, procedures, and criteria; and providing technical guidance and supervision, evaluation, and reporting in the field of improvement for nutrition and maternal and child health. Directorate of Referral Health Services, Article 106: The Directorate of Referral Health Services has the tasks of carrying out the formulation and implementation of policies; preparation of norms, standards, procedures, and criteria; and providing technical guidance and supervision, evaluation, and reporting in the field of referral health services. Directorate of Quality of Health Services (MPK)#}, Article 122: The Directorate of Quality of Health Services has the tasks of carrying out the formulation and implementation of policies; drafting norms, standards, procedures, and criteria; and providing technical guidance and supervision, evaluation, and reporting in the field of quality and accreditation of health services. Directorate of Governance of Health Service (TKPK), Article 114: The Directorate of Governance of Health Service has the task of carrying out the formulation and implementation policies; formulation of norms, standards, procedures, and criteria; and providing technical guidance and supervision, evaluation, and reporting in the field of governance of health service. The Data and Information Technology Centre (PUSDATIN)$}, Article 238: The PUSDATIN has a duty to carry out data management and information technology including the DTO. Under MOH Regulation No. HK.01.07/MENK ES/3605/2021, the DTO is established to transform health care delivery in Indonesia to transform healthcare in Indonesia through data and technology with three priority missions: 1. To enable electronic medical records to be accessible for every individual and across health care facilities in Indonesia (by user’s consent); 2. To simplify systems for health workers to make them more focus on saving lives rather than administrative matters; and 3. To propel policy innovation to strengthen the health-tech ecosystem. *} Key partner of MCGL #} Key partner of MPHD $} Has central roles in data management and information of technology in which MOMENTUM coordinate in supporting SISRUTE GH EvaLS MOMENTUM Mid-Term Process Evaluation / 119 3D: Indonesia Case Study — Health Facility Participation Participation of Health Facilities (Public Hospital, Puskesmas, Private Hospital, Private Clinic and Private Midwife Practice/PMB) on Trial of SISRUTE on Maternal and Neonatal Health in NTT and Jakarta NTT (MCGL) June 9–15, 2022 Number of Health Facilities Participating: 53 Notes on Trial of SISRUTE in NTT: • 100 percent of the health facilities involved in the trial were successful in using SISRUTE to send maternal and neonatal referrals. • 100 percent of hospitals that receive referral requests via SISRUTE can also respond to referrals via SISRUTE. • Five DHO involved in the pilot have not been fluent in demonstrating SISRUTE to send referrals, and demonstrating monitoring of maternal referrals and monitoring of neonatal referrals at SISRUTE. Assistance is needed for retraining for the role of the health office in demonstrating the use and monitoring of SISRUTE. JAKARTA (MPHD: June 30 - July 1, 2022 Number of Health Facilities Participating: 17 Note on Trial of SISRUTE in Jakarta: • 100 percent of the health facilities involved in the trial were successful in using SISRUTE to send maternal and neonatal referrals. • 100 percent of hospitals that receive referral requests via SISRUTE can also respond to referrals via SISRUTE. • Three sub-DHO from North, West and East Jakarta can also demonstrate SISRUTE to send referrals, and demonstrate monitoring of maternal referrals and monitoring of neonatal referrals at SISRUTE. GH EvaLS MOMENTUM Mid-Term Process Evaluation / 120 Photos from SISRUTE Trial in Jakarta Collected During This Study GH EvaLS MOMENTUM Mid-Term Process Evaluation / 121 ANNEX 4: DATA COLLECTION TOOLS 4A: Informed Consent GH EvaLS 024 MOMENTUM Mid-term Evaluation Informed Consent Statement Introduction My name is _________________, and I am an independent consultant working for GH EvaLS. Thank you for making the time to talk with me today. You were suggested as a key person to inform the evaluation of the MOMENTUM Suite of awards, and we greatly appreciate your perspective, experiences and views on the successes, challenges, barriers and lessons learned from your experience. Our interview will take between 45 minutes to 1 hour. Before we begin, I want to inform you that any information or examples we gather during this interview process will not be attributed to any specific person, or otherwise attributed to you. If we include quotations from our interview in the final report, they will be attributed only to a stakeholder group. You are also free to not respond to any of our questions or to stop the interview at any time. Please know that anything you say during the interview will be kept confidential within the GH EvaLS team, and that our interview notes and any recordings will be erased when the report is completed. Do I have your permission to begin? [ ] Consent provided ?________ [Interviewer/Recorder initials] Before we begin, do you have any questions about this interview? GH EvaLS MOMENTUM Mid-Term Process Evaluation / 122 4B: Key Informant Interview Guides GH EvaLS 024 MOMENTUM Mid-term Evaluation Key Informant Interview Guide / Basic Form Interviewer: Interview Date: Start time: Interviewee’s Name: First: Last: Current Position: Org: Contact email: Has the KI affirmed Informed Consent? Y____ N____ (Interviewer’s initials) Has the KI consented to recording the interview? Y____ N____ NA___ Respondent’s Unique ID: ______________________ Time at End of Interview: Interviewer’s observations or interpretations: GH EvaLS MOMENTUM Mid-Term Process Evaluation / 123 Questions for KII Guide (AORs, IPs, other USAID/W staff) Core Questions 1. What is your role/affiliation with MOMENTUM? 2. Please describe how you have interacted with the MOMENTUM project. (Area 1): Design 3. Have the awards been able to leverage their relationships with other MOMENTUM awards to benefit their work? (Efficiency, effectiveness and/or scale of programming)? If so, how? 4. (OPTIONAL) What has been the benefit of coordinating with MKA to USAID and Implementing Partners and across the Suite? 5. (OPTIONAL) What has been the benefits and costs of in-country coordination to USAID and Implementing Partners? 6. What adaptations or changes in systems or processes, would lead to more efficient and/or more effective coordination? (Area 2): Adaptation 7. What operational shocks and stressors are driving programmatic adaptations? 8. How has the Suite collectively, and as individual awards adapted to the realities of working in dynamic country contexts, particularly the ongoing effects of the COVID￾19 pandemic? 9. (OPTIONAL) Do any of these adaptations require adjustments to the development hypothesis (project design) and/or key assumptions of the MOMENTUM award Suite and/or implementation at the country or global level? 10. Are additional adjustments needed to achieve the vision for this Suite of awards? If so, what? (Area 3): Stakeholder Satisfaction 11. (USAID only) How does the MOMENTUM Suite address the needs and expectations of the USAID? 12. Are their technical or functional skills not available or accessible in the current configuration? 13. (OPTIONAL) Does the Suite design inhibit or facilitate meeting USAID’s needs? (Area 4): Future Trends 14. What are the emerging trends and needs in FP/RH/MNCH that might be addressed in the current or future awards? GH EvaLS MOMENTUM Mid-Term Process Evaluation / 124 Questions for KII Guide (USAID Mission staff) Core Questions 1. What is your role/affiliation with MOMENTUM? 2. Please describe how you have interacted with the MOMENTUM project. (Area 1): Design 3. Have the awards been able to leverage their relationships with other MOMENTUM awards to benefit their work? (Efficiency, effectiveness and/or scale of programming)? If so, how? 4. What has been the benefits and costs of in-country coordination to USAID and Implementing Partners? 5. (Optional for USAID Mission staff) Has the design of MOMENTUM enhanced or strengthened local technical leadership? If yes, how? (Area 2): Adaptation 6. What operational shocks and stressors are driving programmatic adaptations? 7. Has MOMENTUM adapted to the realities of working in your dynamic country context, particularly the ongoing effects of the COVID-19 pandemic? (Area 3): Stakeholder Satisfaction 8. Does the MOMENTUM Suite address the needs and expectations of your Mission? 9. Are their technical or functional skills not available or accessible in the current configuration? 10. Have you read or utilized MKA’s regional briefs pertaining to your region? If so, are they useful? (Area 4): Future Trends 11. What are the emerging trends and needs in FP/RH/MNCH that might be addressed in the current or future awards? GH EvaLS MOMENTUM Mid-Term Process Evaluation / 125 Questions for KII Guide (Global Partners) 1. What is your role/affiliation with MOMENTUM? 2. Please describe how you have interacted with the MOMENTUM project. 3. Has the MOMENTUM project design enhanced USAID’s catalytic, global leadership role to advance MNCH/FP/RH? If yes, how? 4. Based on your interactions with MOMENTUM, are their technical or functional skills not available or accessible in the current configuration? 5. What are the emerging trends and needs in FP/RH/MNCH that might be addressed in the current MOMENTUM project or in the future? GH EvaLS MOMENTUM Mid-Term Process Evaluation / 126 Questions for KII Guide (MoH and other in-country partners) Core Questions 1. What is your role/affiliation with MOMENTUM? 2. Please describe how you have interacted with the MOMENTUM project. (Area 1): Project Design 3. Has the design of MOMENTUM enhanced or strengthened local technical leadership? If yes, how? 4. Has the MOMENTUM design been responsive to MOH and/or MOH partner’s needs and goals? If yes, how? 5. What type of perception does MOH staff (or MOH partner staff) have of the MOMENTUM project at the country level? (Area 2): Adaptation 6. Has MOMENTUM adapted to the realities of working in your dynamic country context, particularly the ongoing effects of the COVID-19 pandemic? (Area 3): Stakeholder Satisfaction 7. Are there technical or functional skills not available or accessible in the current project? 8. Have you read or utilized MKA’s regional briefs pertaining to your region? If so, are they useful? (Area 4): Future Trends 9. What are the emerging trends and needs in FP/RH/MNCH that might be addressed in the current project or in the future? GH EvaLS MOMENTUM Mid-Term Process Evaluation / 127 4C: Electronic Survey GH EvaLS MOMENTUM Mid-Term Process Evaluation / 128 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 129 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 130 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 131 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 132 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 133 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 134 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 135 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 136 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 137 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 138 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 139 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 140 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 141 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 142 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 143 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 144 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 145 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 146 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 147 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 148 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 149 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 150 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 151 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 152 ANNEX 5: SOURCES OF INFORMATION 5A: List of Persons Interviewed Name Organization Title MOMENTUM Suite of Awards 1 John Borrazzo MCGL Director of Child Health 2 Adhish Dhungana MCGL Senior Program Manager, Nepal 3 Elizabeth Eastman MCGL Knowledge Management and Learning Advisor 4 Paul Odila MCGL COP Uganda 5 Angela Pereira MCGL Communications Team Lead 6 Jim Ricca MCGL Director Adaptive Management, M&E and Learning 7 Laura Skolnik MCGL Senior Director of Programs 8 Suzanne Stalls MCGL Director MNH 9 Lillian Tumuhairwe MCGL Uganda Technical Advisor 10 Kirk Dearden MIHR Team Leader Nutrition and WASH 11 Lorelei Goodyear MIHR Deputy Technical Director FP/RH Lead 12 George Hanna MIHR Director 13 Kamden Hoffman MIHR Social and Behavior Change Lead 14 Sarah Kellogg MIHR Country Support Lead 15 Christine Lasway MIHR Senior Technical Advisor for FP 16 Christopher Lindahl MIHR Knowledge Management Lead 17 Eric Ramirez￾Ferrero MIHR Technical Director 18 Pat Taylor MIHR Acting Deputy Director 19 Soumya Alva MKA Senior M&E Director 20 Goli Fassihan MKA Former Senior Director of Strategic Communication 21 Reshma Naik MKA Senior Direct KM 22 Barbara Seligman MKA Director 23 Lara Vaz MKA Technical Director 24 Erin Dumas MPHD Senior Program Manager 25 Mariela Rodriguez MPHD Senior Learning and Communications Advisor GH EvaLS MOMENTUM Mid-Term Process Evaluation / 153 Name Organization Title 26 Gaurav Sharma MPHD Senior MNH Lead 27 Nicole Castle M-RITE Program Officer, KM and Communications 28 Grace Chee M-RITE Director 29 Katie Cook M-RITE Communications and KM Lead 30 Rebecca Fields M-RITE Immunization Technical Lead 31 Isaac Mugoya M-RITE Country Director, Kenya 32 Vanessa Richart M-RITE Deputy Director 33 Jessica Shearer M-RITE M&E Lead 34 Gloriose Abayisenya MSSFPO MCH Advisor Rwanda 35 Karen Levin MSSFPO MEL Director 36 Jen Snell MSSFPO Senior Technical Manager 37 Renae Stafford MSSFPO Deputy Technical Director 38 Vandana Tripathi MSSFPO Director Partner Organizations 39 Lynn Freedman AMDD, Columbia University SPH Director 40 Jeff Smith BMGF Deputy Director, MNCH 41 Willabald Zeck United Nations Population Fund Interim Sexual Reproductive Health Branch Chief 42 Bernadette Daelmans WHO Unit Head Child Health and Development 43 Allisyn Moran WHO Head of Maternal, Newborn. Child, and Adolescent Health and Aging USAID 44 Sachin Gupta USAID/India Advisor (Maternal and Child Health) 45 Vijay Paulraj USAID/India Senior Health Systems Strengthening Advisor 46 Moni Sinha Sagar USAID/India Division Chief for Family Health 47 Patrice Coulibaly USAID/Mali Program Management Specialist HR/FP 48 Emily Hillman USAID/Mozambique Division Chief, Preventing Child and Maternal Deaths GH EvaLS MOMENTUM Mid-Term Process Evaluation / 154 Name Organization Title 49 Gertrude Odezugo USAID/Nigeria Program Development Specialist - Senior Reproductive Health Manager 50 Homsuk Swomen USAID/Nigeria Maternal and Child Health Specialist — Program Manager 51 Maria Kabanyana USAID/Rwanda Team Lead for Health Systems and Service Delivery, Senior Specialist for Reproductive and MNCH 52 Richard Munyaneza USAID/Rwanda Maternal and Child Health Specialist 53 Malia Boggs USAID/W Senior Technical Advisor for Child Health 54 Clancy Broxton USAID/W Private Sector Health Team Leader AOR — MPHD 55 Kate Crawford USAID/W Global Director of MCH and Nutrition 56 Benjamin Isquith USAID/W Division Chief, Service Delivery Improvement MOMENTUM Coordinator 57 Rebecca Levine USAID/W Senior Maternal Health Advisor AOR, MSSFPO 58 Shawn Malarcher USAID/W Senior Best Practices Utilization Advisor AOR, MKA 59 Lory Meoli USAID/W Health Development Officer AOR, MCGL 60 Erin Mielke USAID/W Reproductive Health Senior Technical Advisor 61 Elizabeth Pleuss USAID/W Acting Division Chief, Maternal and Newborn Health 62 Barbara Rawlins USAID/W Senior Implementation Research Advisor (MCHN) 63 Intissar Sarker USAID/W Private Sector Service Delivery Technical Advisor 64 Ellen Starbird USAID/W Director, Office of PRH 65 Alanna White USAID/W Health Development Officer AOR, MIHR GH EvaLS MOMENTUM Mid-Term Process Evaluation / 155 5B: Bibliography of Documents Reviewed 1. Adaptive Learning Resource Collection, MOMENTUM Knowledge Accelerator. 2020. 2. Anita Shet, Kelly Carr, M. Carolina Danovaro-Holliday, et al., Impact of the SARS-CoV-2 pandemic on routine immunisation services: evidence of disruption and recovery from 170 countries and territories, www.thelancet.com/lancetgh Vol 10 February 2022. 3. Annual Program Statement, MOMENTUM (Moving Integrated, Quality Maternal, Newborn, and Child Health and Family Planning and Reproductive Health Services to Scale). May 2019. 4. Cooperative Agreement Number: 7200AA20CA00002 (redacted), MOMENTUM 2A: Global Technical Leadership and Targeted Technical Assistance for MNCH/FP/RH. December 2019. 5. Cooperative Agreement Number: 7200AA20CA00003 (redacted), MOMENTUM 2C: Advancing Collaboration and Communication on Evidence and Learning (ACCEL) for MOMENTUM. December 2019. 6. Cooperative Agreement Number: 7200AA20CA00005 (redacted), MOMENTUM INTEGRATED HEALTH RESILIENCE PROJECT (IHRP). May 2020. 7. Cooperative Agreement Number: 7200AA20CA00007 (redacted), MOMENTUM Targeted MNCH/FP/RH Technical Assistance to Private Providers. June 2020. 8. Cooperative Agreement Number: 7200AA20CA00011 (redacted), MOMENTUM 3A: Expanding Access to Safe Surgery for Family Planning and the Reduction of Maternal and Neonatal Mortality and Morbidity. September 2020. 9. Cooperative Agreement Number: 7200AA20CA00017 (redacted), MOMENTUM 3B: Overcoming Entrenched Obstacles in Routine Immunization. July 2020. 10. COVID-19 Response and Mitigation Strategics to maintain essential MNCH and FP/RH services, MOMENTUM Country and Global Leadership. 2021. 11. Developing and Rolling Out Adaptive Management and Learning Approaches, MOMENTUM Country and Global Leadership. July 2021. 12. Distance Learning Resource Collection, MOMENTUM Knowledge Accelerator. 2021. 13. Emily Stammer, Lisa Hirschhorn, Katherine Semrau, Lara Vaz. Rapid Evidence Summary: Lessons for COVID-19: Impact of prior disease outbreaks on maternal, newborn and child health, voluntary family planning, and reproductive health services, MOMENTUM Knowledge Accelerator. July 2020. 14. Ensuring the Continuity of Voluntary Family Planning and Reproductive Health care and Interventions for Youth during the COVID-19 Pandemic, MOMENTUM Country and Global Leadership. 2020. 15. Exley J., Gupta P.A., Schellenberg J., Strong K.I., Requejo J.H., Moller A-B., Moran A.C., Marchant, T. Child Health Accountability Tracking Technical Advisory Group (CHAT) and the Mother and Newborn Information for Tracking Outcomes and Results Advisory Group (MoNITOR). J Glob Health 2021; 12:04001 A rapid systemic review and evidence synthesis of effective coverage measures and cascades for childbirth, newborn and child health in low-and middle-income countries. 2022. 16. Guidance on MOMENTUM Research Transparency and Data Sharing. December 2020. GH EvaLS MOMENTUM Mid-Term Process Evaluation / 156 17. Job Aids: Recommendations for Maternal and Newborn Care during the COVID-19 Outbreak, MOMENTUM Country and Global Leadership. 2022. 18. Lara Vaz, Megan Ivankovich, and Gaby Nguyen, Lessons from COVID-19 and Beyond, An Introduction to the MOMENTUM Learning Agenda, MOMENTUM Share Fair. July 2021. 19. Larson, C.P., Plamondon, K.M., Dubent, L., et al. The Equity Tool for valuing global health partnerships. Glob health Sci Pract. 2022;10(2): e2100316. 20. MAKLab Update Activities to date and their progress, MKA Team, AOR Meeting. April 2022. 21. MOMENTUM 101. March 2021. 22. MOMENTUM 101 for USAID Operating Units. April 2022. 23. MOMENTUM: A New Approach to Supporting Countries in Accelerating Progress on the Health and Well-being of Women and Children, 2021 Progress Report. 2022. 24. MOMENTUM, Accelerating reductions in maternal, newborn, and child. 2021. 25. MOMENTUM All Hands Meeting (PowerPoint). March 2021. 26. MOMENTUM All Hands Meeting March 23, 2021. February 2021. 27. MOMENTUM Awards: Quick Reference Sheet. December 2021. 28. MOMENTUM Communications Report. October-December 2021, 2022. 29. MOMENTUM Communications Report,.September 2021. 30. MOMENTUM Country and Global Leadership, Annual Report, October 1, 2020–September 30, 2021. January 2022. 31. MOMENTUM Country and Global Leadership, Contributing to Global Technical Leadership and Elevating Country Voices to Improve Health Outcomes. 2021. 32. MOMENTUM Country and Global Leadership, Final Approved Workplan 3. November 2021. 33. MOMENTUM Integrated Health Resilience, COVID-19 Adaptation Updates. May 2022. 34. MOMENTUM Integrated Health Resilience, Strengthening Health Resilience in Fragile Settings,.2021. 35. MOMENTUM Integrated Health Resilience, Program Year Two Core Work Plan. October 2021. 36. MOMENTUM Knowledge Accelerator, Activity Manager Orientation, Presentation. April and May 2021. 37. MOMENTUM Knowledge Accelerator, Coordinating MOMENTUM Learning, Monitoring and Evaluation, Knowledge Management and Communications. 2021. 38. MOMENTUM Knowledge Accelerator and COVID-19: Rapid Response Activities. 2020. 39. MOMENTUM Knowledge Management Plan, MOMENTUM Knowledge Accelerator. February 2021. 40. MOMENTUM Knowledge Accelerator Workplan, October 1, 2021–September 30, 2022. December 2021. GH EvaLS MOMENTUM Mid-Term Process Evaluation / 157 41. MOMENTUM Learning Agenda, Capturing learning to better adapt and implement programs. December 2021. 42. MOMENTUM Learning Agenda, Defining and operationalizing learning across the Suite. May 2021. 43. MOMENTUM Mission Bilaterals. 2022. 44. MOMENTUM Monitoring, Evaluation, and Learning Framework. September 2021. 45. MOMENTUM Private Healthcare Delivery, Increasing the Capacity of Private Providers to Deliver Evidence-Based, Quality Health Care. 2021. 46. MOMENTUM Private Healthcare Delivery Year 2 Core Workplan, October 1, 2021–September 30, 2022. August 2021. 47. MOMENTUM Promising Practices for Engaging Local Faith Actors to Promote Uptake of COVID-19 Vaccination Lesson Learned from Four Countries: Ghana, Indonesia, Sierra Leone and Uganda, Policy Brief, MCGL. 2021. 48. MOMENTUM Routine Immunization and Transformation and Equity, Helping to Strengthen Equitable Coverage of Routine Immunization. 2021. 49. MOMENTUM Routine Immunization Transformation and Equity, Year 2 Workplan, October 1, 2021–September 30, 2022. October 2021. 50. MOMENTUM Safe Surgery in Family Planning and Obstetrics Year 2 Core Workplan, October 1, 2021–September 30, 2022. November 2021. 51. MOMENTUM, Strategic Communication Plan. July 2021. 52. MOMENTUM Suite Strategic Communications Clearance Process and Checklist. April 11, 2022. 53. MOMENTUM Technical Coordination Mapping Reference Document. 2022. 54. MOMENTUM Theory of Change, Guiding coordinated action for mothers and children. January 2021 55. Neill, Rachel. Approaches to Partnership Measurement: A Landscape Review. Washington, D.C.: USAID MOMENTUM Knowledge Accelerator. 2021. 56. Pfitzer A., Lathrop, E., Bodenheimer, A., et al., Opportunities and challenges of delivering postabortion care and postpartum family planning during the COVID-19 pandemic. Glob Health Sci Pract. 2020;8(3):335-343. 57. Recommendations for Effectively Starting Programs Remotely, Lessons Learned from MOMENTUM Integrated Health Resilience, MOMENTUM Integrated Health Resilience. 2021 58. Reshma Naik and Maia Johnstone. MOMENTUM HUB Orientation, MOMENTUM Knowledge Accelerator. January 2022. 59. Responding to COVID-19, Emerging Learning from across MOMENTUM, MOMENTUM Knowledge Accelerator. December 2021 60. Summary of MOMENTUM Workplans. February 2021. 61. USAID, Automated Directives System, Chapter 201. 2022. GH EvaLS MOMENTUM Mid-Term Process Evaluation / 158 ANNEX 6: SURVEY GRAPHICS Where are you located? N=88 If you are located in-country, please choose the country from the following list. 60% 40% Where are you located? In-country Headquarters 4% 4% 10% 13% 6% 23% 6% 6% 2% 12% 6% 2% 6% Country Participation Ghana Kenya Niger India DRC Nigeria Rwanda South Sudan Nepal Mozambique Mali Madagascar Burkina Faso N=51 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 159 Which technical area do you primarily focus on? N=86 Please indicate your affiliation with the MOMENTUM Suite. 1% 15% 5% 1% 1% 1% 20% 2% 1% 1% 18% 19% 4% 2% 9% Capacity Strengthening Technical Area Child health and/or immunization Community health COVID-19 vaccination Digital Health Finance and administration FP/RH Gender and Youth Improving health systems Maternal and Neonatal Health, and WASH Maternal and/or newborn health Monitoring, Evaluation, Research, and Learning (MERL) Overall program 6% 75% 7% 4% 8% MOMENTUM Affiliation Global partner (e.g., WHO, UNICEF, GAVI, FP2020, etc.) Implementing partner Ministry of Health / in-country government counterpart USAID Mission Staff USAID/W staff N=88 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 160 To the best of your knowledge, are there any technical needs that are currently not met by the MOMENTUM Suite? N=70 Please rate your overall level of satisfaction with the MOMENTUM Suite, specifically in terms of ease or coordination and collaboration. Consider issues such as quality of interaction, management burden and responsiveness, partner feedback, and other relevant factors. 34% 36% 30% Are there technical needs that are not met by MOMENTUM? Yes No No Opinion 9% 8% 25% 50% 8% Satisfaction with MOMENTUM Highly Dissatisfied Somewhat dissatisfied Neutral Somewhat satisfied Highly satisfied N=12 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 161 How useful has it been to have the Suite of Awards grouped under one mechanism? N=69 How would you rate the efficacy of the MOMENTUM award holders in leveraging relationships amongst themselves to benefit their work? 3% 9% 10% 16% 40% 22% How useful has it been to have the Suite of awards grouped under one mechanism? Useless Somewhat useless Neutral Somewhat useful Useful No Opinion 36% 32% 8% 24% How would you rate the efficacy of the MOMENTUM award holders in leveraging relationships amongst themselves to benefit their work? Very Effective Somewhat effective Not at all effective No Opininon N=66 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 162 How would you rate the benefit of coordinating with MKA to USAID and Implementing Partners? N=7 How would you rate the benefit of coordinating across the MOMENTUM Suite to USAID and Implementing Partners? 14% 43% 43% How would you rate the benefit of coordinating with MKA to USAID and Implementing Partners? Low Medium High 14% 57% 29% How would you rate the benefit of coordinating across the MOMENTUM Suite to USAID and Implementing Partners? Low Medium High N=7 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 163 How would you rate the level of effort required to manage and access the various awards under the MOMENTUM Suite? N=10 How familiar are you with the MOMENTUM Suite in your country? 10% 50% 40% How would you rate the level of effort required to manage and access the various awards under the MOMENTUM Suite? Low Medium High Not Applicable 67% 16% 17% How familiar are you with the MOMENTUM Suite in your country? Very familiar Somewhat familiar Not so familiar Not at all familiar N=6 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 164 Has the design of MOMENTUM enhanced/amplified local technical leadership? N=6 How responsive is the MOMENTUM Suite to your needs and goals? 50% 17% 33% Has the design of MOMENTUM enhanced/amplified local technical leadership? Yes No Don't know 16% 67% 17% How responsive is the MOMENTUM Suite to your needs and goals? Very responsive Somewhat responsive Not so responsive Not at all responsive Don't know N=6 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 165 Do you perceive the various MOMENTUM activities as working under “one” Project in your country? 83% 17% Do you perceive the various MOMENTUM activities as working under “one” Project in your country? Yes No Don't Know N=6 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 166 ANNEX 7: DISCLOSURE OF ANY CONFLICTS OF INTEREST GH EvaLS MOMENTUM Mid-Term Process Evaluation / 167 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 168 GH EvaLS MOMENTUM Mid-Term Process Evaluation / 169 ANNEX 8: EVALUATION TEAM MEMBERS Tim A. Clary, M.A., EMBA, M.S./Ph.D., Team Leader Dr. Tim A. Clary is an independent consultant who focuses on international health and development. An infectious diseases epidemiologist by training, he has provided consulting services for organizations such as USAID, PEPFAR, the Global Fund, several United Nations agencies, the World Bank, the International Finance Corporation, GIZ, the U.K. Department for International Development, and several nongovernmental organizations. His consulting assignments have taken him to more than 65 countries in all geographic regions of the world. From 2011–2013, Dr. Clary was the Director of Health for the Millennium Challenge Corporation. During that time, he oversaw a portfolio of approximately $300 million in USG investments in health, including a $131.5 million nutrition, maternal, and child health and results-based financing initiative in Indonesia; a $120.5 million health systems strengthening and HIV/AIDS project in Lesotho; and a $38.5 million noncommunicable disease and injury program in Mongolia. From 2000–2006, he was with USAID, first as the HIV/Infectious Diseases Epidemiologist for Europe and Eurasia overseeing several grants, cooperative agreements, and contracts, and then as a Senior Public Health Advisor for USAID/Ukraine. During his time in Ukraine, Dr. Clary designed, developed, and oversaw several projects covering infectious diseases, reproductive health, maternal and child health, and birth defects for Ukraine, Belarus, and Moldova. Prior to his work with USAID, Dr. Clary was with the U.S. Centers for Disease Control and Prevention’s Office of Research Methodology. He also had an earlier career as a journalist working for Sierra Magazine and National Geographic. Pamela Putney, RN, MSN, Partnership Specialist Ms. Putney is a nurse/midwife with more than 40 years of domestic and international experience in 40 countries in the design, implementation, management, and evaluation of programs focusing on maternal, neonatal, child, and reproductive health; gender issues; health policy; health sector reform; health system strengthening; sustainable health systems; human resource development; training; strategic planning; organizational development; quality of care; HIV/AIDS; behavior change/communication; proposal and grant writing; fundraising; and leadership development. She is also the co-founder of two non-profit organizations. Bambang Heryanto, Ph.D., In-Country Specialist Dr. Heryanto is an evaluator with experience in the evaluation of policies to inform programmatic impact. He has led data collection exercises to inform the direction of USAID maternal and child health projects and led and conducted a mid-term review of the USAID Jalin project. He has nearly 30 years of experience working with international donors such as USAID, the United Nations Population Fund, IFRC. and the Asian Development Bank, as well as GOI ministries, ensuring effective project implementation. Dr. Heryanto has advised on and presented recommendations to inform strategy, policy, and future project designs and scaling across Indonesia. During his work with USAID/Indonesia (2009–2017), Dr. Heryanto managed the Highly Pandemic Avian Influenza and the Emerging Pandemic Threat Programs and managed the Global Health Security Agenda meeting in Jakarta (2014). He has developed policy briefs on health issues for USAID and the Indonesian GH EvaLS MOMENTUM Mid-Term Process Evaluation / 170 Ministry of Health and has advised USAID and other donors on deforestation and emerging pandemic disease and sustainability options. Dr. Heryanto received several awards from the U.S. Embassy in Jakarta for his work with the USAID Mission. He has published his academic works in several peer￾reviewed journals.