July 2022 This report is made possible by the support of the American People through the United States Agency for International Development (USAID). The content of this report is the sole responsibility of Social Impact, Inc. for USAID/Vietnam under USAID Learns and does not necessarily reflect the views of USAID or the United States Government. Photo: USAID Learns IMPACT-MED ACTIVITY REVIEW COMPETENCY-BASED MEDICAL EDUCATION (CBME) ADVANCEMENT AND SUSTAINABILITY IMPACT-MED ACTIVITY REVIEW Competency-Based Medical Education (CBME) Advancement and Sustainability USAID Learns Contracted under 72044019C00003 Prepared for: Le Tong Giang, Contracting Officer’s Representative (COR) United States Agency for International Development/Vietnam 15/F, Tung Shing Square 2 Ngo Quyen Street Hoan Kiem District Hanoi, Vietnam Prepared by: USAID Learns Social Impact, Inc. Level 6, BIDV Tower 194 Tran Quang Khai Street Hoan Kiem District Hanoi, Vietnam iii | IMPACT-MED ACTIVITY REVIEW USAID.GOV TABLE OF CONTENTS Aronyms v Executive Summary vi Background and Purpose vi Methods vi Research Questions vii Findings and Conclusions vii Recommendations viii Introduction 1 Background 1 Purpose and Audience 3 Research Questions 3 Methodology 4 Overview 4 Research Design 4 Recruitment Process 8 Data Analysis 11 Limitations 11 Findings and Conclusions 13 Research Theme 1: UMP Beneficiaries And CBME 13 Research Theme 2: Stakeholder Engagement and Partnership in cbme implementation 25 Research Theme 3: Sustainability 29 Recommendations 33 Advancement and Sustainability of CBME 33 Utilization 37 Annex I: Full Listing of Research Participants 38 Annex II: IMPACT-MED Activity UMP profiles 39 Annex III: Full listing of references and reports utilized 40 Annex IV: Data Collection Instruments 42 Annex V: Utilization Workshop Outcomes 49 Annex VI: Study Statement Of Work 67 Annex VII: Policy Recommendation Brief 74 Annex VIII: Thai Nguyen UMP’s ESP Program 76 iv | IMPACT-MED ACTIVITY REVIEW USAID.GOV LIST OF TABLES AND FIGURES Table 1: Research Design.................................................................................................................................................5 Table 2: Number of KII Respondents by Key Stakeholder Group, Disaggregated by Sex ..............................9 Table 3: Number of FGD Respondents by University and Employer, Disaggregated by Sex.........................9 Table 4: Number of Participants in Participatory Events, Disaggregated by Sex .............................................11 Table 5: Study Limitations and Mitigation Strategies...............................................................................................12 Figure 1: Small Group Classes at Hue UMP and HCMC UMP.............................................................................14 Figure 2: A Training Session on Clinical Teaching and Simulation Provided by the HCMC UMP UFD.....15 Figure 3: OSCE facilities allowing remote monitoring and assessment..............................................................19 v | IMPACT-MED ACTIVITY REVIEW USAID.GOV ACRONYMS CBME Competency-Based Medical Education COVID-19 Coronavirus Disease 2019 CQI Continuous Quality Improvement EPA Entrustable Professional Activities ESP English for Special Purposes FGD Focus Group Discussion FY Fiscal Year GVN Government of Vietnam HAIVN The Partnership of Health Advancement in Vietnam HCMC Ho Chi Minh City HEI Higher Education Institutions HMS Harvard Medical School HPET Health Professionals Education and Training for Health System Reform Project IMPACT-MED Improving Access, Curriculum and Teaching in Medical Education & Emerging Diseases IP Implementing Partner IQA Internal Quality Assurance KII Key Informant Interview MCQ Multiple-Choice Question Mini-CEX Mini-Clinical Evaluation Exercise MOET Ministry of Education and Training MOH Ministry of Health MOLISA Ministry of Labor, Invalids, and Social Affairs OSCE Objective Structured Clinical Examination SLO Student Learning Outcomes UFD Unit for Faculty Development UMP University of Medicine and Pharmacy USAID United States Agency for International Development WHO World Health Organization vi | IMPACT-MED ACTIVITY REVIEW USAID.GOV EXECUTIVE SUMMARY The Improving Access, Curriculum and Teaching in Medical Education and Emerging Diseases (IMPACT￾MED) Alliance is a Global Development Alliance project funded through United States Agency for International Development (USAID) Cooperative Agreement No. AID-440-A-16-00002. During the past five years (2016-2021), IMPACT-MED has been supporting five Vietnamese Universities of Medicine and Pharmacy (UMPs) - Ho Chi Minh City (HCMC) UMP, Hue UMP, Thai Nguyen UMP, Thai Binh UMP, and Hai Phong UMP - to reform the medical education curriculum and improve institutional governance practices and systems for continuous quality improvement (CQI). IMPACT-MED was extended for an additional five years, from 2022 to 2026. To complement the work planning process involved with this extension, USAID/Vietnam commissioned USAID Learns to conduct an Activity Review of IMPACT-MED. BACKGROUND AND PURPOSE The objectives of this Activity Review are first, to learn about the extent to which the Activity has met its beneficiaries’ educational needs, and second, to identify barriers and enablers to the overall sustainability of the Activity's interventions, both of which will inform planning for the Activity's extension period. The findings from this research are expected to assist the Vietnam Mission and its implementing partner (IP), the Partnership for Health Advancement in Vietnam (HAIVN), in making decisions related to evaluating the effectiveness of the current approaches to promote competency-based medical education (CBME) and preparing and adapting future work planning based on lessons learned from the current Activity and its sustainability. The recognition of the need for change in medical education in Vietnam comes in the context of an international movement toward CBME. CBME focuses on specific domains of competence, such as medical knowledge and communication skills, which students must be able to attain and demonstrate to graduate and practice. Within longstanding traditional six-year medical education programs in Vietnam, undergraduate students learn basic science for the first two years. They do not approach clinical learning until the first semester of their third year. Clinical training starts from Year 3 to the end of the course. In the reformed, competency-based curriculum at IMPACT-MED UMPs, clinical training is designed to take place one year earlier or more, to build upon preclinical knowledge at the beginning of Year 2. This change allows students to see patients and develop their professional knowledge and skills better than traditional programs. At the postgraduate level, universities such as the HCMC UMP have embarked on reforming their curricula to adapt to the undergraduate CBME curriculum. The Activity Review focuses on the investigation of three main aspects: (1) IMPACT-MED Activity support and the development and implementation of CBME at the five UMPs, (2) stakeholder engagement and partnership in the development of CBME, and (3) the sustainability of CBME. More specifically, the Activity Review looks into the approaches to advance CBME development with the goal of drawing lessons learned from what has been implemented in Phase 1. The Activity Review also sheds light on the barriers and enablers of CBME to inform work planning for Phase 2 and safeguard the sustainability of CBME in Vietnam. METHODS An adaptable and comprehensive mixed-method qualitative approach helped gather input through key informant interviews (KIIs), focus group discussions (FGDs) with key stakeholders, direct observations with two out of five activity UMPs, and a series of participatory events. By the end of the data collection at each UMP, a two-way consultation meeting was held for the research team to present the preliminary findings drawn from the time spent with or at each UMP. Toward the end of data collection, a validation vii | IMPACT-MED ACTIVITY REVIEW USAID.GOV workshop with key stakeholders was organized to validate collected and analyzed data and co-develop recommendations. Additionally, a two-way learning and utilization workshop with the five active partner UMPs; HAIVN; and government, external, and Phase 2 stakeholders was organized to promote the utilization of research findings and recommendations. In total, the research team conducted 15 group KIIs and 16 FGDs, which engaged 139 participants; in addition to 138 participants from participatory events, including 20 participants from an initial consultation meeting, 46 participants from five two-way consultation sessions at UMPs, 34 participants from the Plenary Validation Event, and 38 participants from the Learning and Utilization Event. There was a total of 277 participants in the review, although many participants attended several of the events. These collected data were iteratively analyzed to provide inputs for the Findings, Conclusions, and Recommendations Matrix. The participatory events integrated into this research provided opportunities for local stakeholders to share their valuable insights and co￾develop recommendations for HAIVN to reflect on their work planning and USAID’s future Activity design. RESEARCH QUESTIONS This Activity Review is organized around three main themes and eight sub-questions: 1. UMP beneficiaries and competency-based education: a) What significant changes have resulted for students and faculty as a result of the Activity’s support surrounding CBME? b) To what extent has the Activity helped UMPs achieve their priorities linked to developing a CBME curriculum? c) What institutional challenges should be addressed in Phase 2 (2022-2026) of the Activity to better measure, institutionalize, and sustain CBME at UMPs? d) In what ways were students and faculty engaged or involved during the Activity, and how did the Activity empower them in reforming the CBME curriculum? 2. Stakeholder engagement and partnership: a) What were the opportunities, challenges, and lessons learned that emerged from aligning priorities and activities between different stakeholders and/or complementary projects in developing a CBME curriculum? b) What worked well with private sector involvement to advance CBME, and where is there room for improvement in the future to enhance the benefit of private sector involvement? 3. Sustainability: a) What developments, for example, in terms of support for institutional capacity, internal quality assurance (IQA), or UMP policy, could help to ensure the sustainability of CBME? b) What actions should be taken to safeguard the sustainability of CBME in Vietnam and how should any barriers or enablers presented by national policies be navigated? FINDINGS AND CONCLUSIONS Substantial, positive, and permanent changes were found in the teaching and learning practices of faculty and students at the five Activity UMPs. UMPs have adopted CBME and implemented their own CBME programs with varied progression, achievement, and levels of comprehensiveness. HCMC UMP has led curriculum innovation in all CBME aspects that were adopted and deployed in Vietnam, with Thai Nguyen UMP and Hue UMP also having made significant progress. With fewer resources, progress at Hai Phong and Thai Binh has been slower, but they have still achieved positive results. The support provided to UMPs by HAIVN to develop technical expertise on the basis of UMP-specific needs was timely and critical to the deployment of CBME programs. Further, technical assistance effectively complemented the streamlined Health Professionals Education and Training (HPET) project. viii | IMPACT-MED ACTIVITY REVIEW USAID.GOV Support mechanisms orchestrated through a UMP-based IP contact point and technical working groups with the active participation of Harvard Medical School (HMS) experts have proved effective and have satisfied UMP needs. The Activity has helped to develop the capacity of faculty and students in medical education and subsequently enhanced the institutional capacity of UMPs in the Activity system. UMPs have now developed the fundamental skills to introduce and develop further educational innovations, and their faculties have the capacity to develop other CBME curricula while managing the delivery of CBME programs. Students have actively participated in CBME programs, become more self-directed in their learning, and demonstrated improved clinical skills relative to before programming. UMP leaders, particularly managers and faculty members, expended significant levels of effort to overcome challenges involved with the CBME implementation process. Key remaining challenges belong to three domains: (1) resources for CBME, including financing, knowledge and know-how, and human resources issues such as low faculty/student ratios and physical facilities; (2) policy and institutional autonomy; and (3) curriculum organization. The challenges UMPs face vary based on their individual circumstances, so their approach and required level of effort to tackle the challenges also differs. Activity UMPs have not adequately mobilized resources from external stakeholders other than hospitals for CBME implementation since the UMPs have generally not met extensively with the private sector. The partnership with the private sector has not been well-prioritized by UMP leaders and a detachment persists, which prevents both sides from making partnerships successful. For CBME to be sustainable at the UMPs, a number of developments are required in the coming years. All UMPs receiving development assistance have to take a cooperative and collegial orientation toward one another and remain determined to evolve the initial developed CBME programs for the group as a whole. CBME ought to be integrated within other programs at both undergraduate and postgraduate levels. To do this successfully, UMPs will have to utilize their resources to overcome significant barriers, one of which remains the low faculty/student ratio. In regard to CBME sustainability in Vietnam more widely, technical resources should be shared among non-Activity UMPs. A national policy environment fostering CBME also ought to be advocated for, which ultimately requires the involvement of various unaligned institutions in a collaborative spirit, especially the Ministry of Health (MOH) and the Ministry of Education and Training (MOET). UMPs have achieved the initial success of introducing CBME into their curriculum reforms. For the long￾term development of CBME and its extensiveness and presence in other programs, UMPs’ efforts to evolve curricula and explore the utilization of all available resources will play a crucial role. However, for the sustainability of system-wide CBME, the support and participation of different stakeholders, state agencies, international development organizations and donors, and the private sector are of continued significant importance. RECOMMENDATIONS This Activity Review’s recommendations center around two main objectives: (1) safeguarding the sustainability of CBME and (2) strengthening the partnership between UMPs and the private sector. To safeguard the sustainability of CBME at Activity’s UMPs, it is recommended that UMPs: ● Continue the development and implementation of existing CBME curricula systematically and consistently and expand CBME to revise and improve other programs. ix | IMPACT-MED ACTIVITY REVIEW USAID.GOV ● Intensify IQA and CQI efforts, including curriculum review and revision, a graduate competency assessment, the development of a stakeholder feedback system, and specialized program accreditation. ● Further develop and formally launch a staff professional development unit and specific support services for CBME students. ● Improve internal and external cooperation and communication about CBME and curriculum innovation. ● Develop community engagement as an operational domain, potentially as an extra-curricular option. Actions requiring joint efforts of multiple actors include: ● Improve faculty/student ratios. ● Share knowledge about CBME, local experiences, and practical lessons in CBME implementation through e-libraries and workshops. ● Develop legal corridors that foster CBME, such as regulations on competency-based standards for professional licensing. ● Develop a map of CBME components as a point of reference for new CBME adopters. ● Develop networks of local CBME UMPs, faculty, and students to build a local CBME community and connect with regional and international CBME communities. ● Develop specialized accreditation at both program and institutional levels. ● Increase access and opportunities to develop local professional expertise from consultation with technical advisors. To advance partnerships with private sector partners, interventions can be made at different levels. For UMPs, leaders should: ● Strengthen alumni networks and alumni networking activities. ● Prioritize and be strategic in developing partnerships with industries. ● Share the experience of implementing Decree 111 to strengthen relationships with affiliated hospitals, as a way to increase the number of clinical faculty and to set up UMP dialogues on this issue. For HAIVN: ● Expand the network of private sector partners to participate and provide additional technical support to local UMPs. ● Expand collaboration with HMS, Beth Israel Deaconess Medical Center, and Brigham and Women’s Hospital to explore faculty and student exchange and research collaboration. For USAID, the Government of Vietnam (GVN), and other relevant actors: ● Set up a board or a committee to act as an intermediary connecting different actors and to facilitate information-sharing dialogues among parties. ● Organize annual donor and sponsor roundtables or forums. 1 | IMPACT-MED ACTIVITY REVIEW USAID.GOV INTRODUCTION The Partnership for Health Advancement in Vietnam (HAIVN) began implementing the Improving Access, Curriculum and Teaching in Medical Education and Emerging Diseases (IMPACT-MED) Alliance in May 2016 with a budget of 8.76 million United States Dollars. HAIVN is a collaboration between Harvard Medical School (HMS), Beth Israel Deaconess Medical Center, and Brigham and Women’s Hospital, and has been working to develop the capacity for high-quality medical education, research, and healthcare quality in Vietnam since 2003. The goal of IMPACT-MED is to improve the quality and effectiveness of medical education in Vietnam to produce graduates and health care workers who can meet emerging health needs. Under IMPACT-MED, HAIVN has been supporting five Universities of Medicine and Pharmacy (UMPs) across Vietnam to comprehensively reform their six-year undergraduate medical education curriculum and three-year postgraduate surgical residency program toward competency-based medical education (CBME). Following its first phase, IMPACT-MED was extended for another five years (2022-2026). In light of the extension, this Activity Review provides an opportunity to investigate the implementation gaps and contextual factors influencing CBME delivery at UMPs, as well as the barriers and enablers to advancing CBME in the Activity’s extension phase. The overarching objectives of this Activity Review are to assess the suitability of the Activity’s assistance in CBME development and implementation at UMPs, the effectiveness of what has been carried out with regards to CBME development, and the overall sustainability of CBME at these UMPs. This research set out to: (1) draw lessons from collaboration with a bilateral funding mechanism, private sector partnerships, and future key priorities; (2) gauge the sustainability of CBME and provide recommendations to enhance the sustainability of new CBME curricula; and (3) reflect on the overall trajectory of the Activity and contextual factors that ultimately influence subsequent work planning. BACKGROUND The World Bank identifies “skills gaps of university graduates relative to labor market needs” in its 2020 Vietnam Higher Education Sector Report. The report raises “content-based curriculum” and “weak involvement of industry partners into curriculum development process[es]” as two causes for insufficient education quality and skills gaps. Although the Ministry of Education and Training (MOET) has requested that universities develop student learning outcomes (SLOs) for all programs and courses since 2009, competency-based education has not yet been fully implemented in the local higher education system. This is a major bottleneck that hinders the improvement of teaching and learning practices in Vietnamese higher education in general, and medical education in particular. The recognition of the need for change in medical education in Vietnam comes in the context of an international movement toward CBME. CBME focuses on specific domains of competence, such as medical knowledge and communication skills, that students must be able to attain and demonstrate in order to graduate and practice. CBME was first introduced in medical education in North America in the 1960s and was recognized by the World Health Organization (WHO) in 1978 as a preferred model of training that generates a health professional “who can practice medicine at a defined level of proficiency, in accord with local conditions, to meet local needs” (McGaghie et al., 1978). In 2015, the Vietnamese Ministry of Health (MOH) produced a document that specifies the competency standards for general practitioners, which includes four general domains: professional practice, application of medical knowledge, competence in medical care, and communication (Decision 1854/QD-BYT, dated May 18, 2015). Within longstanding, traditional six-year medical education programs in Vietnam, undergraduate students typically learn basic science in the first two years with limited clinical exposure. Clinical training starts 2 | IMPACT-MED ACTIVITY REVIEW USAID.GOV from Year 3 to the end of the course. In the reformed CBME curriculum at the five IMPACT-MED UMPs, early exposure to patients and the clinical environment starts as early as the second semester of Year 1. This change allows students to see patients and develop their professional knowledge and skills better than traditional programs. HCMC UMP adopted CBME to revise its three-year surgical residency curriculum and this new curriculum has been implemented over the past two years. The network of higher education institutions (HEIs) of medical education in Vietnam includes 30 universities and colleges, most of which are public institutions. The state management of these HEIs is fragmented and complicated due to the involvement of the MOET; the MOH; the Ministry of Labor, Invalids, and Social Affairs (MOLISA); and local authorities. According to data compiled by the MOH and the Administration of Science Technology and Training, the 16 universities under their oversight, which include 14 public universities and two private universities, educate a total of 100,418 students, 81 percent of whom are undergraduates and 19 percent of whom are postgraduates. These HEIs each offer between four to 16 undergraduate degree programs and many programs at the postgraduate level. IMPACT-MED works with five key UMPs: the Ho Chi Minh City (HCMC) UMP, Hue UMP, Thai Nguyen UMP, Thai Binh UMP, and Hai Phong UMP. Three UMPs (HCMC UMP, Hue UMP, and Thai Nguyen UMP) have participated in the Activity since its beginning, while Thai Binh UMP and Hai Phong UMP, began participating later, in 2017. These five universities have 35,838 undergraduate students and 7,716 postgraduate students. They offer more than 150 degree programs, most of which are for postgraduates. Some programs are accredited by local independent agencies and several programs have been assessed by regional and international accrediting agencies, such as the Association of Southeast Asian Nations University Network – Quality Assurance and the Conférence Internationale des Doyens des facultés de PHARMacie d’Expression Française (HCMC UMP, 2021).1 At the institutional level, most UMPs are accredited for the national standards by local accreditors, but none have achieved international or regional accreditation (MOET, 2021). To improve and innovate undergraduate medical education, IMPACT-MED’s interventions provided technical assistance to comprehensively reform the six-year training program for general medical doctors. Training led by HMS educators was provided to improve faculty capacity and skills in curriculum design, active learning, and clinical teaching. The Activity also created a faculty community to promote innovation. A close partnership with the MOH was intended to facilitate the sharing of successful models and lessons learned with universities nationwide. IMPACT-MED also aimed at improving postgraduate medical education through reforming surgical training programs and policies. By the end of 2022, the new six-year undergraduate CBME curriculum and the three-year surgical residency program will be completed and implemented at the five UMPs at different stages. The first phase of IMPACT-MED has made important progress in the development and initial implementation of CBME curricula in Vietnam. The next phase should continue to provide technical assistance for both active and new UMPs to implement and evolve on these curricula. Nuanced 1 For example, Bachelor of Pharmacy and standard Doctor of Medicine programs at Vietnam National University￾Hanoi, are accredited by the Vietnam National University-Ho Chi Minh Center for Education Accreditation and the Center for Education Accreditation-Association of Vietnam Universities and Colleges, respectively; Doctor of Odonto-Stomatology, Masters of Preventive Medicine, and Specialist Doctor 1 programs provided by Thai Nguyen UMP are accredited by the Center for Education Accreditation-Association of Vietnam Universities and Colleges and others. 3 | IMPACT-MED ACTIVITY REVIEW USAID.GOV programming focused on adapting lessons learned from the first phase of IMPACT-MED will help sustain CBME gains in the Activity UMPs and boost spillover effects to other local UMPs. PURPOSE AND AUDIENCE The objectives of this Activity Review are: 1. Learn about the extent to which the activity has met its beneficiaries’ educational needs. 2. Identify barriers and enablers to the overall sustainability of the activity's interventions, both of which will inform planning for the activity's extension period. The primary audience for this report includes: the United States Agency for International Development (USAID) Vietnam Mission, HAIVN, the five active UMPs that have participated in IMPACT-MED activities, private stakeholders working with these universities, and the various Government of Vietnam (GVN) ministries (MOH, MOET, MOLISA) whose interests align with IMPACT-MED’s objectives, and other international agencies, such as the WHO and the World Bank. Secondary audiences include other UMPs in Vietnam that are being added to IMPACT-MED’s Phase 2 extension, where CBME programming will be developed and implemented next. A tertiary audience is the wider medical education sector in Vietnam. RESEARCH QUESTIONS As developed by the Vietnam Mission, USAID Learns, and HAIVN, the research themes and questions that guide this Activity Review are: 1. UMP beneficiaries and competency-based education: a) What significant changes have resulted for students and faculty as a result of the Activity’s support surrounding CBME? b) To what extent has the Activity helped UMPs achieve their priorities linked to developing a CBME curriculum? c) What institutional challenges should be addressed in Phase 2 (2022-2026) of the Activity to better measure, institutionalize, and sustain CBME at UMPs?2 d) In what ways were students and faculty engaged or involved during the Activity, and how did the Activity empower them in reforming the CBME curriculum? 2. Stakeholder engagement and partnership: a) What were the opportunities, challenges, and lessons learned that emerged from aligning priorities and activities between different stakeholders and/or complementary projects in developing a CBME curriculum? b) What worked well with private sector involvement to advance CBME, and where is there room for improvement in the future to enhance the benefit of private sector involvement? 3. Sustainability: a) What developments, for example, in terms of support for institutional capacity, internal quality assurance (IQA), or UMP policy, could help to ensure the sustainability of CBME? b) What actions should be taken to safeguard the sustainability of CBME in Vietnam and how should any barriers or enablers presented by national policies be navigated? 2 Challenges or opportunities may relate to the granting of greater autonomy to universities or universities’ capacity in practicing autonomy. 4 | IMPACT-MED ACTIVITY REVIEW USAID.GOV METHODOLOGY OVERVIEW The methodological approach of this research was designed to generate findings that supported reflection and planning, particularly concerning the sustainability of IMPACT-MED’s focal activity, the advancement of CBME in Vietnam, and how CBME related to grassroots stakeholders. For this purpose, the research team integrated a participatory approach along with standard qualitative methods. Within the scope of a cross-sectional study, through various meeting sessions, including an initial consultation meeting; mini￾validation sessions and consultations; a plenary Validation Event; and a Learning and Utilization Event, the Activity Review provided opportunities for local stakeholders to co-develop recommendations and identify key actions required to address challenges for the extension phase and the overall sustainability of CBME implementation. The employed methods that were utilized in the review are described below. RESEARCH DESIGN The study employed a thorough document review along with standard qualitative methods integrated with a participatory approach to data collection. The qualitative approach consisted of both individual and group key informant interviews (KIIs), focus group discussions (FGDs), and participatory sessions. Respondents were selected from a range of stakeholder groups including UMP students and faculty, USAID and HAIVN staff, GVN officials, private sector actors, and international agency representatives. A full list of KII and FGD respondents is included in Annex I. Table 1 summarizes the research design for the study by linking the data collection tools, data sources, and analysis focus for each research question. 5 | IMPACT-MED ACTIVITY REVIEW USAID.GOV Table 1: Research Design RESEARCH QUESTION DATA COLLECTION TOOLS DATA SOURCE CONTENT ANALYSIS FOCUS 1. UMP beneficiaries and competency-based education 1a) What significant changes have resulted for students and faculty as a result of the Activity’s support surrounding CBME? − Desk review and document analysis − FGDs with current students and faculty − KIIs or facilitated dialogues with UMP leaders and managers, implementing partners (IPs), private sector partners, and other external stakeholders, such as state agencies and international organizations − Mini-questionnaire − Direct observation − Curricular documents, course syllabi, reports, websites − Discussion notes, transcripts − Interview notes, transcripts − Quick checklist/questionnair e for students − Physical facilities for teaching and learning; labs To identify changes in UMP practice and literature during the IMPACT-MED’s active period under its activities 1b) To what extent has the Activity helped UMPs achieve their priorities linked to developing a CBME curriculum? − Desk review and document analysis − FGDs with current students and faculty − KIIs or facilitated dialogues with UMP leaders and managers, IPs, private sector partners, and other external stakeholders, such as state agencies and international organizations − Strategies and plans; reports; project documents − Discussion notes, transcripts − Interview notes, transcripts To identify the gaps between UMPs’ needs and what has been provided in relation to CBME development, financially, technically, and contextually (policies and legislative corridors, etc.) 1c) What institutional challenges should be addressed in Phase 2 (2022 - 2026) of the Activity to better measure, institutionalize and sustain CBME at UMPs? − Document analysis − FGDs with current students and faculty − KIIs or facilitated dialogues with UMP leaders and managers, IPs, private sector partners, and other external stakeholders, such as state agencies and international organizations − Reports, journal articles − Discussion notes, transcripts − Interview notes, transcripts To pinpoint the issues, with regards to institutional arrangements, that may hinder the implementation, effectiveness, and sustainability of CBME at UMPs. 6 | IMPACT-MED ACTIVITY REVIEW USAID.GOV RESEARCH QUESTION DATA COLLECTION TOOLS DATA SOURCE CONTENT ANALYSIS FOCUS 1d) In what ways were students and faculty engaged/involved during the Activity and how did the Activity empower them in reforming the CBME curriculum? − Document analysis − FGDs with current students and faculty − KIIs or facilitated dialogues with UMP leaders and managers, IPs, private sector partners, and other external stakeholders, such as state agencies and international organizations − Reports − Discussion notes, transcripts − Interview notes, transcripts To identify whether and how the level of engagement of students and faculty changed over the course of the Activity 2. Stakeholder engagement and partnership 2a) What were the opportunities, challenges, and lessons learned that emerged from aligning priorities and activities between different stakeholders and/or complementary projects in developing a CBME curriculum? − Desk review and document analysis − FGDs with faculty − KIIs or facilitated dialogues with UMP leaders and managers, IPs, private sector partners, and other external stakeholders, such as state agencies and international organizations − Reports − Discussion notes, transcripts − Interview notes, transcripts To explore the issues in aligning different projects and activities with CBME priorities at UMPs 2b) What worked well with private sector involvement to advance CBME, and where is there room for improvement in the future to enhance the benefit of private sector involvement? To detect the elements that help to strengthen the involvement of the private sector in CBME advancement 3. Sustainability 3a) What developments, for example, in terms of support for institutional capacity, IQA, or UMP policy, could help to ensure the sustainability of CBME? − FGDs with current students and faculty − KIIs or facilitated dialogues with UMP leaders and managers, IPs, private sector partners, and other external stakeholders, such as state agencies and international organizations − Discussion notes, transcripts − Interview notes, transcripts An action plan for the sustainability of CBME 7 | IMPACT-MED ACTIVITY REVIEW USAID.GOV RESEARCH QUESTION DATA COLLECTION TOOLS DATA SOURCE CONTENT ANALYSIS FOCUS 3b) What actions should be taken to safeguard the sustainability of CBME in Vietnam and how should any barriers or enablers presented by national policies be navigated? − Desk review − FDGs − KIIs or facilitated dialogues with UMP leaders and managers, IPs, private sector partners, and other external stakeholders, such as state agencies and international organizations − Reports, journal articles − Discussion notes, transcripts − Interview notes, transcripts Recommendations on how to sustain CBME 8 | IMPACT-MED ACTIVITY REVIEW USAID.GOV RECRUITMENT PROCESS The approach to respondent selection for the KIIs and FGDs was as follows: UMP students, faculty, and leadership: To recruit UMP respondents, the research team used a snowball sampling approach. HAIVN provided a primary point of contact at each university, with whom the research team worked to identify respondents. The research team organized three FGDs for each institution, with up to ten individuals who had been involved in and benefitted from the IMPACT-MED Activity, and one KII per university with leadership. The research team requested equal numbers of male and female respondents during FGD recruitment. The research team also requested the recruitment of disadvantaged students and students from ethnic minorities for student FGDs. The primary point of contact of each UMP also provided the research team with contacts of employers (hospitals and clinics); one KII and one FGD were conducted with these stakeholders. Other stakeholder groups: The research team consulted with HAIVN to generate a list of potential respondents and contact information for each other stakeholder group. The research team then contacted all individuals whose contact information was shared and invited them for KIIs. Out of five contacts provided for private sector partners, three organizations participated in KIIs, and while the other two did not respond or indicated they were not available to participate. Two available private sector partners had two respondents participate in the KIIs, so there were a total of five KII participants. For state agencies and international donors, the team was able to reach all the given contacts, except for the MOET. DOCUMENT REVIEW For the desk review, the team reviewed a broad range of documents to ensure that the research design, analysis, and recommendation development process remained both evidence-driven and tied to the local context. Documents included project documents and reports, such as quarterly and annual progress reports between 2017-2021. The research team also reviewed relevant reports produced by other international donors, as well as locally and internationally published peer-reviewed journal articles related to medical education in Vietnam. Legal documents concerning higher education autonomy were also an important component of the desk review. These documents included the Revised Higher Education Law and Decree 99; the National Qualification Framework; regulations on common quality framework and standards in medical education, and others specific to the development of CBME. Finally, to capture contextual information about training scale, curriculum development, and CBME practices, the team also conducted a rapid review of gray sources, such as relevant websites and internal documents related to the medical education system in Vietnam and the Activity UMPs. Annex III shows the list of documents reviewed. KIIs The research team conducted both individual and group KIIs with key stakeholders, including UMPs leaders, HAIVN staff, GVN counterparts, private sector partners, international donors, and employers to gather in-depth qualitative information from a wide range of perspectives. In total, the research team conducted a total of 15 semi-structure KIIs, two of which were in-person while the rest were carried out virtually on Zoom based on respondent preference. All KIIs were conducted by at least three research team members: a key researcher, a research specialist, and a research coordinator to take notes. To facilitate data analysis and to assure the precision of the KII notes, the KIIs were recorded upon the agreement of respondents and then transcribed. The KIIs were conducted in either English or Vietnamese, depending on respondent preference. All KII notes and 9 | IMPACT-MED ACTIVITY REVIEW USAID.GOV transcripts were professionally translated into English as needed, filed electronically on Social Impact’s secure server, and included as part of the qualitative data set. Table 2 details the number of respondents engaged in KIIs, disaggregated by sex. Table 2: Number of KII Respondents by Key Stakeholder Group, Disaggregated by Sex STAKEHOLDER GROUP TOTAL KII RESPONDENTS MALES FEMALES USAID 1 0 1 IP (HAIVN) 3 3 0 GVN Agencies 1 1 0 UMPs 7 4 3 Private Sector Partners 5 1 4 International Agencies 3 2 1 Employer Partners 1 1 0 TOTAL KII RESPONDENTS 21 12 9 FGDs The review also included a series of FGDs, which allowed respondents to formulate and articulate their opinions and perceptions related to the Activity and the development of CBME in a group setting. The research team conducted three FGDs at each UMP with students and faculty. In selecting participants for the students’ group, the evaluation team ensured that the discussions included students who had experience learning under the newly developed CBME curriculum. Similarly, the research team required the lecturers invited for FGDs to have participated in the delivery of the new CBME curriculum or related elements. The research team ensured that the group of managers included the heads of relevant functional departments, such as academic affairs and quality assurance; deans of departments that have employed CBME; and Activity coordinators. Several lead lecturers who were directly involved in the development of CBME were also invited to participate. Each FGD was composed of five to ten participants. Similar to KIIs, FGDs were conducted online and in-person. At least three members of the research team took part in each FGDs, with two researchers facilitating the dialogue and a coordinator taking notes. The FGDs were recorded after receiving the respondents’ consent at the beginning of the discussion. Vietnamese was primarily used to conduct the FGDs; however, all notes were translated into English for analysis. Data collection instruments for KIIs and FGDs are included in Annex IV. Table 3 enumerates the FGD participants, disaggregated by sex. Table 3: Number of FGD Respondents by University and Employer, Disaggregated by Sex STAKEHOLDER GROUP TOTAL FGD PARTICIPANTS MALES FEMALES HCMC UMP 22 12 10 Hue UMP 26 17 9 Thai Nguyen UMP 22 10 12 Thai Binh UMP 23 11 12 Hai Phong UMP 23 13 10 Employers 2 1 1 TOTAL FGD RESPONDENTS 118 64 54 10 | IMPACT-MED ACTIVITY REVIEW USAID.GOV DIRECT OBSERVATIONS The research team conducted site visits at two UMPs: HCMC UMP, which was a frontrunner among the five Activity UMPs, and Hue UMP, which was part of a regional university. At Hue UMP, the research team was able to integrate more disadvantaged students from central Vietnam into the analysis. Direct observations at the two sites allowed researchers to better understand the physical arrangements and settings at faculty workplaces, lecture halls, smart classrooms, laboratories, a simulation center, computer rooms, and an objective structured clinical examination (OSCE) center.3 Direct observations at the sites also complemented the FGD and KII data by allowing the research team to observe and interact with stakeholders in a less formal setting. As a result, these direct observation data collected helped facilitate data triangulation to increase the credibility and validity of the research findings. Given the multiple partners involved in the Activity, triangulation was particularly important to shed light on complex organizational dynamics. MINI-SURVEY To assist with the discussions with undergraduate students, the research team conducted an online mini￾survey on Google Forms. The short questionnaire included three main sections: students’ personal information; learning experiences, such as learning methods, activities, and assessments; and extracurricular activities, such as participation in hackathons or start-ups. The link to the questionnaire was emailed to all students with a request to complete the survey by the day prior to the expected FGD. Most of the participating students completed the survey on time. The research team processed the data on Excel, and the results helped to modify and tailor discussion questions for students so as to increase the effectiveness of the discussions. The mini-survey helped to validate information collected from faculty about changes in the teaching and learning practices and the program literature published by UMPs. It also helped the research team save time for deeper investigations of other learning aspects. PARTICIPATORY EVENTS The Activity Review also included a series of participatory events that provided stakeholders with the opportunity to engage in the research process and reflect on its findings. These events allowed the Activity Review to benefit from the participation of key actors in the research process, and also helped to improve the utilization of the findings and create a foundation for sustainability. To enable the research to be informed by a participatory approach throughout the process, the team conducted participatory events during the inception phase, concurrently with data collection, and at the end of data analysis, prior to reporting. The three types of participatory events that the team conducted are described below. Stakeholder Consultation Meeting: The initial stakeholder consultation meeting was the first step to engage different stakeholders in the Activity Review and was carried out by the team during the development of the research design. Through the meeting, the research team acquired UMP-specific information to refine the Activity Review design and work plan. Participants' views also assisted the research team in shaping the definition of sustainability, outlining its dimensions, and revising the interview guides. UMP leadership, key stakeholders, and other senior actors were invited to provide the top-level management perspectives. 3 The OSCE center is where students learn in different formats such as lab practice, projects, tutorials, modeling, and also do multiple-choice questions and clinical examinations. 11 | IMPACT-MED ACTIVITY REVIEW USAID.GOV Facilitated Discussion and Validation Sessions: The research team organized a set of six two-way facilitated dialogues and mini-validations sessions throughout the data collection process to assure that decision-makers at UMPs provided their reflections on the research process and on the preliminary findings of the Activity Review at their institution. Five sessions were organized with smaller groups of participants from each UMP, along with one master plenary validation event for all stakeholders. Sessions at UMPs were held either on Zoom or in-person during site visits at HCMC and Hue UMPs. The plenary event was conducted via Zoom to facilitate the participation of a large number of stakeholders from different locations and provinces. The research team, using its perspective as an outside investigator, shared key findings and highlighted critical issues found during the Activity Review process. Participants were encouraged to express their thoughts, request further information, ask questions, and raise any relevant issues and concerns. The research team’s international and local experts on CBME, governance, quality assurance, and higher education helped provide basic technical advice within the context of CBME to UMPs based on the needs and urgency of the matters in question. Learning and Utilization Event: A Learning and Utilization Event led by USAID was held after data analysis. This full-day event served as an opportunity for USAID, HAIVN, and prospective partners to reflect on achievements and challenges to date. USAID Learns facilitated an afternoon workshop at this event to brainstorm with HAIVN and key local stakeholders on how to operationalize the research’s recommendations. This opportunity gave the five Activity UMPs the opportunity to coordinate and align on goals for the next phase of the Activity, share perceptions about the barriers and enablers to CBME advancement, and discuss the roles and requirements of each party going forward. Details of this event are provided in the Utilization section and within Annex V. Table 4: Number of Participants in Participatory Events, Disaggregated by Sex PARTICIPATORY EVENTS TOTAL PARTICIPANTS MALES FEMALES Stakeholder Consultation Meeting 20 12 8 Two-Way Consultations at UMPs 46 29 17 Validation Event 34 19 15 Learning and Utilization Event 38 19 19 TOTAL PARTICIPANTS 138 79 59 DATA ANALYSIS The research team employed qualitative data analysis methods, specifically thematic analysis, to analyze these data collected from KIIs, FDGs, site observations notes, and CBME program documents. Interview transcripts and field notes were coded by the research themes. Codes were identified for sub-themes as the analysis progressed. The research team used NVivo, a qualitative analysis software, to support the process. Results were triangulated across different data sources for validity. LIMITATIONS The research team recognized a number of potential threats to the quality of the research during its inception and subsequently proposed the following mitigation plan and strategies to address them. The biggest threat to the Activity Review remained staffing issues incurred by COVID-19, which limited the opportunity to conduct site observation. Other potential threats to the Activity Review were those applicable to any mixed methods qualitative research, such as respondent bias, language loss in translation, and bias due to researchers’ perspectives and sample selection. Table 5 provides a list of the research 12 | IMPACT-MED ACTIVITY REVIEW USAID.GOV limitations and the mitigation strategies that were employed to lessen their impact on data collection approaches and data validity. Table 5: Study Limitations and Mitigation Strategies LIMITATIONS GENERAL PROBLEMS & MITIGATION STRATEGIES COVID-19 Staffing issues among the research team due to COVID-19 infection limited the number of site visits proposed in the research design, while virtual data collection relied on many technological factors such as internet connections, services provided by third-party platforms, and device availability. To mitigate those shortcomings, the research team: (1) intensified the effort expended on the literature review and interview protocol development and (2) maintained personalized communication channels with UMP partners to gain insights. The team also used different online platforms to facilitate meetings, including Google Meet, Microsoft Teams, and Zoom. For meetings with a large number of participants, Zoom with registered hosting was used to assure smooth operations. Two site visits were able to be conducted in person and three were done virtually. Potential threat to data validity Since CBME is a relatively new concept, students had different perceptions of the CBME terminology and related terms. To reduce discrepancies and encourage a focus on facets related to competency development, the research team prepared a short questionnaire that included the key characteristics of CBME developed at Activity UMPs for student participants to complete at the start of FGDs. This enabled the research team to validate each FGD sample and collect precise data relevant to USAID’s programing at UMPs. Researcher Bias While the Activity Review was mainly focused on CBME, the related factors that affect its effectiveness and sustainability cover a wide range of aspects of university life, including policy; governance and management; staff professional development; curriculum development, testing, and assessment; and quality assurance in medical education. To be able to gauge a comprehensive picture of CBME implementation at UMPs, the research team included four local and international researchers with different backgrounds and expertise in higher education and medical education. The research team’s tools and analysis underwent an internal quality control process and were finalized in collaboration with an international medical education expert. Language issues Conducting grassroots-level interviews and dialogues in English proved inefficient due to limited stakeholder English comprehension, time constraints, and poor local translation service quality. To mitigate this, Vietnamese was mainly used in KIIs and FDGs with local participants to assure respondents had no difficulties in expressing themselves and no key information was missed. Vietnamese language interview notes were carefully and professionally translated into English for analysis and were reviewed by an international medical education expert. 13 | IMPACT-MED ACTIVITY REVIEW USAID.GOV FINDINGS AND CONCLUSIONS In this section, we present the findings and conclusions from the review. The presentation is organized in terms of the research questions listed in at the end of the report introduction section, which are grouped into the three research themes of (1) UMP Beneficiaries and CBME, (2) Stakeholder Engagement and Partnership, and (3) Sustainability. RESEARCH THEME 1: UMP BENEFICIARIES AND CBME RESEARCH QUESTION IA WHAT SIGNIFICANT CHANGES HAVE RESULTED FOR STUDENTS AND FACULTY AS A RESULT OF THE ACTIVITY’S SUPPORT SURROUNDING CBME? FINDINGS Faculty-wise: apparent shifts in methods and mindsets, along with heavier workloads The KII and FGD data illustrate a number of important changes experienced by faculty over the course of the Activity. These include new mindsets in relation to teaching, the application of new pedagogical approaches and methods, assessment methods, and improved class arrangement and delivery. Many faculty respondents indicated that the fundamental shift from a teacher-centered approach to a student-centered approach has led to major changes in how they perceive their roles and how they deliver classes. During discussions with faculty, respondents explained to the research team that the integrated and modularized curriculum focusing on competency development as student learning outcomes (SLOs) requires the development of new pedagogical and assessment methods, which places significant demands on faculty members. Student, faculty, and leadership FGD data from each UMP confirmed that lectures had been replaced with various new methods that facilitated active learning, such as flipped classrooms, problem-based learning, case-based learning, and group discussion. Likewise, assessments not only included summative methods, but also focus on formative and authentic assessments with various work￾based, competency-based methods, such as mini-clinical evaluation exercises (mini-CEX), entrustable professional activities (EPAs), and OSCE. Final tests are now mainly based on multiple-choice questions (MCQs). Other changes at all UMPs include the introduction of new subjects such as Scholarly Project, the promotion of research among students, an emphasis on professionalism, and the development of highly important professional skills and attitudes for medical students, such as empathy. Finally, some respondents indicated that the adoption of CBME has started to bring about spillover effects where faculty members at Activity UMPs have applied the CBME educational approach to curriculum revisions of other programs. In terms of curriculum organization, the research team observed that the new teaching and assessment methods require small groups of between ten to fifteen students, so the conventional large class size of 100 students or more must be split into several groups. The research team noted splitting students into smaller groups was apparent at four of the five Activity UMPs. Discussions with faculty members and managers at all UMPs indicated that the workloads of faculty members have increased significantly as a result of the innovations since they have to teach more classes for different groups of students, develop new teaching materials, write MCQs, and revise syllabi and lesson plans. The research team observed that some faculty working within the renovated program were no longer considered “teachers,” but instead as facilitators or instructors, which leaves greater room for student participation in the learning process and promotes the introduction of more active learning methods. The integrated curriculum requires lecturers to work more closely and maintain regular contact with students and other faculty so that they can jointly deliver more tailored classes. Faculty respondents 14 | IMPACT-MED ACTIVITY REVIEW USAID.GOV saw increasing collaboration in a positive light, as a motivator rather than a challenge; however, this perspective was not unanimously agreed upon by faculty members. Figure 1: Small Group Classes at Hue UMP and HCMC UMP Photo credit: USAID Learns An illustration of the changes in faculty culture and mindsets is provided by the Unit of Faculty Development (UFD) at HCMC UMP (see Box 1). The UFD is composed of core faculty members and was established to implement a systematic approach to providing internal training services. The UFD is a model that other UMPs are in the process of adopting. BOX 1: HCMC UMP’S UFD Established in June 2018, the UFD is an important part of a memorandum of understanding signed between HCMC UMP and Texas Tech University Health Sciences Center. Since September 2020, the UFD has supported HCMC UMP’s Medical Education Center as part of its mission to advance faculty development in medical education. The ultimate duties of the UFD are to develop and enhance faculty capacity in teaching and assessments of medical education; clinical practice, simulation, and scientific research; and practical competence for health care staff so that they are able to provide better services for their patients and community. The UFD is run by a core group of 78 qualified staff who have been thoroughly trained by a partnership between UFD and Texas Tech University Health Sciences Center, HMS, Ludwig Maximilian University of Munich, and Maastricht University. The UFD has developed a wide range of courses in CBME, such as Teaching Pedagogies and Assessment in CBME, Clinical Teaching and Simulation in CBME, and Integrating Basic Sciences in CBME. The UFD has also provided capacity training for many UMPs in Vietnam including Military Medical University, Hai Phong UMP, and Pham Ngoc Thach UMP. UFD also offered training of trainers in clinical practice for Cho Ray Hospital, Nhi Dong I Hospital, Nhan Dan Gia Dinh Hospital, Hung Vuong Hospital, and 146 core trainers from other UMPs all over South Vietnam. 15 | IMPACT-MED ACTIVITY REVIEW USAID.GOV Figure 2: A Training Session on Clinical Teaching and Simulation Provided by the HCMC UMP UFD Photo credit: UFD Student-wise: greater participation and higher motivation FGD data from sessions with students and faculty across the UMPs indicates that the shifts in curriculum design, educational approaches, and teaching methods have led to changes in students’ learning practices. At all five UMPs, student respondents indicated that they had shifted from passive learning to active learning due to the introduction of new pedagogical methods. Peers, faculty, and leaders reported that students under the renovated program have demonstrated more active learning behaviors and habits, instead of the passive learning dominant within local higher education settings. Specific changes in learning practices that students described during the FGDs included spending more time on independent learning, preparing for classes in advance, searching for materials and information, and taking pretests or readiness assessment tests. As a result of learning in a “flipped classroom,” students explained they are now required to absorb theoretical knowledge prior to classes, and which maximizes the in-class time spent working in groups, joining group discussions, and doing lab practice. Both undergraduate and postgraduate student respondents tended to express the view that the CBME curriculum with clearly defined SLOs had helped students to better develop their self-directed learning capability. As a Year 3 resident student described, “With clear learning goals, we know what to learn, where to go next; with closer supervision and instant feedback, we know what we have achieved, what should be improved.” KIIs with other UMP stakeholders corroborated the view that student learning practices had improved. Details concerning student participation are discussed in greater detail under Research Question 1D. There was some concern expressed by junior students, mostly in Year 1 and Year 2, about the difficulties they faced and the effectiveness of the new learning methods such as flipped classrooms; however, the concern diminished and effectiveness increased in later years as more senior students commented. This issue is related to student academic support, which can be solved with customized or personalized student services provided at the early stage of the program, as discussed under research question 1D. 16 | IMPACT-MED ACTIVITY REVIEW USAID.GOV Positive changes in clinical practice One aspect of the renovated curriculum’s design is accelerated clinical practice for students, as early as Year 2 of the revised six-year undergraduate program instead of Year 3 in the traditional program. Faculty respondents expressed the view that smaller group learning with closer supervision has resulted in more effective clinical practice instruction. Similarly, respondents cited the role of timely and constant feedback in improving learning effectiveness and students’ clinical practice experiences. At the undergraduate level, as articulated by student respondents, earlier exposure to clinical settings helps students to feel that their daily study is more applicable to their medical professional career, which students said resulted in greater motivation for their study. At the postgraduate level, resident students at HCMC UMP reported satisfactory, less stressful clinical learning experiences thanks to clear SLOs and positive, constructive feedback, which helped increase their confidence. Faculty members who worked as supervisors during clinical practice and local hospital doctors who worked as invited lecturers to support UMPs with student supervision recognized improvements in student outputs. At the majority of the UMPs, students reported that “soft skills” that are highly applicable to clinical practice, such as communication, teamwork, questioning, and time management had improved, while students at Thai Nguyen UMP highlighted improvement in English for Special Purposes (ESP). Supervisor and employer respondents confirmed that CBME students showed a high level of confidence when communicating with patients and asking for patient medical records. Gender mainstreaming and social inclusion The research team did not identify immediate concerns related to gender equity and social inclusion in implementing CBME at the Activity’s UMP partners. None of the female faculty members who participated in KIIs and FGDs reported that they had been treated unfairly or unequally, though they had not received any special privileges in the workplace. They expressed that they preferred to be treated based on their capabilities rather than their sex. The majority of female faculty who were asked felt that the incentive policies for female staff as per state regulations were sufficient. Selected disadvantaged students, including minority ethnic students and students with financial difficulties, indicated that the Activity’s support conformed with local and national policies. During FGDs, student respondents confirmed that apart from ordinary state policies,4 no other forms of specific support for these groups of students were apparent and that the challenges in student life that they faced were common to all students and not specifically associated with their socio-economic backgrounds.5 CONCLUSIONS Students and faculty respondents consistently reported experiencing positive changes as a result of the Activity. Various pedagogical methods employed by faculty members have motivated students to more actively participate in the learning process. Clinical teaching and learning have developed significantly, as earlier, hands on, clinical training now appears more applicable to students. However, it is important to note that while CBME has been deployed at the five UMPs participating in the IMPACT-MED Activity since 4 Depending on the socio-economic status of an ethnic minority pupil’s family (often based upon being from a Program 135 locality), she or he will receive (minimal) additional financial support from various sources. 5 The Law on Gender Equality 2006, the Law on Vocational Education 2014, the Law on Amendments to the Law on Higher Education 2018, and the Law on Education of Vietnam 2019 all address gender equality in education. 17 | IMPACT-MED ACTIVITY REVIEW USAID.GOV 2016, the implementation progress and scope varied. UMPs are at different stages of implementation, so different needs and challenges exist. After five years, key elements of CBME for long-term implementation are present. The key enablers of CBME, such as faculty capacity, facilities, and varieties of competency-based teaching and assessment methods have been sufficiently developed relative to before the Activity.6 With the adoption of CBME, teaching and learning practices at Activity UMPs have undergone substantial and irreversible changes. Positive responses to the changes were shown by all internal stakeholders, including female faculty. In parallel with the positive changes, some problems and concerns were also reported, including excessive workloads for faculty, students learning difficulties in early years, and the accessibility of student support services beyond longstanding UMP arrangements. RESEARCH QUESTION 1B TO WHAT EXTENT HAS THE ACTIVITY HELPED UMPS ACHIEVE THEIR PRIORITIES LINKED TO DEVELOPING A CBME CURRICULUM? FINDINGS High levels of satisfaction due to the application of need-based support As discussions with HAIVN and university leadership confirmed, IMPACT-MED commenced in 2016 and complemented the World Bank/MOH Health Professionals Education and Training (HPET) project (2015- 2020). HPET mainly invested in “hardware,” such as physical facilities and equipment for medical education curriculum innovation, while IMPACT-MED supported UMPs to develop “software” for the introduction of CBME, namely the theoretical and practical technical substance behind the curriculum. As a leader at Hue UMP confirmed: “IMPACT-MED came after HPET; the University knew which resources could serve which priorities... IMPACT-MED came timely; HPET provided huge support in physical facilities but lacked technical support. The two projects complemented one another, so very effective.” As mini-validations undertaken by the research team at each UMP confirmed, need-based support for each UMPs’ specific context and condition was provided through the UMP-based contact point and technical working groups. Through such contact points, faculty and management at all five UMPs expressed that they were well informed about available support from HAIVN. UMPs were able to tailor their development assistance, so management thereby selected options they had pre-prioritized as internal goals for their agenda. As a result, the CBME Doctor of Medicine programs across the five UMPs appear to vary and the apparent variations reflect the strategic priorities of each UMP and their readiness or capacity to undertake renovations. Further, KII data suggests the involvement of the Activity’s private sector partners in CBME programs also differed because of the individual strategic interests of each UMP. Respondents from the IP (HAIVN) and university leadership explained that technical working groups, composed of two to three members from each UMP and relevant HMS experts, were set up as part of the Activity to work intensively on particular issues such as clinical teaching, student support, ESP course development, and MCQ development and improvement. The discussions and agenda for the technical 6 How these enablers more specifically apply to CBME sustainability is discussed more under Research Theme 3: Sustainability. 18 | IMPACT-MED ACTIVITY REVIEW USAID.GOV working groups were developed on the basis of the needs of UMPs. According to faculty and management respondents, this support mechanism was effective and popular among UMP actors. Finally, faculty and leadership respondents indicated that the annual national medical education conference facilitated by the Activity provided a valuable opportunity for UMPs to share their interest and priorities in CBME advancement, seek assistance, and develop CBME networks. CONCLUSIONS The Activity’s support has met each UMP’s needs for evolving CBME technical knowledge. University leadership at each UMP indicated that as a technical support package, IMPACT-MED effectively assisted UMPs in developing and implementing curriculum reform. This was evidenced by how IMPACT-MED dovetailed with other development assistance projects, such as HPET, in helping UMPs to better utilize investments in physical facilities and equipment. The Activity’s support mechanisms, including technical working groups and UMP-based contact points, proved adaptable, since programming was tailored toward particular stakeholder interests and strategic goals. RESEARCH QUESTION 1C WHAT INSTITUTIONAL CHALLENGES SHOULD BE ADDRESSED IN PHASE 2 (2022 - 2026) OF THE ACTIVITY TO BETTER MEASURE, INSTITUTIONALIZE, AND SUSTAIN CBME AT UMPS? FINDINGS Limited or a lack of resources is a key institutional challenge The FGD and KII data illustrate that technical expertise and knowledge of CBME curriculum development and practices will continue to be a critical resource for the long-term development of CBME at UMPs. Expertise needed for the second phase will center on discipline-specific CBME for residency education, clinical pedagogies, and assessment methods. As the research team observed, the shortage of physical facilities for competency-based training and assessments that enable active learning methods and clinical practice remains an issue for UMPs. HCMC UMP was able to invest in advanced equipment for OSCE facilities and a simulation center, and the university leadership shared that they plan to further develop existing facilities. However, leadership and faculty inferred in FGDs for other UMPs that the lack of equipment and the maintenance of available facilities are major challenges to the advancement of CBME. Given that the HPET project has ended, this issue will likely become more urgent, as a Vice Rector at Thai Nguyen UMP commented: “In order to accomplish our mission of training medical staff for Northern mountainous provinces and to meet the changing needs of the labor market, the University has determined to innovate toward CBME. Nevertheless, we have met numerous difficulties, in human resources, such as the shortage of highly qualified faculty for the innovated curriculum, and in financial resources. The University must have sufficient counterpart funding in order to receive loans… it is very hard to provide proof of our capability of providing counterpart funding… in general, financial issues remain a dilemma how to have finance to sustain our developments.” 19 | IMPACT-MED ACTIVITY REVIEW USAID.GOV Figure 3: OSCE facilities allowing remote monitoring and assessment Photo credit: USAID Learns In terms of human resources, discussions with faculty and UMP leadership, and on several occasions with students, revealed that short staffing remains an issue at all UMPs. Short staffing frequently causes work overloads at different levels. Faculty members at all UMPs reported they had to undertake multiple tasks at the same time during Activity-specific staff training. Discussions with UMP leadership revealed that UMPs have had to expand their networks of invited ad-hoc lecturers and utilize the manpower of hospital staff as a solution to rectify the low faculty/student ratio. KII and FGD data illustrate that limited financial resources have been a persistent major challenge for public UMPs, both inside and outside the Activity. As leadership continually expressed at each of the five UMPs, more financial resources are required for building new facilities and upgrading and maintaining existing equipment. Some UMPs wish to seek funding from international organizations, but are unable to secure it due to a lack of counterpart funds. Resources for covering extra costs incurred due to small group teaching and faculty/student ratio improvements were also asked for from Activity stakeholders. As reiterated during discussions with leaders, future financial scenarios at UMPs in the coming years will be different relative to before since they have recently been granted Level 2 financial autonomy status. With their Level 2 financial status, UMPs will no longer receive a budget line from the state for running costs; in return, UMPs will receive more freedom in decision-making in academic affairs, personnel, and institutional organization. Discussions with leaders confirmed this is both a challenge and an opportunity for UMPs. Finally, leadership and faculty expressed in FGDs that having access to authentic learning materials, such as real-world clinical texts or multimedia accounts, to develop course syllabi for new subjects, such as ESP and Professionalism, were particularly helpful where applied. Policies and institutional autonomy: major obstacles to innovation The policy-related challenges that UMP faced in Phase 1 and will continue to face in Phase 2 are divided into two blocks: systemic legislative arrangements and institutional policies. System-wide, desk review, FGD, and KII data corroborate that known policy conflicts among state management agencies limit innovations. Inter-ministerial discord and management overlaps persist and problematize cooperation. Conflicts exist between the MOET, the MOH, the Ministry of Finance, and the Ministry of Home Affairs, and impact UMP autonomy in academic affairs, finance, personnel, and organization. Respondents expressed that policy conflicts, regardless of the prospective increased autonomy from the MOET, the MOH, or provincial entities, remained major obstacles to the applicability of educational innovations. As explained by a leader from HCMC UMP, the CBME-integrated curriculum entails “a new system with different arrangements,” such as the position of module lead, and UMPs require 20 | IMPACT-MED ACTIVITY REVIEW USAID.GOV a good level of freedom for making decisions on internal issues to create enablers for CBME and to be the most effective. Discussions with leadership revealed that another challenge to quality assurance efforts for CBME is the unfavorable systemic arrangements for local and international specialized accreditation. The existing accreditation system lacks a professional buffer body to recognize foreign accreditors. Specialized accreditation for medical education is not present because there is neither a specialized accreditor nor a set of standards for the assessment of medical institutions and programs. Institutionally, UMP managers shared with the research team that a major challenge to policy and management lies in the fact that the CBME curriculum exists in parallel with conventional programs, which requires a dual system of academic management. One example provided by both leadership and management respondents is that the rewards and compensation provided to faculty members for their contribution to CBME development have not been adequate, although faculty were quick to express that UMPs have improvised some support and allowance mechanisms for their efforts. UMP managers tended to see the issue of dual curricula as one that will be resolved in time as the expansion of CBME leads to the phasing out of traditional programs. Several of the UMP leadership respondents indicated that limited institutional autonomy has always been a major challenge to UMPs’ innovation processes and developments more broadly, including the rollout of CBME.7 One obstacle shared by most UMP leaders is that UMPs have limited power to set up a new functional unit, so they had to ask for approval to establish the Staff Development Unit, which is a critical enabler for CBME. In addition, since Hue UMP and Thai Nguyen UMP are members of regional universities, they will have to go through an additional layer of management and bureaucracy from their umbrella universities. Training/Curriculum organization UMP leaders and managers across the five UMPs pointed to low faculty/student ratios as a significant concern. The average ratio at the five Activity UMPs is estimated at around 1:12; at leading UMPs, such as HCMC UMP and Hue UMP, the ratio is higher, at 1:10, compared with the 1:15 ratio specified by the MOET in Circular 03/2022/TT-BGDĐT. Since CBME pedagogical practices require small group teaching and closer supervision, a ratio of between 1:8 and 1:6 would be more effective to develop CBME. Many of these respondents saw low faculty/student ratios as the biggest challenge threatening the effectiveness and long-term development of CBME. The organization of the CBME curriculum by module also posed difficulties to UMPs, since current management tools do not support this approach. The joint delivery of classes by lecturers of different subjects was said to have been an issue at the early stages of the Activity, but this has been resolved with time as faculty became familiar with the new practice. CONCLUSIONS Throughout the implementation of CBME at five UMPs in IMPACT-MED’s Phase 1, the challenges reported by UMP leaders and managers varied across several domains. Differences were underscored due to the 7 The research team learned that even though there is now more inertia to implement Decree 99/2019/NĐ-CP on the Revised Law on Higher Education, which could serve as a potential enabler of more empowered UMPs, UMPs still face considerable challenges to renovate their programs while being subject to varied state interests. A sample of UMP leaders said that the lack of an appropriate legal corridor for hospital-university partnerships and practical guidelines on the implementation of Decree 111/2017/NĐ-CP unnecessarily complicated hospital-UMP partnerships. Leaders from three UMPs - HCMC UMP, Thai Nguyen UMP, and Hue UMP - directly mentioned this issue. Thai Binh UMP’s leadership did not perceive the lack of guidance documents under Decree 111 to be a pressing problem, and no opinion on this issue was provided during a KII with a leader at Hai Phong UMP. 21 | IMPACT-MED ACTIVITY REVIEW USAID.GOV geographical locations where UMPs are based, such as economic centers versus provinces; the type of HEI, such as whether the UMPs were members of regional universities; and financial autonomy status, as seen in the UMP profiles in Annex II. Some of the challenges have been resolved over time, such as the resistance to change at the beginning from faculty. When facing key challenges that may impact motivation and effectiveness, such as a strenuous work culture, UMPs have taken different approaches and strategies based on the local context and availability of resources. The level of proactiveness of the leaders and managers to tackle different issues influenced how each UMP overcame initial start-up challenges. The magnitude of the challenges in the coming phase will vary across UMPs. Continued challenges for all UMPs inside and outside the development assistance programing exist in three major domains: (1) scarce resources for CBME, including knowledge, physical facilities, human and financial resources; (2) unharmonized policies and institutional autonomy; and (3) difficulties in curriculum organization based on the maintenance of traditional teaching. Institutional autonomy appears to be an important sticking point, especially when UMPs will have to experiment with their new status, as some will gain more autonomy than others. RESEARCH QUESTION 1D IN WHAT WAYS WERE STUDENTS AND FACULTY ENGAGED/INVOLVED DURING THE ACTIVITY AND HOW DID THE ACTIVITY EMPOWER THEM IN REFORMING THE CBME CURRICULUM? FINDINGS Faculty-wise: as observers, implementers and sharers, decision-makers, change agents, and leaders FGD and KII data from across our stakeholder groups indicate that the capacity of faculty members and management staff at UMPs has been developed gradually over the past five or more years. Faculty reported that they had benefitted initially as observers of CBME following HMS and their colleagues at the beginning of the program, and then by directly delivering active learning pedagogies and assessments as implementers of CBME. Later, they were able to share their CBME knowledge, localized experiences, and lessons learned with their individual UMPs and other Activity UMPs. During the mini-validation sessions at UMPs, the research team ascertained that over the course of CBME implementation, several active faculty members have been decision-makers on the localization of the CBME curricula and syllabi that fit the context of their UMP and their learners.8 Further, while facing huge obstacles, doubts, and resistance during the lengthy innovation process, several faculty members across the IMPACT-MED system have become active agents, leading the change.9 The improved capacity of faculty has contributed to the development of 8 Discussions on the Mentor-Tutor-Tutee program at HCMC UMP revealed that several lead faculty members also worked as mentors. In other UMPs, the student support program may take different forms and faculty members may take different roles. As explained by faculty members across the UMPs, the introduction of the new subject of Professionalism in the CBME program as part of the renovations brought by IMPACT-MED has helped lecturers to learn how to be professional and how to act as role models for their students. 9 Specifically, in terms of CBME curriculum development, discussions at UMPs conferred that faculty are now capable of developing competency standards with SLOs, concept maps, modules, syllabi, learning materials, MCQs, and other assessment methods. To deliver the CBME curricula, faculty showed that they were empowered to instigate change by adopting various new teaching methods such as flipped classrooms, clicker questions, and group discussions. Under the renovated program, faculty expressed that they are now capable of implementing various competency￾based assessment methods like OSCE and mini-CEXs, thanks to the technical assistance provided by HAIVN. 22 | IMPACT-MED ACTIVITY REVIEW USAID.GOV institutional capacity. As a Rector of a UMP said, “the UMP is now aware of where to go, what to do, and how to reach the destination. If we wish to develop CBME curricula in other disciplines, we know where we should start.” Student-wise: as active participants, peer-peer leaders, and decision makers During the FGDs, students at both undergraduate and graduate levels highlighted a more active role in their learning process relative to before the Activity. For example, in flipped classrooms, which respondents at all five UMPs said were commonplace, students have to prepare for class by searching and gathering materials and literature, studying theoretical knowledge, and taking pre-tests or readiness assessments. Faculty at each of the UMPs confirmed during discussions that these activities promoted both independent learning and teamwork and helped to build up students’ capacity, especially for self￾directed learning. Several students mentioned in KIIs that because they were aware of “where to go” in the learning process, they were increasingly motivated to learn and they participated more actively. The respondents highlighted that the student feedback system, which collects students’ opinions and feedback on various issues at different stages of courses, requires extensive participation. Feedback collection is administered in numerous formats, including online and paper-based surveys, face-to-face class manager meetings, and face-to-face Student-Rector dialogues. The research team learned that, apart from academic activities, a number of students also participated in the student support activities as student tutors, which improved their soft skills and leadership abilities. An example of the remarkable shift in student engagement is reflected in students’ participation in a HCMC UMP student support service called the Mentor-Tutor-Tutee program (see Box 2 below). 23 | IMPACT-MED ACTIVITY REVIEW USAID.GOV BOX 2: STUDENT SUPPORT PROGRAM IN HCMC UMP The student support program in HCMC UMP is a comprehensive program covering many aspects of support, including: ● Enrollment counseling and advice, ● Orientation for newly-enrolled students and orientation for the new academic year, ● Management of students living on and off the campus, ● Youth Union and Association Activities, ● Personal development support and counseling, ● Support to student groups who have specific needs such as adults, those with young children; ethnic minority students; lesbian, gay, bisexual, and transgender students, etc., ● Mental health and health counseling and support, and ● Support for the victims of sexual harassment. One prominent activity identified by respondents as particularly helpful for students is the Mentor￾Tutor-Tutee program, in which faculty members take on the role of a mentor to counsel students in academic activities, especially for Year 1 and Year 2 students. Tutors and tutees are respectively played by senior and junior students. The tutor student provides advice and support to their fellows on both academic and non-academic issues. In spite of the fact that a senior student supporting a junior student is a tradition among students in many UMPs, the Mentor-Tutor-Tutee model is a formal and systemic program at HCMC UMP. Stakeholders believed that this useful student support activity ought to be further developed and promoted under IMPACT-MED. These data suggest some limitations on the availability, awareness, and utilization of student support services related to CBME more broadly. When the research team asked HCMC UMP students about their Mentor-Tutor-Tutee program, student respondents reported that the program sounded meaningful and of great importance, and that they wished to participate, but many were not aware of the program’s existence. During KIIs with UMP leaders and support staff outside of HCMC, the research team learned that programs to offer CBME-relevant student support are still being developed at the other four UMPs. Students with learning disabilities who participated in our FGDs tended to indicate that existing student support services do not adequately meet their needs under CBME. Although student respondents at all five UMPs reported increased learning engagement, it should be noted that these data show differences in the roles and levels of participation of students across UMPs. HCMC UMP students appeared to the research team to be the most active in their learning, while those at Hue UMP and Thai Nguyen UMP suggested that their levels of participation were significantly lower than others in the IMPACT-MED system.10 As stated by UMP leaders during FGDs, especially at Hue UMP, region￾10 As the leading UMP in the development of CBME in Vietnam, HCMC UMP is implementing two CBME programs at both the undergraduate and postgraduate levels; one is the six-year Doctor of Medicine, and the other is the three-year Surgical Residency. The first cohort of CBME Doctor of Medicine students are now in Year 6 and are expected to graduate imminently. HCMC UMP also expects to have the first graduates of the CBME Surgical Residency program later in 2022. The remaining four UMPs have run the CBME Doctor of Medicine program for 4 years and the developments for Year 5 and Year 6 are underway. 24 | IMPACT-MED ACTIVITY REVIEW USAID.GOV specific characteristics of students were recognized by educators as an important factor. Lower levels of socio-economic development and other region-specific characteristics may play a role in explaining differences in student participation. CBME students in HCMC UMP and some in Hai Phong UMP have been organizing study-related student activities. Students also expressed an expectation to have support from UMPs to develop their capacity in project management, planning, and other soft skills, such as time management. However, KII and FGD data suggests that the role of students and the level of student proactiveness and participation in learning varies within each UMP and across UMPs. Students at HCMC UMP were presented by respondents as the most active and are pioneering in many activities for students, while those at Hue UMP tend to be reserved and a lot less active as identified by UMP leaders. Hue UMP students corroborated that CBME students may be less active for the reasons expressed above. Student respondents at most UMPs commented that their learning and soft skills had improved as a result of the CBME programs. Faculty and doctor supervisors consulted at hospitals by the research team expressed that CBME students had better generic and clinical skills and showed greater confidence in working with patients, compared with students of traditional programs. As a hospital representative conferred: “They are proactive, in foreign language, searching for learning materials, only with suggestions from doctors then they can go and find the materials by themselves. Year 3, Year 4 CBME students must be like Year 5, Year 6 of the old program. Their proactiveness is high; they have good approach to work, able to meet the job requirements at hospitals straight away, and do not need to go through re-training as the graduates of the old programs often do…” However, UMP managers and faculty stressed that a formal assessment of CBME students’ competencies was necessary. CONCLUSIONS Faculty across the five UMPs have acquired a new skill set that is essential to the development and implementation of CBME thanks to the training provided by the Activity, such as opportunities to gain hands-on experiences in CBME curriculum development and extensive practice during the implementation of the CBME programs. This is a significant advancement in the career development of most faculty members, given that many of them have not received formal training in medical education pedagogies and curriculum development, which is needed to become university lecturers. Faculty have been on a journey as part of the innovation process and some are now positioned to lead the change themselves. Faculty have begun factoring student satisfaction levels into their thinking. Students’ engagement in their studies and in the wider learning process has increased due to the Activity’s pedagogical interventions and strengthening of UMP student support services to improve student camaraderie via peer-to-peer learning. Improvements in faculty, institutional, and student capacity reflect the richness and depth of CBME programing. Among the Activity’s UMPs, HCMC UMP has led the development of CBME in terms of progression of reforms, richness of CBME elements, and level of depth and performance. Although HAIVN and the UMPs have not yet proctored any formal, dedicated assessment of students’ competency, knowledge, and technical expertise, the qualitative judgments by students, faculty, and employers shows that CBME students possess remarkably better soft and clinical skills than students of traditional programs. 25 | IMPACT-MED ACTIVITY REVIEW USAID.GOV RESEARCH THEME 2: STAKEHOLDER ENGAGEMENT AND PARTNERSHIP IN CBME IMPLEMENTATION RESEARCH QUESTION 2A WHAT WERE THE OPPORTUNITIES, CHALLENGES, AND LESSONS LEARNED THAT EMERGED FROM ALIGNING PRIORITIES AND ACTIVITIES BETWEEN DIFFERENT STAKEHOLDERS AND/OR COMPLEMENTARY PROJECTS? Partnerships in the health sector can serve a variety of functions. These partnerships are often new arrangements that may give multiple partners the chance to contribute toward the same objective. Developing technology for tropical diseases; advancing surveillance and screening methods; contributing to technical aspects of sustainable medication development, such as CBME; and implementing vector control are only a few examples of past partnerships that have favorably impacted health outcomes.11 FINDINGS Opportunities and challenges The five UMPs taking part in the IMPACT-MED Activity constitute a diverse range of UMPs, as some were members of regional universities and local universities, and others were considered “key” universities. Each UMP had its own opportunities and challenges. UMP actors said during FGDs and KIIs that the UMPs face numerous challenges in aligning priorities and activities between different stakeholders. Some of the challenges are inherent to the institutional characteristics of the UMPs, such as an HEI’s type, size, location, and governance structure, as discussed above in Research Question 1C. FGDs with university leadership across UMPs revealed that “key” universities, like HCMC UMP and members of regional universities, are more prone to accessing development projects and to involve multiple national and international stakeholders. FGD and KII data suggest that while UMPs in the provinces have developed good partnerships with local and foreign partners, most partnerships are limited to research collaborations and knowledge transfers and are not curriculum-centric. For UMPs in rural regions, including Hue and Thai Nguyen, opportunities have presented themselves to connect more easily with local development partners and international corporations who happen to have projects or factories or based in their vicinity. Conversely, small, non￾key UMPs located in provinces, such as Hai Phong and Thai Binh UMP, are less frequently prioritized for development investments and have fewer chances to receive funding from international corporations. Further, while key UMPs located in big cities generally have better access to partnerships, KII data suggest they face intense competition from other UMPs to establish partnerships and bring in resources from a limited number of potential partners. Being a member of a regional university system may also present challenges for UMPs, according to some UMP respondents. For example, Hue UMP must utilize tuition provision of other member institutions in Hue University, which restricts their opportunities to leverage external resources brought to them by IMPACT-MED. More specifically, Hue UMP was not able to make use of Shorelight’s support in developing English language programs since the English courses in its programs were delivered by the Hue College of 11 Nishtar, S. Public – private 'partnerships' in health – a global call to action. Health Res Policy Sys 2, 5 (2004). 26 | IMPACT-MED ACTIVITY REVIEW USAID.GOV Foreign Languages. This issue was recognized by Hue UMP’s leaders.12 Conversely, Thai Nguyen UMP managed to make full use of Shorelight support to develop its ESP curriculum (see Box 3). BOX 3: ESP PROGRAM AT THAI NGUYEN UMP The ESP program in Thai Nguyen UMP is one of the highlights of the IMPACT-MED Activity. ESP courses have been integrated with medical majors and modules within the CBME Doctor of Medicine program. Every syllabus and lesson of ESP courses has been developed with the consultation of the faculties who are developing and implementing relevant medical majors and modules. Therefore, the knowledge and skills students are expected to gain in ESP courses are mapped according to the Doctor of Medicine Program’s knowledge and skills. Pedagogical approaches and methods are student-centered and include project-based learning and case-based learning. There are a variety of learning activities. Teachers bring discussion cases to class in which students have to practice English to analyze and diagnose diseases that they could encounter during their shifts at hospitals and clinics. Students are also given opportunities to role-play as a doctor, a patient, or a patient’s relative. Students learn how to keep diaries, describe a shift, report a case, ask about the medical history of a patient, communicate with patients, and more. All of these essential skills are taught in English so that students will be able to communicate and work in an English-speaking environment after graduating. Assessment activities are formative and integrated with listening, speaking, reading, and writing skills. The ESP Department at Thai Nguyen UMP received continuous and significant support from Shorelight Education, a private sector partner of the IMPACT￾MED Activity in the development of its ESP program. More details are provided in Annex VIII. UMP managers emphasized during KIIs that the partnerships between UMPs and hospitals play a critical role within medical education. Respondents at UMPs and hospitals shared that the relationship has been traditionally close thanks to well-maintained connections among UMPs’ faculty and hospitals, owing to the fact that many faculty members also work as doctors at local hospitals. KII data infer that hospitals have actively participated in the medical training process by providing work placements for medical students' clinical practice, and limitedly in developing SLOs and defining the competencies of medical professions. FGD and KII data from discussions with UMP faculty and leadership show that the challenges UMPs face in developing partnerships with hospitals differ. In big cities, the competition for hospital placement has been growing due to competition among UMPs. Moreover, patients’ expectations, choices, and demand in urban areas have increased, along with a greater recognition of the right of a patient to refuse treatment by a student. These factors pose difficulties for big UMPs like HCMC UMP. UMPs located in provinces, on the other hand, have little competition for hospital placement; however, the district-level hospitals and clinics are often in far-off regions, where transportation is costly and not convenient. The capacity of medical staff at hospitals and clinics at these levels is also lower than in larger healthcare institutions. The GVN promulgated Decree 111/2017/NĐ-CP to regulate UMP-hospital relationships. Leaders and managers from four of the five UMPs reported challenges in implementing the decree. They mentioned that more practical guidelines were needed, given that the healthcare sector is getting more complicated. Several UMP leaders believed the amount paid to hospitals to accommodate student placements may be 12 Hue UMP’s leaders expressed that they knew how to technically resolve the issue in the future and take advantage of this opportunity, without specifying how to the research team. 27 | IMPACT-MED ACTIVITY REVIEW USAID.GOV insufficient to fully compensate for the costs that hospitals cover during placement. UMPs also said they needed greater autonomy in financial management to flexibly cope with new challenges. Apart from HCMC UMP, which effectively utilized resources provided by foreign partners and parallel projects to develop physical facilities for the implementation of CBME, the involvement of external stakeholders in CBME implementation at the other four UMPs was limited. These four UMPs relied heavily on HPET, as discussed under Research Theme 1, and partners from IMPACT-MED. These challenges are anticipated to multiply after HPET’s end. Further discussions on how to tackle these challenges and overcome these difficulties are presented in the next section, Research Theme 3. Lessons Learned Interviews with UMP leaders and managers revealed several lessons for better leveraging resources from external stakeholders and strengthening UMP-hospital partnerships. The usefulness of empowering mid￾level managers to advance partnerships was a key reflection that emerged from the discussions with HCMC UMP leaders and managers. The managers were given autonomy and trust to develop partnerships with relevant stakeholders and to make strategic planning in relation to CBME implementation. With the greater autonomy, they have demonstrated a high level of proactiveness in seeking partners and strategic planning. In regard to the UMP-hospital relationships, according to hospital representatives, partnerships with UMPs bring multiple benefits to hospitals and their doctors, which helps to consolidate the relationship. First, partnerships give hospitals access to high-quality expertise, namely professors and researchers. Hospitals may also gain access to the high-tech equipment available at UMPs for medical examinations, tests, and diagnoses. Importantly, hospitals receive students to share the heavy workload of their medical professionals. For hospital-based doctors, they receive training on CBME, pedagogies, and assessments, which helps them develop their teaching capacity. They are also exposed to updated professional knowledge, gain the signifier of being a university-invited lecturer, and receive payment from UMPs. For UMPs, hospitals most significantly provide a space for clinical practice and authentic assessments, which are critical to developing student competencies. Local and international collaborations at some UMPs demonstrated that some governmental development funds are available, although they are not equally accessible for smaller UMPs. Some UMPs have been active in seeking funding from different sources and developing collaborative projects with foreign partners to attract investments into physical facilities for CBME. One common strategy that the research team observed at the majority of the UMPs was the development and utilization of alumni networks to further partnerships with external stakeholders. Thai Binh and Hue UMP were able to develop local partnerships and HCMC UMP was able to develop international partnerships. CONCLUSIONS The different institutional context of each UMP impacted how the UMPs aligned their priorities with those of other stakeholders. Numerous challenges remain in leveraging UMPs’ resources for the implementation of CBMED, despite the opportunities from close connections with local and foreign enterprises based in their provinces and the strong partnerships with provincial hospitals. The most prominent challenge that needs resolving is limited autonomy due to policy restrictions and governance structure of regional universities. 28 | IMPACT-MED ACTIVITY REVIEW USAID.GOV Several lessons have been drawn from UMP experiences and achievements in developing UMP-hospital partnerships. First, mutual benefits are fundamental. Second, strategic planning, autonomy, and empowering mid-level managers are important. Additionally, small UMPs can also reach out to development funds provided by other governments instead of relying on the GVN. Alumni networks are also important in maintaining and developing relationships with hospitals. Further discussions about how to better align the priorities of different parties will be provided in the Recommendations section. RESEARCH QUESTION 2B WHAT WORKED WELL WITH PRIVATE SECTOR INVOLVEMENT TO ADVANCE CBME, AND WHERE IS THERE ROOM FOR IMPROVEMENT IN THE FUTURE TO ENHANCE THE BENEFIT OF PRIVATE SECTOR INVOLVEMENT? FINDINGS FGD and KII data illustrate that there are only a few private sector partners participating in the implementation of CBME at UMPs. Some UMP leaders admitted that they had not prioritized developing private sector partnerships, such as with local pharmaceutical companies, local private hospitals and clinics, and international healthcare companies. Their level of proactiveness in reaching out to local enterprises and transnational corporations to engage them in CBME implementation is consequently low. Another reason cited was the lack of information about one another’s visions, interests, needs and priorities. As a UMP leader said: “…we don't know about private enterprises; and they do not know about us…”. The lack of information leads to low transparency and low trust, which is considered the foundation for the relationship between UMPs and private sector partners. From the perspective of private sector partners, some enterprise representatives said that the reason for their limited involvement in CBME implementation was a lack of motivation to join UMPs in the training process. There have not been policies implemented to incentivize them to invest in higher education. The discussions with some UMPs where a private partnership has been successful revealed some factors that enabled the partnership. A Thai Binh UMP leader shared during a discussion that local pharmaceutical companies provided potential, mainly untapped, opportunities for their staff to realize their scientific initiatives and later launch their products on the Vietnamese market. Further, according to a representative of Novartis International AG, if international healthcare and pharmaceutical corporations were better acquainted with the working agenda of UMPs, they would more actively engage with UMPs to support disadvantaged students and provide research grants. Developing partnerships with local private clinics and hospitals presents students with opportunities and varied settings for clinical placement, such as interacting with a variety of patients, including international visitors. In some cases where partnerships with local private sector partners have been successful, long-term relationships between UMPs and their alumni were key. According to private sector representatives, mutual trust is a prerequisite for developing UMP-private sector partnerships. Their interest in setting up partnerships with UMPs and providing support to them was the result of their awareness and understanding of what the UMP was doing (i.e., developing CBME) to benefit learners and to improve medical staff capacity. One private sector respondent in a successful case described the partnership as “a win for patients, a win for the healthcare system, and a win is for us…UMPs help us to realize our vision of re-imagining medicine.” 29 | IMPACT-MED ACTIVITY REVIEW USAID.GOV CONCLUSIONS Considerable distance remains between UMPs and healthcare industry partners. Generally, the participation of private sector partners in the development and implementation of CBME and curriculum reform has been limited. There are few channels where UMPs and enterprises can meet and share their missions, visions and interests to one another. This is largely because developing relationships with private sector partners has not been among the UMPs’ main priorities. This may be partially a reflection of the culture of public institutions in Vietnam more broadly. Private sector partners also lack incentives to join UMPs in the training process, especially when the working agenda of each UMP is not transparent. Inadequate involvement of the private sector equates to a loss of opportunities for faculty, students, and the applicability of CBME. This is especially the case with international corporations who are willing to collaborate as part of their corporate social responsibility initiatives. UMPs have relied on their alumni as connectors to local and international enterprises and have demonstrated that mutual trust is a key enabler of UMP-private sector partnerships. When partnerships do succeed, capacity increases and mutual gains for UMPs and the private sector are apparent. RESEARCH THEME 3: SUSTAINABILITY A key part of this participatory research was to establish a definition of sustainability from the perspective of grassroot UMP stakeholders during an initial consultation. The user-owned definition of sustainability is presented above in Box 4 for reference. RESEARCH QUESTION 3A WHAT DEVELOPMENTS, FOR EXAMPLE, IN TERMS OF SUPPORT FOR INSTITUTIONAL CAPACITY, IQA, OR UMP POLICY, COULD HELP TO ENSURE THE SUSTAINABILITY OF CBME? FINDINGS The sustainability of CBME after Phase 1 of the IMPACT-MED Activity can be roughly assessed against the dimensions of sustainability co-defined by UMPs and stakeholders, including the existence of an IQA system and the establishment of foundational elements, such as essential quality assurance processes in BOX 4: SUSTAINABILITY AS CO-DEFINED BY STAKEHOLDERS DURING THE INITIAL CONSULTATION MEETING According to the participants from UMPs, private sector partners, and international organizations at the initial consultation meeting, the sustainability of CBME can be assessed in the following dimensions: • An IQA system in place: quality assurance processes in curriculum development, evaluation, and revision; a stakeholder feedback system; quality assurance in testing, assessment, and program accreditation, etc. • Resources for CBME advancement: human resources and staff professional development, physical facilities, information technology infrastructure, and finance. • Policy and strategies: national and institutional policies and strategies, governance, and autonomy. • Leadership and stakeholders’ consensus and commitment to CBME. • Replicability of CBME inside IMPACT-MED UMPs and other UMPs in the system. 30 | IMPACT-MED ACTIVITY REVIEW USAID.GOV curriculum review and revision, testing and assessment, student feedback surveys, and program accreditation. The Sustainability of CBME at Activity’s UMPs Based upon the positive results and momentum gained, UMP faculty and managers across all five IMPACT￾MED UMPs mentioned that the curriculum innovation was irreversible during discussions. UMP leadership shared that four UMPs, all except for HCMC UMP, are required to continue developing and implementing the Year 5 and Year 6 curriculum for the six-year Doctor of Medicine. The completion of this segment is critical to assure the consistency, systematic design, and overall effectiveness of CBME within the first cohort of IMPACT-MED UMPs. With the knowledge and lessons learned from implementing the first CBME program within specific programs, the UMPs plan to expand CBME to other programs of different majors at the undergraduate and postgraduate levels, depending on their priorities and strategies. Such an expansion and application of IMPACT-MED’s technical assistance elsewhere requires a wide range of activities to sustain future developments. With regard to CBME technicalities, faculty in FGDs expressed that ongoing professional development is essential in galvanizing clinical pedagogical skills and assessment methods such as MCQ, OSCE, EPA, and mini-CEX. Some faculty members believe that such supportive subjects like Professionalism and ESP likely require further development, in terms of content and teaching methods. HCMC UMP is preparing its application for international specialized program accreditation under different bodies, including the World Federation for Medical Education, Accreditation Commission for Education in Nursing, and Commission on Collegiate Nursing Education. Other UMPs expressed their intention to follow HCMC UMP; however, what to share is a question. HCMC UMP leadership stressed that knowledge and experience could be shared, but not the “products”, for example, a completed CBME curriculum or a test item bank, since other UMPs ought to develop their own research capacity in medical education among their faculty. In discussions with the research team, Thai Nguyen UMP leaders expressed their interest in developing community engagement as a part of their operations. . Evidently, this would prove helpful to connecting with local communities in the healthcare setting and to ensure UMPs move to work more closely with private sector partners. To assure the quality and effectiveness of these activities, stakeholder feedback at our participatory events suggested that continual support to UMPs in developing professional technical expertise and technological solutions is critical. BOX 5: SUSTAINABILITY PRELIMINARY ASSESSMENT With regard to the resources for CBME advancement, although critical resources, such as human resources and CBME knowledge, have been strengthened, there are still lots of challenges, specifically in relation to physical facilities, such as simulation labs, OSCE centers, and information technology infrastructure for CBME. Financial challenges remain paramount. The legal landscape has shown a promising future with the establishment of the National Medical Council, the promulgation of competency standards of some medical professions, and the greater autonomy enjoyed by UMPs. Leadership commitment and stakeholder consensus have been strengthened and will no longer be a challenge to IMPACT-MED Activity UMPs, as compared to the beginning of Phase 1. Finally, regarding the replicability of CBME, there is significant potential to scale up CBME within IMPACT￾MED Activity UMPs and other UMPs in the system. The biggest threat to the system-wide sustainability is the low faculty/student ratio. The first stepping stones for CBME advancement have been laid. However, for CBME to continue, there is a lot ahead for the IMPACT-MED Alliance and UMPs to accomplish. 31 | IMPACT-MED ACTIVITY REVIEW USAID.GOV For IQA and continuous quality improvement (CQI) development, discussions with UMPs inferred that constant curriculum revisions for CQI would be continued according to the processes already in place. Importantly, stakeholders across the board recognized that a comprehensive review of the entire six-year undergraduate and three-year residency program, starting at HCMC UMP, would be beneficial. UMP leadership explained that several challenges remain, including scarce access to financial resources, as discussed under Research Question 1B. CONCLUSIONS With the knowledge and lessons learned from implementing the first CBME program, UMPs will expand CBME to other programs of different majors at the undergraduate and postgraduate levels, depending on their individual priorities and strategies. Current expansion plans appear to be carefully tailored to each context and reflect great promise for the future. Leadership commitment and stakeholder consensus have been strengthened and will no longer be a challenge to IMPACT-MED Activity UMPs, as compared to the beginning of Phase 1. The further multiplication of CBME requires UMPs to strengthen and evolve technicalities, including curriculum development and implementation, staff development, CQI, and physical facilities that enable CBME. Having improved autonomy appears to generally promise UMPs more independent decision-making and access to resources. Other inhibitors remain in the systemic governance arrangements and will be discussed in the section below. RESEARCH QUESTION 3B WHAT ACTIONS SHOULD BE TAKEN TO SAFEGUARD THE SUSTAINABILITY OF CBME IN VIETNAM AND HOW SHOULD ANY BARRIERS OR ENABLERS PRESENTED BY NATIONAL POLICIES BE NAVIGATED? FINDINGS The Sustainability of CBME Across the Local Medical Education System It is widely agreed among UMP leaders and managers, as well as other stakeholders, that the sustainability of CBME depends on whether and how successfully it can be scaled up to other UMPs in the local medical education system. FGD and KII data from discussions with UMPs’ leaders and managers reveal a number of facilitators that would help successfully replicate CBME. First, it is important that the knowledge and expertise developed in Phase 1 and the experiences of localizing CBME is shared with other UMPs. UMPs leaders realized the importance of knowledge-sharing. Regarding this issue, a leader at HCMC UMP shared, “…for the sake of the development of the entire medical education system, we are willing to. Local experts on CBME curriculum development are not sufficient. If only we adopt CBME, then it is not possible to bring about systemic changes. We need to support other UMPs with our experience, so that the entire system can be changed, too.” Second, to encourage other UMPs to adopt CBME, UMP leaders said that the development of a legal framework and a policy and professional environment to foster CBME is critical. While the establishment of the National Medical Council and the MOH’s establishment of competency standards for some medical professions were recognized by leaders as enablers of CBME, other legislative issues remain. Specific issues raised included the presence of two parallel management systems, one for CBME implementation and the other for the traditional curriculum; a lack of full autonomy for UMPs; the absence of a legal framework for specialized accreditation; and the low faculty/student ratio. UMP leadership explained that the potential 32 | IMPACT-MED ACTIVITY REVIEW USAID.GOV for greater UMP autonomy is complex and may be enabling or hindering depending on the individual UMP’s context. Further, regional universities could face limited autonomy as they are overseen by local umbrella institutions. Several UMP actors pointed out that low faculty/student ratios are a major issue that needs to be addressed first, and that efforts by UMPs alone may not be sufficient. During the Learning and Utilization Event, HAIVN took the opportunity to make it clear to UMP stakeholders that while the low faculty/student ratio was indeed a potential barrier inherent in Vietnamese medical education, an important enabler of CBME sustainability would be for Vietnamese institutions to develop a wider collegiate culture of CBME, as IMPACT-MED has begun to instigate. Annex VII presents national policy barriers, enablers, and hypothetical policy development recommendations. CONCLUSIONS IMPACT-MED Activity’s UMPs recognize the importance of expanding CBME to other UMPs in the local medical education system and are willing to share their experience and knowledge gained with others, especially for the larger goal of advancing Vietnam’s health security. Other actions to safeguard CBME’s sustainability and improve the likelihood of scaling up CBME beyond IMPACT-MED include widening the scope of Activity interventions; developing a more favorable legal environment for CBME, including by realizing increased autonomy and a framework for accreditation; and, critically, improving faculty/student ratios. However, these facets are outside of USAID’s sphere of control, unless there is a shift in the culture of medical education in Vietnam with respect to the renovations IMPACT-MED has brought. 33 | IMPACT-MED ACTIVITY REVIEW USAID.GOV RECOMMENDATIONS ADVANCEMENT AND SUSTAINABILITY OF CBME ADVANCEMENT Future developments are critical to the Activity’s UMPs: As far as the sustainability of CBME in the Activity’s UMPs is concerned, there are several developments that are critical to the future advancement of CBME programs in Vietnam. UMPs should focus on the development and implementation of the CBME six-year Doctor of Medicine curriculum at Hue, Thai Nguyen, Hai Phong, and Thai Binh UMPs until the program is completed. When finished, if possible, there should be a comprehensive assessment of the entire six-year curriculum. For HCMC UMP, it is important to conduct an overall assessment of the program to draw practical lessons and to consolidate the theoretical and localized knowledge of CBME. UMPs should embark on the development of new CBME programs on the basis of the local results of current CBME programs. New CBME programs can be either at the undergraduate, postgraduate masters, or residency program level, depending on the UMP’s resources and priorities. In order to achieve successes in the future development of CBME, UMPs need to continue to bring fence resources for professional technical aspects, including clinical pedagogical skills; assessment methods such as MCQ, OSCE, EPA, and mini-CEX; and the further development of such subjects as Professionalism and ESP. Specialized program accreditation by foreign accreditors and the development of research capacity in medical education also are important priorities for UMPs to be able to advance CBME. It is also advisable that UMPs develop formalized community healthcare service programs to serve commune and district health stations as an operational domain and as part of their development strategies. IQA-CQI is another factor that affects the sustainability of CBME programs at IMPACT-MED UMPs. In addition to the constant yearly curriculum revision and comprehensive reviews of current CBME programs, UMPs should conduct assessments of CBME graduate competency to support evidence-based decision-making in curriculum innovation. Additionally, current quality assurance tools, including stakeholder surveys and student feedback surveys, should be reviewed and improved. Specifically, the survey instruments should be assessed for reliability and validated to assure that the survey results reflect the opinions of stakeholders. Feedback channels should be diversified to allow key stakeholders like students and employers to express their views and comments more often and in a variety of ways. Two key enablers of CBME, the development of student support programs and staff professional development, should be prioritized, especially at those UMPs which have not set up such systems. Student services should be expanded upon and customized to better fit CBME students’ problems, such as difficulties in new learning methods, approaches for students with learning difficulties, and ESP reading skills. The services should also be more personalized and follow the mentor-tutor-tutee model developed at the HCMC UMP. Similarly, a staff professional development unit should be inaugurated to deliver continuous technical training to all full-time faculty members teaching within CBME programs and invited lecturers. Last, but not least, UMPs should enhance internal communication activities on CBME and curriculum innovation. It is likely that both faculty and students will be more supportive and engaged in the innovation process if they are better informed on the core values of competency-based education, national competency standards, program SLOs, and UMPs’ action plans. By developing a more nuanced understanding of the changes taking place around them, students will become more self-directed in learning and leading the changes. Thorough and clear communication will also help reduce the chances of misunderstandings between students, faculty, and UMP leadership concerning the purpose of additional 34 | IMPACT-MED ACTIVITY REVIEW USAID.GOV work or effort required as a result of renovations. Apart from those developments of critical importance, there are several recommendations worthy of the IMPACT-MED Alliance’s consideration presented in Box 6. SUSTAINABILITY Actions to safeguard the sustainability of CBME in Vietnam: The ability to expand CBME to other UMPs in the system is critical to the long-term development of CBME. To facilitate the scale-up and success of CBME adoption, the CBME knowledge and technical know-how of CBME curriculum development that were developed in Phase 1 should be compiled systematically and shared with new UMPs who join the “CBME community.” The sharing channels can be flexibly organized through an e￾library or a database that is publicly available and/or distributed among the network. Since the localization of CBME models and practices have shown positive results, UMPs’ practical experiences are of great value to the new UMPs joining the Activity. The sharing of technical and operational lessons can be administered through the development of a national network of UMPs. CBME faculty and students should be connected in organized networks, operating as structured online forums on social media and in-person workshops and conferences. It is also recommended that the local CBME community become better connected to the community in the Association of Southeast Asian Nations region, Europe, and America to receive updates of new CBME developments. This will also help open up opportunities for collaborations on medical education research. An important issue that needs to be resolved is low faculty/student ratios, which causes excessive workloads for faculty members. This issue can be tackled at different levels and will require a cultural shift based upon the actual roles of and relationship between faculty members and students, and the overall expectations of medical professionals in Vietnam to enforce competency. However, to begin, efforts should be made to influence state policy. Specifically, the MOET’s regulations on faculty/student ratio should be decreased gradually, from the current 1:15 ratio, to around 1:10 and then higher to enable CBME in some years to come. Such a change will have initial knock-on effects, but within UMPs, this would help to expand partnerships with hospital doctors, and boosting training and certification to invited lecturers would help reduce the teaching workload of permanent faculty. In the long run, UMPs may exercise the autonomous status granted to them to increase their staff size, tuition fees, and so on to improve faculty/student ratios. Further, UMPs should also apply information technology in teaching and BOX 6: SUPPLEMENTARY ACTIONS To enhance teaching quality and effectiveness in CBME, some class observation tools, such as COPUS, PORTAAL and DART, could be tried in CBME programs. Accreditation-wise, UMPs should set up a technical working group on international specialized accreditation to examine the procedures, assessment standards, and costs, so that they can be well-prepared in advance to assess feasibility and inform decision-making. It is also advisable for UMPs to take actions to lobby for specialized accreditation, which could involve setting up a local specialized accreditor for medical education and developing standards for assessing medical programs and institutions. Finally, a map or a checklist of CBME components in Vietnam should be developed, so that all UMPs in the higher education system have a point of reference when they embark on developing a CBME program and are aware of what to do. This will also inform current Activity UMPs of where they are in comparison with others in the project. 35 | IMPACT-MED ACTIVITY REVIEW USAID.GOV learning to promote pedagogical innovations, such as blended learning and/or adaptive learning, and to help reduce faculty workload. Such efforts will be beneficial when UMPs apply for international specialized accreditation, so these changes would serve dual purposes.Most importantly, legal corridors for CBME should be further developed on the basis of the recent developments in the healthcare sector, including the establishment of the National Medical Council. This may include a list of recognized medical and healthcare professions and competency standards of these professions, a competency-based assessment for the certification and licensing of healthcare occupations, and a quality assurance framework for medical educations that promote CBME. Competency frameworks and standards of healthcare professions should be developed and disseminated, and the certification of medical professionals should be based on competency assessments. STAKEHOLDER PARTNERSHIPS IN CBME Efforts to improve the relationship between UMPs and external stakeholders, especially private sector partners, should strengthen the prospect for the sustainability of CBME, both within IMPACT-MED and beyond. Actions to galvanize partnerships can be taken at three levels: the UMP level, the IMPACT-MED Activity level, and the multi-actor level. UMP-level efforts UMPs need to invest in maintaining close connections with their alumni through formalized alumni networks and activities for the alumni community. UMPs should also be proactive in reaching out to local enterprises and transnational corporations through promoting community services activities, exploring social responsibility statements and commitments, and approaching the prospect of partnerships from an angle of cooperation toward health security. In order to improve mutual understanding, UMPs should promote their public relations and communications about CBME externally to bridge gaps in awareness of other stakeholders about UMPs and medical education. For UMPs to cope with a lack of resources, it is advisable that UMPs intensify their efforts to connect with private sector partners to develop deep partnerships and to be strategic in seeking investment in hardware for CBME. For any UMPs wishing to develop partnerships and collaborations with foreign universities and corporations, improving staff English proficiency should be at the top of the agenda. UMP-Hospital relationships The Activity Review results show that hospitals have been deeply engaged in clinical training at hospitals for both undergraduate and postgraduate students. Since employers, including hospitals and clinics, play an important part in CBME, it is recommended that the UMPs also engage hospital doctors in the early stages of the curriculum development, specifically in the development of SLOs and competency standards, to request authentic materials for case-based learning and assessments such as the mini-CEX. Regarding the implementation of Decree 111, UMPs should share lessons and experiences as well as to discuss and reach agreement on how practices should be improved and contribute to policy formulation. 36 | IMPACT-MED ACTIVITY REVIEW USAID.GOV IMPACT-MED Activity-level efforts Since the need for professional technical expertise from UMPs is high in various areas, IPs are recommended to expand the private sector partner network that engages IMPACT-MED to relieve pressure on development partners. Further, HAIVN should make full use of existing partnerships with current Activity partners such as HMS, Novartis, and Shorelight to develop deeper and more meaningful partnerships while enhancing collaborations that appear to work well. Future collaborations could be staff and student exchanges and joint research and publication with HMS colleagues. Multi-actor-level efforts Several solutions that aim to strengthen the relationships between UMPs and private sector partners require joint efforts of many actors, including state management agencies, international donors, and local enterprises. First, a board or a committee should be formed that can act as an intermediary to connect the UMPs and private sector partners and donors, which could help build up trust between parties. Such an association, composed of representatives of different actors, may also facilitate policy discussions on incentives to encourage stakeholders and enterprises to participate in medical education. In addition, it would also be helpful if there were multiplayer dialogues and forums where donors and sponsors could meet and share interests and strategies. 37 | IMPACT-MED ACTIVITY REVIEW USAID.GOV UTILIZATION At the end of the data collection period, the research team, supported by USAID Learns, produced an Initial Findings Presentation for the Vietnam Mission and HAIVN to review and to serve the plenary Validation Event as a first step in disseminating the results of the IMPACT-MED Activity Review. This session included a full report on the research sample achieved, as well as preliminary findings as the research team understood them at the end of the data collection period, prior to full analysis. The session allowed USAID, HAIVN, UMPs, and other partners to provide feedback and clarification on the preliminary findings. USAID representatives were invited to join breakout groups during this virtual event, listen to the feedback, and interact with key university stakeholders. After the validation workshop, but prior to report submission, the research team hosted an in-person Learning Utilization Event with the engagement of all stakeholders who participated in data collection and several new UMP actors who were expected to join the IMPACT-MED Activity in Phase 2. This full-day session provided an interactive opportunity for the research team to present the full findings, conclusions, and recommendations identified after the full data analysis. The event’s afternoon workshop provided an opportunity to work with HAIVN and active UMPs to establish a clear action plan to promote the full utilization of the results, and for USAID and HAIVN to clarify any remaining issues prior to submission of the first report draft. Annex V details the outcomes of the utilization session. A two-page summary brief of this report will be shared with key stakeholders among UMPs, private sector partners, and other stakeholders to disseminate the results of the study and explain how it will influence the Activity’s strategic plan for the next five years of implementation. 38 | IMPACT-MED ACTIVITY REVIEW USAID.GOV ANNEX I: FULL LISTING OF RESEARCH PARTICIPANTS The full listing of persons interviewed was submitted separately in line with data de-identification policies. Please contact Mai Pham, mai.pham@socialimpact.com, to request the data. 39 | IMPACT-MED ACTIVITY REVIEW USAID.GOV ANNEX II: IMPACT-MED ACTIVITY UMP PROFILES UMP TYPE OF HEI NUMBER OF STUDENTS NUMBER OF FACULTY ACCREDITATION AUTONOMY STATUS UNDER￾GRADUATE POST￾GRADUATE INSTITUTIONAL LEVEL (CYCLE) PROGRAM LEVEL HCMC UMP Key 10,601 3,999 931 National (2017-2022) AUN-QA 3.0 Doctors of Medicine Program Level 2 (state budget cut on running costs) Hue UMP Member of regional university 6,937 2,000 453 National (2016-2021) National Pharmacy, Nursing, Public Health (2019) Level 2 (state budget cut on running costs) Thai Nguyen UMP Member of regional university 7,639 1,279 339 National (2017-2022) AUN-QA Doctors of Medicine, Nursing (2021) Not yet Thai Binh UMP Non-key 6,108 345 366 National (2018-2023) National Traditional Medicine-Bachelor; Public Health (Master) (2022) Level 2, starting in 2022 Hai Phong UMP Non-key 7,288 711 423 National (2018-2023) National Have completed self-report, ready for external assessment Doctor of Medicine; Bachelor of Pharmacy; Master of Public Health Level 2, starting in 2022 40 | IMPACT-MED ACTIVITY REVIEW USAID.GOV ANNEX III: FULL LISTING OF REFERENCES AND REPORTS UTILIZED Administration of Science Technology and Training, MOH. Compiled from: http://asttmoh.vn/category/dao-tao-chuyen-khoa-bsnt/cong-khai-quy-mo-dao-tao-tuyen-sinh/. Decree 99 (December 2019). Decree 99/2019/ND-CP Elaborating and providing guidelines for a number of articles of law on amendments to the Law on Higher Education. https://vanbanphapluat.co/decree-99- 2019-nd-cp-providing-guidelines-law-on-amendments-to-the-law-on-higher-education, Department of Quality Management, MOET. 2021. List of accredited universities by 31 Dec. 2021. https://moet.gov.vn/giaoducquocdan/khao-thi-va-kiem-dinh-chat-luong-giao-duc/Pages/tin￾tuc.aspx?ItemID=7714. Duong DB, Phan T, Trung NQ, et al. 2021. Innovations in medical education in Vietnam. BMJ Innovations. 7: s1–s7. Frenk, J., Chen, L., Bhutta, Z. A., Cohen, J., Crisp, N., Evans, T., Fineberg, H., Garcia, P., Ke, Y., Kelley, P., Kistnasamy, B., Meleis, A., Naylor, D., Pablos-Mendez, A., Reddy, S., Scrimshaw, S., Sepulveda, J., Serwadda, D., Zurayk, H., (2010). Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. The Lancet, Volume 376, Issue 9756, Pages 1923-1958, https://doi.org/10.1016/S0140-6736(10)61854-5. McGaghie, William C, Sajid, Abdul W, Miller, George Edward, Telder, Thomas V, Lipson, Laurette. et al. (1978). Competency-based curriculum development in medical education: an introduction / William C. McGaghie ... [et al.]; with the assistance of Laurette Lipson. World Health Organization. https://apps.who.int/iris/handle/10665/39703. Ho Chi Minh City University of Medical and Pharmacy (HCMC UMP), 2021. Retrieved from: https://ump.edu.vn/tin-tuc-su-kien/tin-tuc/dai-hoc-y-duoc-thanh-pho-ho-chi-minh-hoan-thanh-danh-gia￾chuong-trinh-dao-tao-y-khoa-va-duoc-hoc-theo-tieu-chuan-chat-luong-he-thong-dai-hoc-asean/3189. Nishtar, S. Public – private 'partnerships' in health – a global call to action. Health Res Policy Sys 2, 5 (2004). Parajuli, Dilip, Dung Kieu Vo, Jamil Salmi, and Nguyet Thi Anh Tran. 2020. Improving the Performance of Higher Education in Vietnam: Strategic Priorities and Policy Options. Washington, DC: World Bank. Sang Minh Le, Ramesh Govindaraj, and Caryn Bredenkamp (2020), Public-Private Partnerships for Health in Vietnam, Issues and Options, World Bank Group. Available at: https://openknowledge.worldbank.org/bitstream/handle/10986/33724/9781464815744.pdf. USAID, Improving Access, Curriculum and Teaching in Medical Education and Emerging Diseases (IMPACT-MED) Alliance. Quarterly report, Quarter 1, FY2017. USAID, Improving Access, Curriculum and Teaching in Medical Education and Emerging Diseases (IMPACT-MED) Alliance. Quarterly report, Quarter 2, FY2017. USAID, Improving Access, Curriculum and Teaching in Medical Education and Emerging Diseases (IMPACT-MED) Alliance. Quarterly report, Quarter 3, FY2017. 41 | IMPACT-MED ACTIVITY REVIEW USAID.GOV USAID, Improving Access, Curriculum and Teaching in Medical Education and Emerging Diseases (IMPACT-MED) Alliance. Annual report, FY2017. USAID, Improving Access, Curriculum and Teaching in Medical Education and Emerging Diseases (IMPACT-MED) Alliance. Quarterly report, Quarter 1, FY2018. USAID, Improving Access, Curriculum and Teaching in Medical Education and Emerging Diseases (IMPACT-MED) Alliance. Quarterly report, Quarter 2, FY2018. USAID, Improving Access, Curriculum and Teaching in Medical Education and Emerging Diseases (IMPACT-MED) Alliance. Quarterly report, Quarter 3, FY2018. USAID, Improving Access, Curriculum and Teaching in Medical Education and Emerging Diseases (IMPACT-MED) Alliance. Annual report, FY2018. USAID, Improving Access, Curriculum and Teaching in Medical Education and Emerging Diseases (IMPACT-MED) Alliance. Quarterly report, Quarter 1, FY2019. USAID, Improving Access, Curriculum and Teaching in Medical Education and Emerging Diseases (IMPACT-MED) Alliance. Quarterly report, Quarter 2, FY2019. USAID, Improving Access, Curriculum and Teaching in Medical Education and Emerging Diseases (IMPACT-MED) Alliance. Quarterly report, Quarter 3, FY2019. USAID, Improving Access, Curriculum and Teaching in Medical Education and Emerging Diseases (IMPACT-MED) Alliance. Annual report, FY2019. USAID, Improving Access, Curriculum and Teaching in Medical Education and Emerging Diseases (IMPACT-MED) Alliance. Quarterly report, Quarter 1, FY2020. USAID, Improving Access, Curriculum and Teaching in Medical Education and Emerging Diseases (IMPACT-MED) Alliance. Quarterly report, Quarter 2, FY2020. USAID, Improving Access, Curriculum and Teaching in Medical Education and Emerging Diseases (IMPACT-MED) Alliance. Quarterly report, Quarter 3, FY2020. USAID, Improving Access, Curriculum and Teaching in Medical Education and Emerging Diseases (IMPACT-MED) Alliance. Annual report, FY2020. USAID, Improving Access, Curriculum and Teaching in Medical Education and Emerging Diseases (IMPACT-MED) Alliance. Quarterly report, Quarter 1, FY2021. USAID, Improving Access, Curriculum and Teaching in Medical Education and Emerging Diseases (IMPACT-MED) Alliance. Quarterly report, Quarter 2, FY2021. USAID, Improving Access, Curriculum and Teaching in Medical Education and Emerging Diseases (IMPACT-MED) Alliance. Quarterly report, Quarter 3, FY2021. USAID, Improving Access, Curriculum and Teaching in Medical Education and Emerging Diseases (IMPACT-MED) Alliance. Annual report, FY20 42 | IMPACT-MED ACTIVITY REVIEW USAID.GOV ANNEX IV: DATA COLLECTION INSTRUMENTS Informed Consent Statement Hello, my name is ____________ and I am working with Social Impact on behalf of the United States Agency for International Development (USAID)/Vietnam. We are conducting a review of the IMPACT￾MED Activity (Improving Access, Curriculum and Teaching in Medical Education and Emerging Diseases) supporting five Vietnamese Universities of Medicine and Pharmacy (UMPs) to reform medical education curriculum and improve institutional governance practices and systems for continuous quality improvement. The review’s objective is to assess the extent to which the Activity met the beneficiaries’ needs, to identify the barriers and enablers to the sustainability of the Activity, and to inform the planning of future work. We expect to provide recommendations to improve the effectiveness of medical education in Vietnam and the quality of the country’s healthcare workforce. You are invited to participate in this interview because of your involvement in IMPACT-MED. We kindly request approximately one hour of your time to hear about your thoughts and opinions. There are no direct benefits and risks to you from participation in this interview beyond informing potential improvements in USAID’s programming. Participation in this study is completely voluntary. You are free to decline to participate, to end participation at any time for any reason, or to refuse to answer any individual question without penalty. FOR IN-PERSON MEETINGS ONLY Given the COVID-19 pandemic, there are several reasons you may choose not to participate in the study. If you or someone in your household or workplace has been feeling sick, including having a cough or high temperature in the past two weeks, we ask you not to participate for your safety and the safety of others. All research team members will take quick tests for COVID-19 prior to direct meetings to assure no one within the team is symptomatic. Likewise, if you are not comfortable meeting in person or have concerns about the ability to accommodate safe protocols in your place of work (such as socially distanced seating, personal protective equipment, well-ventilated meeting areas, etc.) or if you do not feel comfortable the day of the interview for any reason, you can decline to participate or end the interview early without any consequence. Also, please note that due to COVID-19, we will be keeping an internal log of all interviews, including your name and contact information, to facilitate contact tracing should any member of the research team become ill so that we would be able to inform you. All responses that you provide during this interview will be kept confidential. All information collected from the interviews will be used only for the purposes of this review, and your personal identifiers will not be shared with anyone outside of the research team. We would also like your permission to audio record this interview to make sure we do not miss any important details in our notes. This recording will be deleted after we have completed typing up all notes. Your name and other identifying information will not be published in any reports, and your responses only from our interview will be combined with others’ responses and presented in a public report. Do you have any questions about this interview? If you have any concerns, you may contact Quyen Do, the Research Team Leader, at quyen.do@socialimpact.com or the Social Impact Institutional Review Board at irb@socialimpact.com or +1 703 465 1884 with questions about the study or results. I can leave a copy/email a copy of this form with you if you would like. I have read the above information, have had the opportunity to ask any questions about this study and agree to participate in this study. 43 | IMPACT-MED ACTIVITY REVIEW USAID.GOV Do you agree to participate? Yes / No Do you agree to have this interview recorded? Yes/No Protocols For KIIs And FGDs A. UMPS – Internal stakeholders Leaders and Managers General respondents’ information: 1) Can you please tell us briefly about your work: your position, roles, and responsibilities? 2) How have you been engaged in the IMPACT-MED Activity? What are your roles in the development and implementation of the new curricula? UMP beneficiaries and competency-based education: ● Significant changes thanks to CBME 3) Can you please share about what major changes occurred when the new curricula, both undergraduate and postgraduate, were implemented? (e.g., management of academic affairs, students, etc.) 4) What important changes have been made in management and policy that are related to the new curriculum? Why are they important? ● UMPs’ priorities in CBME and Activity’s support 5) To what extent did the Activity support help you in the development and implementation of CBME and the new curricula? Did such support meet your university’s needs? Was it relevant and useful to your staff, faculty, and students? Please elaborate your answers. 6) Are there any areas where you or your university expect (further) support in order to implement CBME effectively/successfully? Please specify. ● Institutional challenges 7) From the perspective of Rector/quality assurance/academic manager - what are the obstacles that may prevent the new curriculum’s success? What do you see as major difficulties or challenges (capacity, resources, governance arrangements – line ministry management, and autonomy, policies, etc.) in initiating innovations and implementing changes, in general, and in introducing CBME, in particular, at your university? 8) In your opinion, which resources or conditions should be created for the new curricula to be implemented successfully? How do you assess the availability of those resources and conditions at your institution? 9) In general, what are your recommendations to overcome these challenges in order to implement the new curricula successfully? ● Faculty and student engagement in Activity 10) Please tell us the roles and the level of engagement of faculty and students in IMPACT-MED activities. Did you have difficulties in engaging them in innovation? 11) In what ways do you think these activities help your institution, specifically your faculty and staff, in the development and implementation of the new CBME curriculum? 44 | IMPACT-MED ACTIVITY REVIEW USAID.GOV Stakeholder engagement and partnership: ● University’s priorities – Activity’s support – complementary projects alignment 12) How did you align complementary projects and activities with your institution’s priorities in curricular innovation and CBME, specifically? 13) Did you have difficulties in handling and/or harmonizing the interests and resources provided by multiple partners, and in managing your institutional resources allocated for the development and implementation of CBME? Please share what they were and how you dealt with them. 14) What are your recommendations on how to improve the effectiveness of these activities and projects? 15) Can you please share some lessons you learned from working with different partners to develop and introduce CBME at your university? ● Private sector involvement 16) How do you rate, in terms of importance level, the roles and involvement of private partners in the Activity and the contribution to CBME? 17) What are the difficulties in working with them in those activities? 18) What contributed to or made the collaborations between your university and enterprises successful? 19) In your opinion, how should enterprises (private sector partners) be engaged during the development and implementation of CBME (in what aspects, what stage, and what roles should they play? etc.) ● Sustainability 20) In your opinion, what are the key factors (institutional capacity, IQA, UMP policies on external relationships and university-industry linkages, etc.) that would affect the effectiveness and the long￾term development of the new curriculum/CBME at your institution? 21) Please share with us your recommendations to assure the overall sustainability of CBME and the new undergraduate and postgraduate curricula at your institution, more specifically in IQA, resources for CBME, policies and strategies, leadership, and commitment, etc. Faculty General respondents’ information: 1) Can you please tell us briefly about your work: your faculty and department (where you’re based), seniority (how long have you been working), courses and subjects you deliver, students (of which year, level of education) you work with, etc.? 2) How have you been engaged in the new curriculum? Are you involved in the development and/or are you delivering the new curriculum? Please provide more details about what you did and your roles in these activities, and how important they are to you in terms of personal, professional, and institutional developments. UMP beneficiaries and competency-based education: ● Significant changes thanks to CBME 1) In your opinion, what are the key differences between the new and the old curriculum? (In pedagogies, testing and assessment, contents and/or structure, etc.) 2) What important changes have you witnessed in management and policy that are related to the new curriculum? Why are they important? 45 | IMPACT-MED ACTIVITY REVIEW USAID.GOV 3) What do you think about the changes? In your opinion, are the changes positive and necessary? Please explain why (not). ● UMPs’ priorities in CBME and Activity’s support 4) What support (technically, financially, human resources, etc.) have you received to deliver the new curriculum? What support was available for the development and implementation of the CBME curriculum? 5) To what extent did such support meet your expectations and/or needs? How relevant, useful, and sufficient were the supporting activities? 6) Are there any areas where you and your UMP expect (further) support in order to implement CBME effectively and successfully? Please specify. ● Institutional challenges 7) How do you evaluate the development and implementation of the new curriculum? 8) From the perspective of a lecturer- what are the obstacles that may prevent the success of the new curriculum? Do you have difficulties in delivering the new curriculum? Please provide more details about the areas or aspects you would need assistance with. 9) In your opinion, which resources or conditions should be created for the new curriculum to be implemented successfully? How do you assess the availability of those resources and conditions at your institution? 10) In general, what challenges should be overcome in order to implement the new curriculum successfully? ● Faculty and student engagement in Activity 11) Please tell us about the activities you were involved in - workshops, training, etc. What were your roles in these activities? 12) In what ways do you think these activities help you and your institution in the development and implementation of the new CBME curriculum? Stakeholder engagement and partnership: ● Private sector involvement 13) As we know, many of the support activities were provided and co-organized by enterprises. How do you assess their involvement in the Activity and the contribution to CBME? 14) What are your recommendations to enhance the engagement of enterprises and the effectiveness of the collaboration? Sustainability: 15) In your opinion, what are the key factors (institutional capacity, IQA, UMP policies on external relationships and university-industry linkages, etc.) that would affect the effectiveness and the long￾term development of the new curriculum/CBME at your institution? 16) Please share with us your recommendations to assure the overall sustainability of CBME and the new undergraduate and postgraduate curricula at your institution, specifically in IQA, resources for CBME, policies and strategies, leadership and commitment, etc. Student General respondents’ information: 1) Can you please tell us briefly about yourself: your name, faculty or department, your program, level of education (undergraduate or postgraduate), year of study, your future career, etc.? 46 | IMPACT-MED ACTIVITY REVIEW USAID.GOV 2) Do you enjoy the learning opportunities and experiences at this UMP? Please share a bit more about your learning experiences. UMP beneficiaries and competency-based education: ● Significant changes thanks to CBME 3) Can you please describe the learning activities and tasks that you experience in your program? Which activities and what type of tasks are you most and least interested in? 4) How are you assessed in courses? What do you think about the course requirements? In your opinion, to what extent are the requirements close to job requirements in your field? 5) How practical and relevant do you think the course contents and knowledge provided to you? 6) What skills or competencies (language, soft skills, professional, etc.) do you think the program or courses have helped you to develop? 7) In general, are you confident that you will meet the job requirements at hospitals? Why or why not? 8) Do you think the program will help you to achieve your career goals? Please explain why or not. 9) Please name some shortcomings of your program. 10) If there are recommendations to improve the learning experience and training quality of the program, what are they? 11) Do you have any suggestions to promote valuable aspects and to gradually solve challenges of your program? 47 | IMPACT-MED ACTIVITY REVIEW USAID.GOV B. External stakeholders Activity partners (HAIVN, USAID, private sector partners) General respondents’ information: 1) Can you please tell us briefly about your involvement in IMPACT-MED: areas of support, the UMPs you collaborated with, the purposes of Activity, the direct beneficiaries, and the extent to which you achieved the objectives, etc. UMP beneficiaries and competency-based education: ● Significant changes thanks to CBME 2) Based on what you did/have done with the UMPs, what specific changes do you think your support brings to the teaching and learning practices at UMPs? ● UMPs’ priorities in CBME and Activity’s support 3) To what extent do you think your activities at UMPs are relevant to curriculum innovation and CBME development at UMPs? ● Institutional challenges 4) How do you assess the progress and/or results of Activity at the partner UMPs? 5) From the perspective of businesses, what challenges and advantages did you face working with partner UMPs to carry out the project activities? 6) What are your recommendations to improve the effectiveness of the collaborative work between the two parties? 7) Based on your experience working with partner UMPs, what are the issues or problems that may hinder the reforms or the implementation of innovation initiatives and changes, in general, and of CBME, in particular, at the partner UMPs, and how to overcome or mitigate them? 8) In your opinion, what conditions necessary for curriculum innovation and CBME implementation at UMPs are absent and/or should be enhanced? ● Faculty and student engagement in Activity 9) Who did you engage in your project activities at the UMPs, such as members of the Rectorate Board, managers and Deans, lecturers, and/or students? What were their roles? How did the participation in the activities help them in the development and implementation of the CBME? Stakeholder engagement and partnership: ● University’s priorities - Activity’s support – complementary projects alignment (WHO/MOH HPET Project) 10) As you know, there were other partners participating in the Activity, and UMPs had a number of projects running at the same time as well. How did you align your project activities with other partners' and UMP's priorities? Did you have difficulties working and connecting with them? Please explain why or why not. 11) In your experience, what might help projects and activities better align with one another? Could you please share the basis on which the activities and work agenda were developed? Was it modified in terms of action, content/theme, objectives, methods etc. during the implementation process? 48 | IMPACT-MED ACTIVITY REVIEW USAID.GOV ● Private sector involvement 12) Can you please share with us what you did or have been doing with UMPs in relation to the advancement of CBME, your purposes, if you achieved the objectives, the approaches taken to work with UMPs, the difficulties and advantages you faced/had reaching and collaborative with UMPs. 13) How do you rate or assess the partnership between you and the UMPs in training and training￾related issues in terms of effectiveness and the benefits both sides gained? 14) In your opinion, what should be improved (funding mechanism, collaboration policies and schemes, level of commitment, etc.) to attract enterprises to collaborate with universities in training? Sustainability: 15) In general, what are your recommendations to create favorable conditions for CBME to develop and be sustainable? State agencies – International donors (MOH, WHO, World Bank) Stakeholder engagement and partnership: 1) What do you think about the importance of CBME in Vietnam? 2) How important do you think the partnership between universities/UMPs and enterprises in the development of CBME? What are your recommendations to strengthen the relationship between industry and university? Sustainability: 3) What conditions should be provided to assure the sustainability of CBME at UMPs? Employers (Hospitals and Clinics) Stakeholder engagement and partnership: 1) Do you receive students from UMPs for internships at your hospital or clinic? If no, please share why not; If yes, please share how this happens, how the students are sent to you, officially via cooperation activities between organizations or unofficially through personal contacts with doctor and professor networks or else? 2) What benefits do you get from receiving students for internships? 3) What do you think about the capabilities of graduates that UMPs provide to the labor market? To what extent do UMPs' internship students meet your job requirements in terms of professional and technical skill, knowledge, soft skills, and work attitudes? 4) Have you ever been invited by any UMPs to participate in the curriculum development and training, possibly in the form of commenting on course and curricula design and syllabi, etc., attending workshops or consultation sessions on curriculum development and graduate attributes and skills, and so on? Please share more details if yes. If no, what do you think about the roles of key employers such as hospitals and clinics in medical education? How important is the participation of hospitals and clinics in the training processes? 5) Do you have any recommendations on how to improve graduates from UMPs and CBME? 6) In your opinion, how important is the close connection between hospitals/clinics and UMPs in improving medical education graduate quality and competencies? What are recommendations to enhance/strengthen the relationship/ partnership between hospitals/clinics and UMPs? 49 | IMPACT-MED ACTIVITY REVIEW USAID.GOV ANNEX V: UTILIZATION WORKSHOP OUTCOMES IMPACT-MED ACTIVITY REVIEW UTILIZATION WORKSHOP OUTCOMES Time & Date: 13:30 – 16:30, May 18, 2022 Venue: Pan-Pacific Hotel, 1 Thanh Nien, Hanoi Participants: 24 representatives (afternoon session only) from: ● Five UMPs (current IMPACT-MED activity partners): Thai Nguyen UMP, Hai Phong UMP, Thai Binh UMP, Hue UMP, and HCMC UMP ● HAIVN ● USAID Learns (facilitator). Objectives ● Formulate actionable points for each stakeholder group based on the validated recommendations ● Identify potential areas for and to strengthen multi-stakeholder collaboration for CBME implementation in Vietnam. High-level Agenda ● Welcome ● Presentation on validated recommendations from the IMPACT-MED Activity Review ● Group discussions Block 1: Formulating action points ● Select and prioritize recommendations based on level of impact and effort ● Identify actions to be implemented by UMPs, HAIVN, and GVN to realize the recommendations ● Exchange ideas about action planning and implementation experience among UMPs Block 2: Identify opportunities and ways to strengthen collaboration in CBME implementation ● Identify ways to improve inter-UMP collaboration, especially sharing developed tools and resources ● Identify opportunities for multi-sector collaboration for CBME implementation and development ● Overall feedback and closing by HAIVN Overall observations The action points that were generated by stakeholders in response to recommendations were identified as priorities by the UMPs. For several UMPs, key priorities were classified as those thought to promise “high impact” for “medium/low” effort. Other UMPs continued exploring specific recommendations that could bring direct results and complement ongoing programming efforts at the UMPs. There was strong recognition and acknowledgment of learning and sharing among UMPs as an essential method to develop programming and capacity moving forward. HAIVN is expected to play the role of facilitator in connecting the universities. The participants recommended that producing a map or diagram 50 | IMPACT-MED ACTIVITY REVIEW USAID.GOV of each UMP’s capacity to implement CBME within the activity system could be a useful starting point to improve collaboration, learning, and adapting. Opportunities were identified for inter-university and multi-sector collaborations to provide a stimulus for several actions to improve and sustain CBME among universities in the activity system; these are identified in the Block 1 highlights below. GVN players were recognized as essential yet challenging stakeholders in issuing the policies and regulations needed to support CBME. Participants recognized that the universities need to make joint efforts and be strategic in involving responsible GVN agencies in various activities organized by the UMPs. Discussion highlights on formulating actions that are required for CBME sustainability and development (Block 1) Pertinent actions were developed based on the actual needs of each UMP through a careful and thorough review. The participants first analyzed each recommendation from the lens of impact and effort and prioritized those with a high perceived impact and lower levels of effort. Some UMPs dug deeper into the current situation of CBME implementation and shortlisted the ones that were new or would directly complement their current activities. This process resulted in the selection of more tailored and context￾specific actions. Even though the majority of actions were formulated based on the needs and priorities of each UMP, there were several common recommendations deemed highly relevant among UMPs, including: ● CBME programing evaluation and development of competency-based assessments ● Staff professional development ● Developing a map of CBME themes and components ● Development of IQA and local and international specialized program accreditation ● Student support This common ground highlighted a number of potential opportunities for coordination, experience, and resource sharing in the future: ● Common actions recommended by participants to HAIVN mainly covered technical assistance support, including access to experts, trainers, and more opportunities for universities to share and learn together. ● Recommended actions for GVN agencies were related to regulations, working mechanisms, and the development of national medical licensing examinations and accreditation. ● UMPs identified a wide range of desired capacity development activities, including technical training, workshops to learn from experts, and the need for increased exchanges of successful program implementation experiences with other universities. After Phase 1 of implementing CBME, USAID’s and HAIVN’s focus may now move to quality improvement and good governance. Actions could include additional evaluations and assessments, drawing on stakeholder feedback, and developing systems to evaluate graduate competencies. Discussion highlights from identification of opportunities and ways to strengthen collaboration (Block 2) It was widely recognized by the UMPs that there is great potential for universities to collaborate more in various areas (e.g., improving technical expertise, funding, human resources, and advocacy for CBME policies). Collaboration was warranted by stakeholders among all five UMPs in the IMPACT-MED system, and could be directed and facilitated among those who share similar interests in CBME development. 51 | IMPACT-MED ACTIVITY REVIEW USAID.GOV A number of ideal conditions were recognized by stakeholders for effective collaboration among UMPs, but the most significant was leadership and mindset (e.g., willingness to share, sense of ownership and buy￾in) and the creation of a specific mechanism to boost collaboration. Participants suggested this could be furthered by developing a memorandum of understanding with defined terms and roles for each side. Participants believed that UMPs with shared objectives and programming goals could collaborate best, but there needed to be more opportunities for experienced UMPs to share skills and techniques in creating CBME-related products, with the recognition that the products may not match each UMP’s condition. Some participants believed that each UMP should create its own products, which could increase buy-in to the process. However, UMPs need to proactively initiate partnerships with private sector actors and industries, while HAIVN or USAID could act as a bridge to effectively connect them with potentially relevant enterprises. Further, UMPs must actively seek partnerships with public and private hospitals to provide visiting lecturers and opportunities for practice or internships for UMP students. DETAILED RESPONSES BY PARTICIPANTS DURING GROUP DISCUSSIONS On formulating action points from validated recommendations (Block 1) Hai Phong UMP: Priority was given to recommendations that (1) had high levels of impact and medium/low levels of effort and (2) those which linked well with the university’s needs and further supported or incentivized the implementation of ongoing reform. Table 1: Hai Phong UMP’s reflections from group discussion PRIORITIZED RECOMMENDATIONS ACTIONS REQUIRED FROM THE UMP ACTIONS REQUIRED FROM HAIVN/USAID IP ACTIONS REQUIRED FROM GVN 1. CBME implementation (undergraduate and postgraduate education) − Develop outcome standards for postgraduate groups − Develop the program curriculum and assessment framework Continue to provide technical assistance and support implementation costs Standardization of undergraduate and postgraduate education programs (outcome standards, national practitioner testing and certifications) 52 | IMPACT-MED ACTIVITY REVIEW USAID.GOV PRIORITIZED RECOMMENDATIONS ACTIONS REQUIRED FROM THE UMP ACTIONS REQUIRED FROM HAIVN/USAID IP ACTIONS REQUIRED FROM GVN 2. Staff Professional development (permanent and visiting lecturers) − Organize training programs for developing faculty staff − Issue new policies to support and incentive teaching staff − Provide technical assistance, training programs (in CBME) − Train core teachers − Support implementatio n costs Develop criteria to standardize teachers’ capacity based on CBME 3. Strengthen relationships between UMP & hospitals Expand number of hospitals for student practice Support implementation costs 4. Student support − Train teaching staff and students; develop regulations for academic support programs for students − Look for funding sources to support implementation − Provide support specific to completing the training materials package − Technical assistance for implementing activities effectively 53 | IMPACT-MED ACTIVITY REVIEW USAID.GOV PRIORITIZED RECOMMENDATIONS ACTIONS REQUIRED FROM THE UMP ACTIONS REQUIRED FROM HAIVN/USAID IP ACTIONS REQUIRED FROM GVN 5. Competency-based assessment − Finalize the strategies and tools for CBME − Train faculty staff on competency￾based assessment − Train core faculty staff − Provide support to create centers for CBME assessment − Provide support to develop assessment tools of CBME − Support to connect with stakeholders − Organize trips for learning and exchange experiences 6. IQA and International Accreditation − Train staff working in IQA − Self-assessment of the education programs − Finalize the IQA system − Learn and exchange experience in IQA and international accreditation with other UMPs Support and review programs according to accreditation criteria Support in establishing Accreditation Centers 54 | IMPACT-MED ACTIVITY REVIEW USAID.GOV Thai Binh UMP: Priority was given to recommendations with high impact and lower levels of effort. Table 2: Thai Binh UMP’s reflections from group discussions RECOMMENDATION S RANKED BY PRIORITIES ACTIONS REQUIRED FROM THE UMP ACTIONS REQUIRED FROM HAIVN/USAID IP ACTIONS REQUIRED FROM GVN AND OTHERS 1. IQA – CQI; Feedback survey of stakeholders − Conduct feedback survey to employers, recruiters − Establish IQA system Support with funding to develop tools Responsible agencies to provide guidance, tools, templates, and applications 2. Evaluate graduates’ competency Evaluate graduate students through feedback of recruiters Analyze, evaluate results objectively and independently Responsible agencies to publish and issue general standards 3. Provide systematic, comprehensive, and individualized support programs to students Establish student support units Train skills in coaching, counseling, and consultation UMP associations organize evaluations on students 4. Frequent adjustment of the program curriculum − Conduct survey among stakeholders − Workshops with experts − Conduct comprehensi ve evaluation Provide feedback and recommendations − Provide feedback from recruiters and experts − GVN agencies issue competency standards for each specialization and program 55 | IMPACT-MED ACTIVITY REVIEW USAID.GOV RECOMMENDATION S RANKED BY PRIORITIES ACTIONS REQUIRED FROM THE UMP ACTIONS REQUIRED FROM HAIVN/USAID IP ACTIONS REQUIRED FROM GVN AND OTHERS 5. Comprehensive evaluation of CBME programs Conduct comprehensive evaluation Provide consultation and technical assistance in techniques, benchmarking Practicums and internship centers to provide feedback and evaluations 6. Mapping of CBME themes − General survey − Planning 7. Scale-up CBME to other UMPs − Technical support − Share experience − Exchange teaching staff Connect, coordinate, develop and share plans − MOH: issue guiding instructions − MOET: agree and support the plan Hue & HCMC UMPs: Priority was given to the recommendations that (1) had high impact, and (2) were new or directly complementary to what has been implemented to date. The group chose not to brainstorm actions required by GVN actors and instead focused on their own institutions and HAIVN. Table 3: Hue and HCMC UMPs’ reflections from group discussion PRIORITIZED RECOMMENDATIONS ACTIONS REQUIRED FROM THE UMP ACTIONS REQUIRED FROM HAIVN/USAID IP Staff professional development Integrate professionalism within clinical training Support in improving capacity in professionalism in clinical training programs Develop research capacity in medical education Maintain and promote publications, participation in conferences, medical education clubs Support research in medical education, editing for to be published articles, support for attendance feed to join conferences 56 | IMPACT-MED ACTIVITY REVIEW USAID.GOV PRIORITIZED RECOMMENDATIONS ACTIONS REQUIRED FROM THE UMP ACTIONS REQUIRED FROM HAIVN/USAID IP Scaling up CBME to Postgraduate programs Renovate programs toward CBME (both universities have started now) Provide technical assistance and experts IQA & CQI Develop professional tools (have not developed yet) Provide technical assistance Clinical simulation centers Possess infrastructure and equipment already, need to develop staff and testing procedures EPA assessment tool Develop tools to monitor students’ progress Thai Nguyen UMP: Priority was given to recommendations with high impact and lower levels of effort. Table 4: Thai Nguyen’s UMP’s reflections from group discussion PRIORITIZED RECOMMENDATIONS ACTIONS FROM THE UMP ACTIONS FROM HAIVN/USAID IP ACTIONS FROM GVN Feedback survey from stakeholders Plan to develop specific tools, organization of survey activities Curriculum revisions Review and adjust Technical assistance, experts Assessment − Provide guidelines on standard assessment − Establish Testing Center 57 | IMPACT-MED ACTIVITY REVIEW USAID.GOV PRIORITIZED RECOMMENDATIONS ACTIONS FROM THE UMP ACTIONS FROM HAIVN/USAID IP ACTIONS FROM GVN Professionalism Curriculum development − Assistance in testing − Teaching and assessment methods − International relations Staff professional development Establish a responsible unit (to make and develop plans) − Provide assistance in developing training materials, sharing experience − International collaboration, workshops Support training for core teaching staff − Clinical pedagogies − Involve hospital staff in teaching − Increase the number of visiting professors − Create budget policies Provide assistance for faculty to experience exchanges Revise the mechanism to enable university autonomy and self-financing Maps and checklist of CBME themes − Develop themes checklists of CBME as reference for universities − CBME experts join the technical working groups of universities Intermediaries to connect UMP and private sectors Look for potential partners, develop proposals for funding Identify potential partners and match with the UMP 58 | IMPACT-MED ACTIVITY REVIEW USAID.GOV PRIORITIZED RECOMMENDATIONS ACTIONS FROM THE UMP ACTIONS FROM HAIVN/USAID IP ACTIONS FROM GVN Student support − Establish student support centers (working mechanism and implementation strategy) − Develop support programs (e.g., mentor and tutor programs) Assistance in training and support programs IQA and international accreditation programs − Plan − Implement training plans − Implement accreditation programs Support access to international conferences, experience sharing Develop criteria, accreditation for each specialization CBME (Undergraduate – Postgraduate) Residency program Inter-UMP workshops Regulations on specialized education HAIVN: Recommendations selected were those with high impact and a lower level of effort. Table 5: HAIVN’s reflections from group discussion RECOMMENDATIONS BE IMPLEMENTED BY THE UMPS RECOMMENDATIONS IMPLEMENTED BY THE UMPS ACTIONS FROM GVN AND OTHERS CBME map and checklist Faculty development Faculty development materials Increase or improve faculty/student ratio Linkage with regional and international CBME associations 59 | IMPACT-MED ACTIVITY REVIEW USAID.GOV RECOMMENDATIONS BE IMPLEMENTED BY THE UMPS RECOMMENDATIONS IMPLEMENTED BY THE UMPS ACTIONS FROM GVN AND OTHERS − Clearly define what UMPs want from private sector actors − Identify needs Linkage with private sector UMPs are willing to share materials Knowledge sharing “e-library” Student support Research capacity-building UMP commitment to accreditation Provide technical assistance − International organizations recognize and support Accreditation. − Government and agencies recognize accreditation On identifying opportunities and ways to strengthen collaboration in CBME implementation (Block 2) A. Inter-UMP collaboration: Learning from experience in collaboration: Elements for good collaboration: ● UMP leadership ● Collaboration strategies among UMPs in place ● Having adequate finance resources ● Suitable human resources (efforts, attitude) ● Shared capacity needs of UMPs ● Inter-university collaboration must start from the shared goal ● Collaboration needs a lead facilitator; the universities can take turns to lead the coordination ● Fee built for technical experts ● UMPs’ readiness ● Leadership ● Teaching staff ● Willing to share, learn among the universities (technical sharing, taking ownership of the work) ● Mutual trust 60 | IMPACT-MED ACTIVITY REVIEW USAID.GOV Ideas on how to share resources and tools among universities: ● HAIVN to connect and create a capacity map of the UMPs ● Develop a memorandum of understanding among universities with clear terms on working together ● Develop mechanism for inter-university collaboration ● Share materials ● Create inter-university research groups (projects with the involvement of students, teaching staff) ● Inter-university workshops could be facilitated by HAIVN ● Inter-university technical groups, core groups with clearly identified needs and objectives are key ● Contribute and share resources by publicizing the materials, promote transparency ● UMPs to be active, planning focused and responsible ● Assignment and coordination ● Connected by focal points (especially when the project is extended, coordination is important) ● Responsibility and benefits of participation ● UMPs bring in the measurable organizational values terms and are proactive ● Between universities: understand others’ capacities and needs ● Between small group of universities: sharing the same needs ● Diversity in implementation levels among old and new universities, the more experienced can share through a training of trainers approach ● Enhance experience sharing between universities Areas for inter-UMP further collaboration: ● Collaboration in exchanges of different programs ● Collaboration in postgraduate education ● Collaboration in medical research and education among universities ● Collaboration in teaching staff development: sharing management and coordination of staff development activities. B. Multi-sector collaboration: What do UMP’s look for in the collaboration? ● Common topics for policy advocacy ● Advocacy for policy, through national medical council, employers ● MPET and UMPs to develop working groups to develop curriculum & outcome standards ● Advocacy activities joined by many universities ● National medical conference ● Advocacy for the involvement of the Examination and Treatment Department (MOH) ● Commitment will result in increased income, which will improve quality ● Collaboration with private sector in scientific research ● Universities need to collaborate and jointly agree to propose innovative policies for CBME ● Look for partnership with private hospitals to increase the faculty/student ratio ● Build good partnership with health centers to create opportunities for internship and scholarships for students (Hanoi UMP has been undertaking this; further efforts are needed from Thai Nguyen and Hai Phong) 61 | IMPACT-MED ACTIVITY REVIEW USAID.GOV Learning experience of working with multi-sectors: ● Elements for effective collaborations: ● Shared capacity needs ● University autonomy ● Collaboration with MOH and MOET ● Consultation with stakeholders ● What has not been doing so well: ● Lack of partnership with private sector actors ● Lack of autonomy and mechanisms to initiate partnership with the private sector ● Few advantages linked to the partnerships with enterprises/industries. ● Yet to start the alumni, other networks ● Yet to develop partnership with private hospitals to find worthwhile conditions and equipment for practice, and hence improve the outcome quality ● Universities need to be more active and strategic in seeking new partnership and promoting and sustaining existing partnerships Ideas to create effective multi-sector collaboration: ● Create local health networks to advocate for improved policies ● Identify needs of partners and other pharmacies ● Extract some funding from CBME implementation for Community of Practice development ● Connect with international agencies ● Create inter-university projects, e.g., HPET, IMPACT-MED, MOH Recommendations ranked by the levels of impact and effort per UMP 62 | IMPACT-MED ACTIVITY REVIEW USAID.GOV Picture 1: Ranking Recommendations of Each Discussion Group 1. Hai Phong UMP 2. Thai Binh UMP 63 | IMPACT-MED ACTIVITY REVIEW USAID.GOV 3. Hue & HCM UMP 4. HAIVN 64 | IMPACT-MED ACTIVITY REVIEW USAID.GOV Event Activities Picture 1: Prioritizing recommendations on the effort & impact diagram Photo credit: USAID Learns Picture 2: Inter-UMP discussion 65 | IMPACT-MED ACTIVITY REVIEW USAID.GOV Photo credit: USAID Learns Picture 3: “CBME gallery” walk Photo credit: USAID Learns Picture 4: Reflecting on the discussion Photo Credit: USAID Learns 66 | IMPACT-MED ACTIVITY REVIEW USAID.GOV Figure 5: Group discussion Photo credit: USAID Learns Figure 6: UMP table discussion Photo credit: USAID Learns 67 | IMPACT-MED ACTIVITY REVIEW USAID.GOV ANNEX VI: STUDY STATEMENT OF WORK STATEMENT OF WORK: IMPACT-MED ACTIVITY REVIEW Background This Statement of Work (SOW) describes the activity review for the USAID IMPACT-MED activity (Improving Access, Curriculum and Teaching in Medical Education and Emerging Diseases). The activity is supporting five Vietnamese Universities of Medicine and Pharmacy (UMPs) to reform medical education curriculum and improve institutional governance practices and systems for continuous quality improvement. IMPACT-MED is focused on advancing competency-based medical education (CBME) in Vietnam. Overview of the IMPACT-MED Activity The goal of the IMPACT-MED Alliance is to build a strong and effective health workforce in Vietnam able to respond to 21st-century priorities and to contribute to the health resilience and health security of the country. (1) To improve undergraduate medical education: through revising and implementing undergraduate programs at partner universities, improving leadership, faculty, and staff capacity to lead the reform; and increasing disadvantaged students’ opportunities to succeed in medical school. (2) To improve postgraduate medical education: through contributing to policy development to standardize and improve postgraduate medical education, developing and implementing a standardized 3- year surgical residency program and framework (added according to Modification #3 dated August 14, 2018). Purpose USAID/Vietnam intends to conduct an activity review of IMPACT-MED. This review will serve a dual purpose: (1) to learn about the extent to which the activity has met its beneficiaries’ educational needs and (2) identify barriers and enablers to the overall sustainability of the activity's interventions, both of which will inform planning for the activity's extension period (Phase 2). This review will assist the Mission and The Partnership of Health Advancement in Vietnam (HAIVN) in reaching decisions related to (1) the effectiveness of the current approaches to promote competency-based medical education; and (2) preparing and adapting future work planning based on lessons learned from the current activity and its overall sustainability. The research will also aim to provide insight into how future programming more generally could benefit Vietnam’s healthcare workforce and the country’s overall health security through exploring domains related to student and faculty needs, stakeholder engagement, and sustainability. Target Audience & Use The primary audience of the Activity Review will be USAID/Vietnam, specifically the Office of Higher Education, and HAIVN, the Mission’s IP. The Ministry of Health (MOH) and five partner UMPs will be tentative secondary audiences. Based upon a flexible dissemination plan, representatives from the MOH, the Ministry of Education and Training (MOET) and up to 20 higher education institutions will act as a tertiary audience. 68 | IMPACT-MED ACTIVITY REVIEW USAID.GOV PRIMARY AND SECONDARY RESEARCH USERS RESEARCH INFORMS USAID Phase 2 work planning, future design and partnering strategies, overall management of the award HAIVN Phase 2 work planning MOH Knowledge on general CBME development barriers/enablers, future CBME development and partnering strategies Select UMPs Knowledge on specific CBME development barriers/enablers Review Questions and Methodology An overarching question for the Activity Review concerns how suitably the activity has aligned with the local context and its overall trajectory, especially in the field of developing CBME in Vietnam. A research paper on this topic was recently published by activity actors: Innovations in medical education in Vietnam (bmj.com). The research questions for the Activity Review will focus on the themes of UMP needs and competency￾based education, stakeholder engagement and partnership, and sustainability. Since IMPACT-MED works with UMPs in the north, center, and south of Vietnam, this assignment will require consultants to travel to Thai Nguyen, Hai Phong, Thai Binh, Hue, and HCMC to collect data. Some data will also be collected in Hanoi. In-person data collection is dependent on epidemic control restrictions stipulated by the Government of Vietnam (GVN). This review will use a mix of traditional and participatory data collection and analysis methods: ● Traditional methods may include, (i) literature / desk review, (ii) key informant interviews (KII), and (iii) focus group discussions (FGD). ● Participatory data collection: contingent on stakeholder enthusiasm and an easing of social distancing, researchers will be encouraged to utilize alternative modes of investigation to empower stakeholders in and through the research process. Participatory methods are envisioned to elicit local perspectives on sustainability and explore how results can be sustained and transferred on to local partners. This is especially relevant in relation to UMP needs, stakeholder engagement and activity partnership. Participatory methods may be utilized during facilitated dialogues/participatory FGDs with selected groups of key stakeholders and throughout the data analysis process, which will be complemented by (i) a Validation Event / consultation workshop to validate findings and co-create recommendations, (ii) a Utilization Event to operationalize recommendations, and (iii) a Two-way Dissemination Event with GVN and wider higher education institution stakeholders. The consultant research team will develop the tools for each method (with support from Learns) starting its work with (i) a review of all the documents provided by USAID and HAIVN, and (ii) an initial consultation with key local stakeholders. The aim of the initial session will be to understand stakeholder’s needs and assess their willingness to engage in a participatory research process. Where appropriate and feasible, the team will make sure that there is an equal representation between males and females among respondents throughout the research. The Mission is looking for suggestions to guide and shape future work planning toward enhanced sustainability and effectiveness. It is anticipated that based on the initial research questions and tentative 69 | IMPACT-MED ACTIVITY REVIEW USAID.GOV methodologies provided in Table I below, the research team will develop a more detailed research design and workplan in the inception report. 70 | IMPACT-MED ACTIVITY REVIEW USAID.GOV Table 1: Questions Matrix REVIEW THEME AND QUESTIONS DATA COLLECTION METHOD(S) DATA SOURCE(S) SAMPLING OR SELECTION CRITERIA DATA ANALYSIS METHOD(S) LEARNING AIMS UMP beneficiaries and competency-based education 1a) What significant changes have resulted for students and faculty as a result of the activity’s support surrounding CBME? 1b) To what extent has the activity helped UMPs achieve their priorities linked to developing a competency-based medical education curriculum? 1c) What institutional challenges should be addressed in Phase 2 (2022 - 2026) of the activity to better measure, institutionalize and sustain CBME at UMPs? 1d) In what ways were students and faculty engaged/involved during the activity and how did the activity empower them in reforming the CBME curriculum? Desk Review, FGD/Participatory FGD, KII/Facilitated Dialogue, Validation Primary/secondary literature, activity reports, discussion notes, interview notes, workshop notes Students, UMP faculty and staff (Thai Nguyen, Hai Phong, Thai Binh, Hue and HCMC), HAIVN representatives Qualitative Content Analysis Implementation gaps and contextual factors influencing programming at UMPs and barriers/enablers to advance this specialism in the activity extension Stakeholder engagement and partnership Desk Review, KII/Facilitated Dialogue, Validation Primary/secondary literature, activity reports, discussion MOH, MOET, WHO representatives, World Bank representatives, Qualitative Content Analysis Lessons learned from collaboration with a bilateral funding 71 | IMPACT-MED ACTIVITY REVIEW USAID.GOV REVIEW THEME AND QUESTIONS DATA COLLECTION METHOD(S) DATA SOURCE(S) SAMPLING OR SELECTION CRITERIA DATA ANALYSIS METHOD(S) LEARNING AIMS 2a) What were the opportunities, challenges, and lessons learned that emerged from aligning priorities and activities between different stakeholders and/or complementary projects in developing a CBME curriculum? 2b) What worked well with private sector involvement to advance CBME, and where is there room for improvement in the future to enhance the benefit of private sector involvement? notes, interview notes, workshop notes USAID, Local activity partners, Private sector (e.g., Microsoft, Samsung), HAIVN representatives, faculty and UMP and other staff, students mechanism, private sector partnership and future key priorities. Sustainability 3a) What developments, for example in terms of support for institutional capacity, IQA or UMP policy, could help to ensure the sustainability of CBME? 3b) What actions should be taken to safeguard the sustainability of CBME in Vietnam and how should any barriers or enablers presented by national policies be navigated? Desk Review, Participatory FGD, Facilitated Dialogue FGD/ KII/ Validation Primary/secondary literature, activity reports, discussion notes, interview notes, workshop notes MOH, MOET, UMP faculty, HAIVN representatives, USAID Qualitative Content Analysis Reflection on the overall trajectory of the activity and contextual factors that ought to influence subsequent work planning. 72 | IMPACT-MED ACTIVITY REVIEW USAID.GOV DELIVERABLES AND TENTATIVE TIMELINE Inception Report and Inbrief: The research team will prepare an Inception Report which will include a detailed research design, key definitions, chosen methodologies, and the overall research plan. The detailed research design and work plan will be developed in close consultation with key partners including HAIVN and local stakeholders who are involved in activity implementation. The Inception Report will be submitted by USAID Learns to the USAID/Vietnam Activity Manager for approval. The research team must prepare the Inbrief summarizing the Inception Report for USAID/Vietnam and HAIVN before implementation. The Inception Report will need to be presented to and approved by USAID before data collection is launched. Validation Event: Following data collection and analysis, the team will present the preliminary finding of the research to USAID, HAIVN partners, and key stakeholders (as appropriate and as defined by USAID/Vietnam or HAIVN) through a PowerPoint presentation and a facilitation event supported by Learns. This will validate findings in front of actors and help to develop/co-develop user-driven recommendations. The research team will consider partners’ comments, develop recommendations, and report accordingly, as appropriate. Utilization Event: The team will develop a refined PowerPoint presentation for a Utilization Event to encourage users to jointly agree on how to operationalize recommendations, collaborate, and follow up on progress in preparation for activity extension. (More details concerning the planned Validation and separate Utilization Event are discussed below under scheduling.) Draft Report: A draft report of the findings and recommendations, no longer than 30 pages excluding annexes, will be submitted to the USAID activity manager and Learns COR (by Learns). The format will include an executive summary, table of contents, methodology, findings, and recommendations. USAID/Vietnam and HAIVN will provide comments on the draft report. Final Report: The team will submit a final report that incorporates responses to USAID’s/partner’s comments and suggestions. The report will be submitted electronically in English. The report will be disseminated within USAID, HAIVN and the MOH. TEAM COMPOSITION The consultant team will consist of a team leader and three technical experts. A mix of gender is strongly encouraged for the team composition. A representative from USAID/Vietnam or USAID/Vietnam Learns may participate as well. 1. The Team Leader and Learning Expert (international or local) must have a postgraduate degree in international development, education, clinical research, or a suitably related field. S/he must have at least 7 years of senior-level experience working in higher education. S/he must have extensive experience in conducting qualitative evaluations/assessments, preferably related to medical education and participatory research. Excellent oral and written skills are required. The Team Leader must also have experience in leading research teams and preparing high-quality documents. S/he will provide leadership for the team, finalize the research design, coordinate activities, arrange periodic meetings, consolidate individual input from team members, and coordinate the process of communicating and assembling the final findings and recommendations into a high-quality document. S/he will take overall responsibility for analysis and write the final 73 | IMPACT-MED ACTIVITY REVIEW USAID.GOV report. S/he will also lead the preparation and presentation of the key findings, conclusions, and recommendations for each deliverable. 2. The Research Specialist (local) must have a postgraduate degree in socio-economic development, education policy, health policy, or a related area. S/he must have at least 7 years of experience conducting high-quality research, preferably on the subject of health security. S/he must be knowledgeable in program assessment and qualitative, preferably participatory, research methodologies. S/he will lead the process of arranging interviews/FGDs and organizing/translating transcripts. S/he will support the development of the research tools, analysis, and reporting. 3. The Medical Education Advisor (local) must have a degree/higher education level qualification in medical education curriculum development and/or research or related fields. S/he must have at least 7 years of experience supporting students within Vietnam’s medical education system preferably in clinical theory, clinical rotations, and apprenticeship/residency. Experience working abroad is highly valued. S/he would preferably be knowledgeable in program assessment and research methodologies. S/he must have experience in conducting qualitative research. 4. The Research Coordinator (local) must have strong experience in coordinating and conducting research. Experience working in the field, collecting data, conducting interviews, preferably on the subject of education, medical education. S/he would preferably be knowledgeable in program assessment and research methodologies. ACTIVITY REVIEW MANAGEMENT USAID Learns and Social Impact’s headquarters will provide overall management, coordination, and quality control for research. Logistical coordination, interpretation, as well as scheduling and several facilitation sessions will be managed by the USAID Learns Research Team in Hanoi, in conjunction with USAID/Vietnam and HAIVN. SCHEDULING 74 | IMPACT-MED ACTIVITY REVIEW USAID.GOV ANNEX VII: POLICY RECOMMENDATION BRIEF BARRIERS/ENABLERS RELEVANT POLICIES/ DEVELOPMENTS HYPOTHETICAL POLICY RECOMMENDATIONS THAT WOULD BENEFIT USAID PROGRAMING Barrier: Dual academic management system Law 34 - Revised HE Law; Decree 99 The year lead and module lead positions should be officially recognized in parallel with subject lead, so that these management positions are entitled to allowance schemes for the duties they undertake. Barrier: Lack of full autonomy Law 34 - Revised HE Law; Decree 99 UMPs should be granted full autonomy in academic affairs, personnel management, and structural organization in a way that allows them to establish relevant internal functional units and set up systems for the management of CBME. Greater autonomy in financial management should also be given to “Level-2- Autonomous” UMPs so that they are free to make decisions on the rates they pay hospitals for placement and the allowances for their staff who take new roles and duties in the CBME management system. Barrier: Unfavorable legal framework for specialized accreditation Circular 12/2017/TT￾BGDĐT; Circular 04/2016/TT￾BGDÐT Specialized accreditation for medical education should be developed, especially at the program level. A CBME￾orientated quality assurance framework for medical education and standards for the assessment of medical education programs should be developed and promulgated. The establishment of local specialized accrediting agencies for medical education should also be considered. In the meantime, internationally-recognized foreign specialized accreditors in medical education should be officially accepted by relevant state management agencies by adding them to the list of recognized foreign accreditors. Barrier: Low Faculty/student ratio Circular 03/2022/TT￾BGDDT, dated January 18, 2022 The faculty/student ratio in medical education as specified in Circular 03/2022/TT-BGDDT, dated January 18, 2022, is 1:15, which has been commonplace for several years. It is recommended that the ratio is raised gradually to around 1:10 to facilitate CBME. Such an improvement entails revisions of other regulations, such as tuition fees, staff quotas, and the availability of other resources. Enabler: National Medical Council, potential of medical professional occupations and competence standards National Medical Council The formation of the National Medical Council is an enabler of CBME. A key task of the Council could be to build upon the MOH’s promulgated standards for some professions. The council should draft a list of all occupations of medical professionals to be promulgated together with their competency standards. This would serve as the starting point for UMPs to initiate curriculum innovations towards CBME. The assessment for certifying or licensing these professional occupations 75 | IMPACT-MED ACTIVITY REVIEW USAID.GOV BARRIERS/ENABLERS RELEVANT POLICIES/ DEVELOPMENTS HYPOTHETICAL POLICY RECOMMENDATIONS THAT WOULD BENEFIT USAID PROGRAMING should also be based on these competency standards and frameworks. Enabler: Legal framework for Hospital-UMP partnerships Decree 111/2017/ND-CP, dated 05/10/2017 Consensus has not been reached among UMP leaders on the clarity and the issues of implementing the Decree. Further discussions amongst UMPs are recommended. 76 | IMPACT-MED ACTIVITY REVIEW USAID.GOV ANNEX VIII: THAI NGUYEN UMP’S ESP PROGRAM The ESP program consists of seven 30-hour modules, ESP 1 to ESP 7, and covers Year 1 to Year 4. ● ESP 1 is taught in Year 1 and is integrated with Year 1 Basic Sciences (Developmental Biology￾Cell Biology, Human Anatomy, Fundamental Histology, and Fundamental Microbiology-Parasite). ● ESP 2 is taught in Year 2 and is integrated with Module 4 - Hematopoietic System, Module 5 - Cardiovascular System, Module 6 - Respiratory System, Module 7 - Medical Practice 1, and Nutrition-Food Safety. ● ESP 3 is taught in Year 2 and is integrated with Module 8 - Digestive System, Module 9 - Urinary System, Module 10 - Skin-Muscle-Bone-Joint, and Module 11 - Medical Practice 2. ● ESP 4 is taught in Year 3 and is integrated with Module 12 - Endocrine- Reproduction￾Metabolism, Module 13 - Nervous System, and Module 14 - Medical Practice 3. ● ESP 5 is taught in Year 3 and is integrated with Environmental Health, Occupational Health, Symptoms of Internal Medicine, Surgery, and Professionalism 2. ● ESP 6 is taught in Year 4 and is integrated with Internal Medicine 1, Surgery 1, Obstetrics and Gynecology 1, Pediatrics 1, Anesthesia and Resuscitation, and Professionalism 3. ● ESP 7 is taught in Year 4 and is integrated with Diagnostic Imaging, Dermatology, Ophthalmology, Odonto-Stomatology-Dentistry, Otolaryngology, Nervous System, Oncology, Tuberculosis and Lung Disease and Rehabilitation, and Mental Health. United States Agency for International Development Hanoi, Vietnam