USAID.GOV USAID-LMI COMET EVALUATION REPORT | i EVALUATION REPORT FINAL PERFORMANCE EVALUATION OF THE USAID-LMI COMET ACTIVITY July 2019 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Julie Younes, Team Leader, and Amy Porter, Senior Workforce Development Specialist, of Social Impact, Inc. under the Asia Learning, Monitoring, and Evaluation (ALME) Support Project. EDUCATION DEVELOPMENT CENTER ii | USAID-LMI COMET EVALUATION REPORT USAID.GOV CONTENTS ACRONYMS ............................................................................................................................IV EXECUTIVE SUMMARY.........................................................................................................V INTRODUCTION....................................................................................................................1 EVALUATION OVERVIEW................................................................................................................................. 1 PROGRAM BACKGROUND.............................................................................................................................. 1 DESIGN AND METHODS .....................................................................................................3 EVALUATION QUESTIONS............................................................................................................................... 3 DATA COLLECTION........................................................................................................................................... 4 SAMPLING............................................................................................................................................................... 7 ANALYSIS ................................................................................................................................................................ 8 LIMITATIONS ......................................................................................................................................................... 9 FINDINGS AND CONCLUSIONS ................................................................................... 10 APPROACH...........................................................................................................................................................10 TEACHING AND LEARNING.........................................................................................................................15 INSTITUTIONALIZATION...............................................................................................................................21 PARTNERSHIPS....................................................................................................................................................25 NETWORK............................................................................................................................................................28 RECOMMENDATIONS ....................................................................................................... 31 ANNEX A: SCOPE OF WORK ......................................................................................... 34 ANNEX B: DATA COLLECTION PROTOCOLS ......................................................... 46 1. KEY INFORMANT INTERVIEW PROTOCOLS..................................................................................46 2. FOCUS GROUP DISCUSSION PROTOCOLS....................................................................................61 3. DIRECT OBSERVATION PROTOCOLS...............................................................................................67 4. ONLINE SURVEY PROTOCOLS ............................................................................................................70 ANNEX C: LIST OF RESPONDENTS .............................................................................. 78 ANNEX D: DOCUMENTS REVIEWED........................................................................... 84 ANNEX E: DISCLOSURE OF CONFLICT OF INTEREST FORMS........................... 86 ANNEX F: DATA DISAGGREGATION FROM SURVEYS.......................................... 89 USAID.GOV USAID-LMI COMET EVALUATION REPORT | iii TABLES Table 1. Evaluation Questions and Related Themes........................................................................................ 3 Table 2. Data Collection Participants by Type and Sex .................................................................................. 7 Table 3. Importance of soft skills for employment (Student Sex Disaggregation).................................... 89 Table 4. Importance of soft skills for employment (Instructor Sex Disaggregation) ............................... 89 Table 5. Student activity awareness (Student Sex Disaggregation) .............................................................. 90 Table 6. Impact on Teaching Practices (Student Sex Disaggregation) ......................................................... 90 Table 7. Impact on Teaching Practices (Instructor Sex Disaggregation)..................................................... 90 Table 8. Instructor Challenges (Instructor Sex Disaggregation)................................................................... 91 Table 9. Sustainability of Active Learning Methods by Current Instructors (Instructor Sex Disaggregation).......................................................................................................................................................... 91 Table 10. Spread of Active Learning Methods to New Instructors (Instructor Sex Disaggregation).. 91 Table 11. Effects of Regional Participation on Program Adoption by Instructors (Instructor Sex Disaggregation).......................................................................................................................................................... 92 Table 12. Effects of Regional Participation on Program Adoption by Instructors who Trained Elsewhere (Instructor Sex Disaggregation) ............................................................................................................................ 92 Table 13. Effects of Regional Participation on Program Support by Local Government (Instructor Sex Disaggregation).......................................................................................................................................................... 92 FIGURES Figure 1. Map of Data Collection Sites................................................................................................................ 4 Figure 2: Importance of Soft Skills for Employment......................................................................................... 10 Figure 3. Factors for Student Employability ....................................................................................................... 12 Figure 4: Student Activity Awareness.................................................................................................................. 13 Figure 5: Impact on Teaching Practices............................................................................................................... 15 Figure 6: Factors for Improved Teaching............................................................................................................ 17 Figure 7. Student and Instructor Engagement.................................................................................................... 18 Figure 8: Instructor Challenges............................................................................................................................. 19 Figure 9: Adoption of USAID-LMI COMET....................................................................................................... 21 Figure 10: Sustainability of Active Learning Methods in MLCs...................................................................... 23 Figure 11: Results of Partnership Approach....................................................................................................... 26 Figure 12: Sustainability of Partnerships.............................................................................................................. 27 Figure 13: Effectiveness of the Network Responses........................................................................................ 28 iv | USAID-LMI COMET EVALUATION REPORT USAID.GOV ACRONYMS ASEAN Association of Southeast Asian Nations COR Contracting Officer’s Representative DUE University of Economics - University of Danang DU University of Danang EDC Education Development Center EQ Evaluation Question EQUI Evaluation, Quality, Utilization, and Impact ET Evaluation Team FGD Focus Group Discussion HCMUTE Ho Chi Minh City University of Technology and Education HueIC HUST Hue Industrial College Hanoi University of Science and Technology ITC Institute of Technology Cambodia KII Key Informant Interview LANITH Lao National Institute of Tourism and Hospitality LMI-COMET Lower Mekong Initiative Connecting the Mekong through Education and Training MLC Mekong Learning Centers MS2W Mekong Skills2Work MTC Maptaphut Technical College M&E Monitoring and Evaluation MU Mahidol University NUOL National University of Laos PPP Public-Private Partnerships RDMA Regional Development Mission for Asia SI Social Impact, Inc. STEM+AT Science, Technology, Engineering, Mathematics, Accounting, and Tourism ToC Theory of Change US United States USAID USEA United States Agency for International Development University of South East Asia USG United States Government UTYCC University of Technology (Yatanarpon Cyber City) USAID.GOV USAID-LMI COMET EVALUATION REPORT | v Findings and Conclusions: The focused approach of USAID-LMI COMET on soft skills is well￾aligned with labor market needs. The Activity aimed to bolster students’ soft skills to close the gap between skills attained at graduation and those demanded by employers; this resulted in participants and stakeholders identifying a strong link between the Activity and employability. EXECUTIVE SUMMARY This document outlines the results of a final performance evaluation of the United States Agency for International Development (USAID) Regional Development Mission for Asia (RDMA) Activity: USAID Lower Mekong Initiative Connecting the Mekong through Education and Training (USAID-LMI COMET). The evaluation was conducted by Social Impact (SI) over the period of September 2018–April 2019. Implemented by the Education Development Center (EDC), USAID-LMI COMET is a five-year workforce development Activity (2014-2019) to support the Association of Southeast Asian Nations (ASEAN) in improving economic integration and global competitiveness by equipping Lower Mekong youth with the skills they need to succeed in the 21st century marketplace. The Activity works to train youth with market￾driven skills, promote gender-balanced employment, increase technology-based learning in classrooms, and expand the number of workers in Science, Technology, Engineering, Mathematics, Accounting, and Tourism (STEM+AT). The purpose of the evaluation was to assess the efficacy of USAID-LMI COMET’s approach to workforce development. It was conducted a year before the conclusion of the Activity in order to generate findings and identify lessons learned that can inform the design of and further investments in workforce readiness projects in the region. The primary audiences of the evaluation are the USAID Activity manager and other USAID decision-makers, as well as USAID’s implementing partner. This evaluation utilized mixed-methods, including the following: 1) Desk Review; 2) Key Informant Interviews (KII); 3) Focus Group Discussions (FGD); 4) Web-based Survey; and 5) Observation. Purposive as well as snowball sampling were applied to select respondents. The evaluation incorporated a gender￾responsive design that included data collection with men and women; in addition, gender-specific questions were integrated into all KII, FGD and observation protocols. To analyze results, the evaluation team (ET) employed data triangulation with qualitative and quantitative data first analyzed independently, in parallel, and then cross-verified for validity. The themes generated through this analysis were then used to identify and present the key findings highlighted in the report. As with all evaluations, this report must be read with limitations in mind, specifically: 1) Limited timeframe for data collection; 2) Remote fieldwork altering interview dynamics; 3) Non-random sample selection; and 4) Gender imbalance among respondents. The ET worked to mitigate many of these limitations through triangulation; by combining information from multiple sources of data, any one piece of information provided by a respondent would not skew the analysis. The evaluation generated several findings, conclusions and recommendations across five major thematic areas, as well as two cross-cutting themes related to unintended outcomes and successes and challenges: Approach EQ1: To what extent is the Mekong Skills2Work (MS2W) approach meeting the needs of today’s workforce and is the approach likely to meet workforce needs in the next five to ten years in the targeted countries? vi | USAID-LMI COMET EVALUATION REPORT USAID.GOV At the same time, it is difficult to assess USAID-LMI COMET contributions to broader, regional workforce impacts, or predict future effects over the long-term. Given available funding, the approach was implemented through a few select colleges and universities, and not all departments within those. The ET—as well as many participants, who had uneven knowledge of the Activity— therefore could not clearly distinguish which changes were the result of USAID-LMI COMET and what was already happening as part of a larger "active learning" movement. Finally, the evaluation revealed a lack of dedicated initiatives to promote female engagement. Though greater inclusion of females was identified as a critical workforce need, their participation in the Activity was only informally supported. Overall, USAID-LMI COMET could do more to provide opportunities for women in STEM+AT, a field where unequal access persists. Recommendations: a) USAID should continue commissioning labor market assessments or similar studies to inform Activity design and implementation; this assessment helped to ensure that the USAID-LMI COMET approach was well-tailored to labor market needs, specifically the development of soft skills among STEM+AT students. b) Through discussions internally and with implementing partners, USAID must ensure that Activity design aligns with the scope of intended impacts. Increasing workforce readiness in the Lower Mekong is a difficult goal for any regional education initiative to attain in five years; in addition, uneven application of the USAID-LMI COMET approach within participating universities and colleges hindered the possibility of more extensive workforce impacts. Nevertheless, key documents (e.g. MEL Plans) reviewed by the ET describe intended changes to broader workforce development. This resulted in tension between USAID-LMI COMET’s stated goals and what unfolded on the ground. c) USAID-LMI COMET must integrate more comprehensive and targeted initiatives to support female inclusion in STEM+AT. For example, the Activity can incorporate gender considerations into all teaching and learning materials. Additionally, data collection revealed a demand among female participants for presentation, public speaking and writing activities; supplemental programming focused on these and other relevant areas can help ensure the Activity is responsive to this participant group. Teaching and Learning EQ2: To what extent has the MS2W approach affected classroom practices and teaching strategies among instructors within the Mekong Learning Centers (MLCs)? What is the instructors’ level of awareness of, commitment to, and implementation of the approach? EQ3: To what extent has the MS2W approach affected student learning? What is students’ level of awareness, receptivity to, and perceived usefulness of the approach as implemented by their instructors? EQ4: What are, if any, unintended (positive and negative) outcomes on (1) the students, instructors, and administrators that have received direct assistance through the program, and (2) other players within the partner institutions? EQ7: What are key factors for the success and/or the challenges to date? USAID.GOV USAID-LMI COMET EVALUATION REPORT | vii Findings and Conclusions: Improvement in instructor teaching was a key success of USAID￾LMI COMET, due particularly to the high-quality of materials such as the Sourcebook, Professional Development Workshops and Online Training Courses. The Activity helped instructors shift away from traditional rote learning methods towards integrating more active learning into the classroom. Students benefitted from this application of active learning and work-based learning and were highly engaged in their courses despite uneven awareness of the Activity. Nevertheless, instructors faced several barriers to engagement in USAID-LMI COMET, including an increased workload, English-language challenges, and lack of administration incentives to take part in the Activity. Language issues in particular resulted in unintended negative outcomes, as this inhibited the reach of the Activity to new instructors with varying English language abilities—in addition to placing a burden on English-speaking instructors who were asked to translate for others. Overall, these barriers were more prohibitive in some universities and colleges than others, depending on whether instructors felt supported to undergo training and implement the materials. Recommendations: a) USAID should continue to invest in high-quality teacher training materials and activities, specifically the Sourcebook, Online Training Courses and Professional Development Workshops. These were critical to the overall success of the Activity. b) USAID-LMI COMET should dedicate budget towards translation of materials (e.g. application materials, teacher training courses, etc.); these must be accessible in local languages if the Activity aims to be inclusive and to broaden its reach to rural areas. Alternatively, funding could be set aside for additional staff to train and interpret for instructors with low English skills. Findings and Conclusions: Support of the university or college administration is the most critical factor for successful institutionalization (i.e., adoption of the USAID-LMI COMET approach) because it helps mitigate common barriers to engagement such as workload and English-language issues. Long-term, government support for the Activity is also important, as institutes are unlikely to dedicate time and resources to initiatives that do not fit within broader educational directives. Evidence points to sustainability of the MS2W approach, particularly for instructor-focused activities (e.g. the Sourcebook). These have positively impacted teaching and learning, generating enthusiasm for their use. Furthermore, one positive unintended outcome observed by the ET was the potential for growth of the Activity outside of the original STEM+AT scope; if properly managed this could contribute significantly to sustainability. Institutionalization EQ5: What are the key factors that support institutionalization and the sustained implementation of the MS2W approach? What are the key challenges? EQ4: What are, if any, unintended (positive and negative) outcomes on (1) the students, instructors, and administrators that have received direct assistance through the program, and (2) other players within the partner institutions? EQ7: What are key factors for the success and/or the challenges to date? viii | USAID-LMI COMET EVALUATION REPORT USAID.GOV Recommendations: a) When universities and colleges apply to join USAID-LMI COMET, administration alignment should be a key criterion for admission (e.g., Does the institute have written policies on active learning? Are instructors provided incentives to learn about and use these methods?). A thorough review of policies towards active learning during the application stage will help ensure selection of institutes that support implementation of USAID-LMI COMET activities. b) USAID and EDC should continue to support expansion and marketing of the Activity to additional academic departments outside of STEM+AT. As this expansion occurs, USAID-LMI COMET should be intentionally marketed to raise awareness of activities and to distinguish from—or potentially find synergies with—other active learning initiatives. c) For future initiatives, USAID and the implementing partner must aim to work more closely with government from the beginning. Though partnering with governments is complex and can aggravate political issues, USAID-LMI COMET and similar programs cannot affect long-term change in education systems without engagement from the Ministry of Education or similar institutions. Findings and Conclusions: Private sector partnerships are a central feature of USAID-LMI COMET, as they create important opportunities for students to engage in internships and work-based learning. Although partners had uneven knowledge of USAID-LMI COMET, for administrators and instructors the Activity improved the management of these relationships—particularly for local industry. However, apart from some cases of technology donations by multinational companies, USAID-LMI COMET has not significantly affected use of technology in the classroom. One key challenge identified by the ET centered on private sector partners’ uneven knowledge of the Activity. This resulted in a missed opportunity to build connections among various local industry partners, and between local and multinational companies. Overall, relationships between educational institutions and local industry partners are deeper (i.e., they have more frequent, direct contact and take part in ongoing activities) and therefore are more sustainable than those with multinational companies. Partnerships EQ6: To what extent has the partnerships approach – including both multinational and local, and both personal engagement with companies and the use of online toolkits – been effective in strengthening the relationships between industry and institutions to increase the use of technology in the classroom and increase work-based learning (WBL) opportunities, so as to close the gap between demand and supply in skilled human resources in the region? EQ 7: What are key factors for the success and/or the challenges to date? EQ8: What evidence exists that the partnerships will likely be sustained – both at the multinational and at the local level? USAID.GOV USAID-LMI COMET EVALUATION REPORT | ix Recommendations: a) USAID should continue to invest in effective local industry partnership activities, specifically the work-based learning toolkit and Industry Advisory Committees; along with teacher training materials, these represented a key success of USAID-LMI COMET. b) USAID and EDC should hire additional dedicated staff to help MLCs manage complex multinational and government partnerships. Relationships with these stakeholders were generally owned by USAID and EDC rather than individual institutes, limiting long-term sustainability. Additional staff can focus on connecting universities and colleges with government and multinational partners, and on building relationships between various partners. Findings and Conclusions: For instructors, the network provided an important space to share best practices and advice. However, the key value-add was not necessarily that the instructors came from different countries, but rather that they all taught in universities and colleges implementing USAID-LMI COMET. Thus, the same benefits of regional networking can also likely be achieved through in-country interactions. The network was not a major driver of administrator and partner participation; when most administrators and partners first engaged in the Activity, they were unaware or marginally aware of the network. It also did not impact reach of the program to rural or hard-to-access areas. USAID-LMI COMET favored urban institutions during the selection process (i.e., through competitive procurement that did not factor in location or institute type); by the time of this evaluation, the Activity had not expanded beyond the original 12 institutes. This limited the potential reach of the network. Recommendations: a) USAID must amend or introduce alternatives to the competitive procurement process in order to reach a more diverse range of institutes. For example, USAID can set concrete targets for accepting a certain number of institutes in rural or hard-to-access areas, and provide support during the application process (e.g., translation support or making materials available in local languages). b) USAID-LMI COMET should transition resources dedicated to the regional network towards strengthening national networks in each participating country: Many of the benefits of the USAID￾LMI COMET network—namely, the sharing of best practices with colleagues—can be attained through in-country activities. This is both cost-effective and eliminates potential language barrier issues. Network EQ9: To what extent has the Network been – or is likely to be – effective in: a. Having early adopters --including instructors and institutions-- teach/support later adopters? b. Broadening its reach across the region, including in harder-to-reach areas? c. Gaining the buy-in and support of (1) the MLC administration and (2) the government? 1 | USAID-LMI COMET EVALUATION REPORT USAID.GOV INTRODUCTION EVALUATION OVERVIEW In August 2018 the United States Agency for International Development (USAID) Regional Development Mission for Asia (RDMA) contracted Social Impact, Inc. (SI) to conduct a final performance evaluation of the USAID Lower Mekong Initiative Connecting the Mekong through Education and Training (LMI COMET), implemented by the Education Development Center (EDC). PURPOSE This final performance evaluation of USAID-LMI COMET examines overall performance and effectiveness of the Activity from 2014-2018. The purpose of the evaluation is to assess the efficacy of the programmatic approach in introducing innovative classroom practices, positioning students for new or better employment opportunities, aligning with appropriate labor market needs, addressing key challenges in skill competency development, and initiating partnerships with businesses. The evaluation was conducted a year before the Activity concludes to generate findings and identify lessons learned that can inform the design of and further investments in workforce development projects in the region. AUDIENCE The primary audiences of this evaluation are the USAID Activity manager and other USAID decision￾makers, as well as USAID’s implementing partner. Other audiences may include educational institutions in the region, the private sector, host country governments, other USAID Missions implementing or planning workforce development programming, and other international development donors and practitioners. EVALUATION TEAM The SI evaluation team (ET) consisted of three core team members, research and logistical support staff, and home office evaluation management staff. Julie Younes served as Team Leader responsible for overall management of the evaluation and production of deliverables. She has a decade of design, monitoring and evaluation experience focused on youth development and education programming. Dr. Amy Porter served as the Senior Workforce Development Specialist, using her subject matter expertise to contribute significantly to the desk review, evaluation design, and data collection tools, as well as data analysis and reporting. Senior Evaluation Specialist Pheak Young led data collection efforts in Cambodia and contributed to document review, evaluation design and data collection tools. The team received support from SI headquarters by Project Director Dr. Andrew Epstein, Project Manager Catherine Villada, Project Assistant Justin Archangel, and Gender Specialist Noshaba Zafar. In-country translators and note-takers in Thailand and Vietnam also supported the ET during data collection. PROGRAM BACKGROUND USAID-LMI COMET is a five-year (2014-2019) workforce development Activity to support the Association of Southeast Asian Nations (ASEAN) in improving economic integration and global competitiveness by equipping Lower Mekong youth with the skills they need to succeed in the 21st century marketplace. Managed by EDC, USAID-LMI COMET works to train youth with market-driven skills, USAID.GOV USAID-LMI COMET EVALUATION REPORT | 2 promote gender-balanced employment in key sectors, increase technology-based learning in classrooms, and expand the number of workers in Science, Technology, Engineering, Mathematics, Accounting, and Tourism (STEM+AT). In addition to hard skills, employers in Southeast Asia have identified non-technical (e.g. English) and soft skills (e.g. leadership, teamwork) to be significant workforce capability gaps that prevent businesses from keeping pace with rapid economic growth experienced elsewhere.1 USAID-LMI COMET aims to bridge this gap by building linkages between education providers and employers through public-private partnerships (PPPs). These include collaboration with local industry, government, and multinational institutions (Cisco, Intel, Texas A&M, and Amazon Web Services) to leverage cutting edge educational approaches and technologies, and to facilitate educational opportunities in the Lower Mekong countries while improving workforce capacity. Mekong Learning Centers (MLCs) are a network of universities and post-secondary vocational colleges that collaborate with local industry partners to equip youth with highly sought-after workplace skills including soft skills, technology, and hands-on work. There are 12 MLCs that make up the USAID-LMI COMET Mekong Skills2Work (MS2W) network, of which 10 are currently active (the Laos National Institute of Tourism and Hospitality [LANITH] moved on from the project in 2017, and Maptaphut Technical College [MTC] in 2018). The MLCs are located across Cambodia, Lao PDR, Myanmar, Thailand and Vietnam. As of the end of year four, the Activity has achieved the following key results.2 ● 820 vocational school and university instructors trained in teaching in-demand skills in STEM+AT fields as a result of USG assistance ● Over 60,000 students reached as a result of USAID-LMI COMET trainings ● Almost 600 user registrations on the MS2W network portal, and over one million user interactions on the MS2W network portal and social media sites 1 “Leading in the new world of work: Southeast Asia Human Capital Trends 2015.” Deloitte Consulting, Southeast Asia. 2 “Thriving in the Fourth Industrial Revolution: USAID-Lower Mekong Initiative Connecting the Mekong through Education and Training Annual Report, October 2017-2018.” USAID. 3 | USAID-LMI COMET EVALUATION REPORT USAID.GOV DESIGN AND METHODS EVALUATION QUESTIONS The evaluation explored five key thematic areas, which reflect the core components of USAID-LMI COMET: 1) Program approach; 2) Teaching and learning; 3) Institutionalization; 4) Partnerships; and 5) Network. Additionally, two cross-cutting themes related to unintended outcomes and successes and challenges were examined. This report addresses nine evaluation questions (EQs) developed by USAID in consultation with EDC: TABLE 1. EVALUATION QUESTIONS AND RELATED THEMES EQ1 To what extent is the MS2W approach meeting the needs of today’s workforce and is the approach likely to meet workforce needs in the next five to ten years in the targeted countries? Program approach EQ2 To what extent has the MS2W approach affected classroom practices and teaching strategies among instructors within the MLCs? What is the instructors’ level of awareness of, commitment to, and implementation of the approach? Teaching and learning EQ3 To what extent has the MS2W approach affected student learning? What is students’ level of awareness, receptivity to, and perceived usefulness of the approach as implemented by their instructors? EQ5 What are the key factors that support institutionalization and the sustained implementation of the MS2W approach? What are the key challenges? Institutionalization EQ6 To what extent has the partnerships approach – including both multinational and local, and both personal engagement with companies and the use of online toolkits – been effective in strengthening the relationships between industry and institutions to increase the use of technology in the classroom and increase work￾based learning (WBL) opportunities, so as to close the gap between demand and supply in skilled human resources in the region? Partnerships EQ8 What evidence exists that the partnerships will likely be sustained – both at the multinational and at the local level? EQ9 To what extent has the Network been – or is likely to be – effective in: a. Having early adopters --including instructors and institutions-- teach/support later adopters? b. Broadening its reach across the region, including in harder-to-reach areas? c. Gaining the buy-in and support of (1) the MLC administration and (2) the government? Network EQ4 What are, if any, unintended (positive and negative) outcomes on (1) the students, instructors, and administrators that have received direct assistance through the program, and (2) other players within the partner institutions? Cross-cutting EQ7 What are key factors for the success and/or the challenges to date? USAID.GOV USAID-LMI COMET EVALUATION REPORT | 4 DATA COLLECTION This evaluation utilized a mixed-methods approach that included the following major types of data collection tools: 1) Desk Review; 2) Key Informant Interviews (KII); 3) Focus Group Discussions (FGD); 4) Web-based Survey; and 5) Observation. Fieldwork was conducted over the period of October 26th, 2018–January 31st, 2019. This included in￾person KIIs, FGDs and observations from October 26th–November 16th, as well as remote KII data collection from November 16th–December 1s2th and web-based surveying from December 12th—January 31st. Data were collected at nine MLCs across the five participating countries, with piloting held at Mahidol University (see Figure 1 below). For KIIs, FGDs, observations and surveying, the ET captured disaggregated responses of female, male, and other respondent types to identify how program activities may have affected skill building and career development differently. Informed consent was obtained from all respondents prior to data collection. From October 30th–November 16th the ET visited Hue Industrial College (HueIC), Ho Chi Minh City University of Technology and Education (HCMUTE), and the University of Economics–The University of Da Nang (DUE) in Vietnam. Data were also collected in-person at: Institute of Technology Cambodia (ITC) in Phnom Penh, Cambodia; University of Southeast Asia (USEA) in Siem Reap, Cambodia; Maptaphut Technical College (MTC) in Rayong, Thailand; and Mahidol University in Nakhon Pathom, Thailand, where all tools were piloted. Remote data collection subsequently took place at the National University of Laos (NUOL) in Vientiane and the University of Technology Yatanarpon Cyber City (UTYCC) in Myanmar (November 16th–December 12th). Prior to the start of data collection, all in-person and remote facilitators Figure 1. Map of Data Collection Sites Data collection was held in-person at MLCs in Vietnam, Cambodia and Thailand. Remote data collection took place in Myanmar and Lao PDR. Mahidol University Hue Industrial College The University of South-East Asia Institute of Technology Cambodia Maptaphut Technical College Ho Chi Minh City University of Technology Education The University of Economics—The University of Da Nang University of Technology Yatanarpon Cyber City National University of Laos 5 | USAID-LMI COMET EVALUATION REPORT USAID.GOV were trained on tools and procedures by the ET Team Lead. Further detail on data collection tools utilized for the evaluation is provided below. DESK REVIEW Prior to fieldwork, the ET conducted a desk review of key program and research documents. The review provided background knowledge on the processes and decisions that went into designing the USAID-LMI COMET Activity, as well as up-to-date information on individual country program outcomes, progress and results. Documents reviewed include monitoring and evaluation (M&E) plans and monitoring data collected, annual reports, labor market assessments, and applicable social media coverage or illustrative participant feedback received and documented by the program (see Annex D: Documents Reviewed for a full list of documents read). In reviewing all documents, the ET worked to capture any specific language regarding critical assumptions or considerations for women or vulnerable populations in project design, strategy, or monitoring to better understand the Activity’s approach to inclusivity. KEY INFORMANT INTERVIEWS The ET spoke with key informants using semi-structured interview protocols tailored to respondent type. A total of 65 KIIs were held with a variety of USAID-LMI COMET participants, partners and stakeholders, including: 1. Students and alumni (12 KIIs conducted); 2. Administrators (10); 3. Instructors, including those responsible for technical support (16); 4. Local industry partners (12); 5. Government and multinational partners (10); and 6. EDC and USAID staff (five). KIIs were conducted individually, except when the ET was requested to hold small group interviews (these requests were for logistical reasons to accommodate participant’s schedules). Of the 65 total KIIs conducted, 56 were individual interviews, and nine were conducted through small group interviews. Interviews were held across a total of nine MLCs, including 18 in Vietnam, 12 in Thailand, 10 in Cambodia, 10 in Lao PDR, seven in Myanmar, and eight with internationally based participants. In Thailand and Vietnam, KIIs were held in-person and conducted in teams of three: one ET member facilitating the interview, one taking notes, and one translator. In Cambodia, a two-person team of a facilitator and note￾taker conducted all KIIs in-person (a translator was not required as the ET members spoke the local language). In Lao PDR and Myanmar, local facilitators conducted interviews over the phone (KIIs were in￾person in a few cases where the facilitator lived close to the interviewee). For all interviews, audio was recorded electronically with consent from the participant and for sole use by the ET to ensure backup in case the ET needed to later revisit or elaborate on notes taken. In Lao PDR and Myanmar, where one local facilitator conducted KIIs, recordings were revisited after the conclusion of the interview and used to take detailed notes. USAID.GOV USAID-LMI COMET EVALUATION REPORT | 6 FOCUS GROUP DISCUSSIONS The ET conducted 12 total FGDs with instructors at seven MLCs in Thailand (two FGDs held), Vietnam (six FGDs) and Cambodia (four FGDs). FGDs were not held in Lao PDR or Myanmar, as this tool is not well-suited to remote data collection. As with KIIs, semi-structured discussion protocols were used to guide the FGDs, which were led by a three-person team (facilitator, note-taker, and translator) in Thailand and Vietnam and a two-person team (facilitator and note-taker) in Cambodia. FGDs were also electronically recorded as a back-up for note-takers with the consent of all participants. All FGDs utilized participatory methods, with participants engaging in an “outcome mapping” exercise where they selected which USAID-LMI COMET activities best link to achievement of various outcomes. Please see Annex B: Data Collection Protocols for further information on FGD methodology. Of the 12 FGDs held, seven took place with instructors and five with those who had obtained the level of “Master” or “Lead” USAID-LMI COMET instructor. FGDs included a mix of male and female participants, as well as instructors from different departments and subject areas. OBSERVATIONS ET members observed classrooms using a structured observation protocol, focusing specifically on teaching and student engagement. The protocol included prompts to note gender- or group-specific behaviors, in order to capture any differences in implementation and outcomes among various participants. Following observations, the ET pulled aside three to five students to ask questions on the use and effect of active learning methods. In total, the ET conducted nine classroom observations across six MLCs in Thailand (one observation held), Vietnam (five) and Cambodia (three). WEB-BASED SURVEY A survey was administered to further capture information from participants at MLCs. Two separate surveys for students and instructors were developed and translated into five local languages: Thai, Vietnamese, Khmer, Lao, and Burmese. The student survey was distributed to those registered on the MS2W Portal; contact information for instructors was obtained from an EDC Activity registration list. This tool was distributed to respondents across all 12 MLCs (the two currently inactive MLCs, which have students registered on the MS2W Portal, were part of this sample), including those not visited for qualitative data collection. However, survey participation was voluntary and thus there were zero or low response rates for some MLCs; for the instructor survey, the ET did not receive responses from UTYCC and NUOL, while student response rates were low at ITC, LANITH, MTC, DUE, and USEA. The surveys were administered via SurveyMonkey and included questions on active learning strategies, student learning, the MS2W regional network, and MLC adoption of the USAID-LMI COMET approach. GENDER CONSIDERATIONS IN DATA COLLECTION To ensure confidentiality and safety for participants, especially women, the ET conducted all KIIs and FGDs in private rooms with no other participants or MLC staff present. In addition, for each line of questioning (e.g. student learning, classroom practices, regional network, etc.) KII and FGD respondents were asked to consider the differing needs and experiences of males and females, as well as of those in different subject areas or departments (e.g. engineering, tourism, etc.). All protocols were reviewed by a Gender Specialist prior to fieldwork to ensure gender considerations were appropriately addressed. 7 | USAID-LMI COMET EVALUATION REPORT USAID.GOV SAMPLING The ET applied purposive sampling to identify KII participants, prioritizing interviewees according to the following criteria: ● Identified as key informants by USAID and/or EDC; ● Activity participation across a range of types (e.g. student, partner, etc.); ● Knowledge of USAID-LMI COMET and its activities; ● Differing range of MLC experiences in the Activity (as determined during the desk review); and ● MLC location and type (university versus college or vocational institute). A total of 79 respondents took part in the 65 KIIs across nine MLCs in Thailand, Cambodia, Vietnam, Lao PDR, and Myanmar. The 79 respondents included 15 students or alumni, 18 instructors, 11 administrators, eight USAID and EDC staff, and 27 local industry, government, or multinational partners. Snowball sampling (i.e., the ET finds participants via recommendations from others taking part in data collection) was used to select FGD participants. At each MLC, the ET worked with administrative points of contact provided by EDC to recruit instructors participating in USAID-LMI COMET, including those certified as “Master” or Lead” instructors. A total of 65 respondents across seven MLCs, including 20 Master or Lead instructors and 45 instructors, took part in 12 FGDs. As previously noted, FGDs were not included in remote data collection and thus did not take place in Lao PDR and Myanmar. During fieldwork, the ET coordinated with MLC points of contact to schedule observations of classroom sessions on a range of academic subjects (e.g. electrical engineering, tourism and hospitality, etc.). A total of 278 students were present in the nine classes visited. Finally, 178 respondents completed the web-based survey, including 129 students and 49 instructors across the five participating countries. However, not all respondents filled out each survey question; sample sizes for individual survey questions are thus provided alongside reported results. Table 2 below details the sample size for each data collection tool by respondent type and sex. In total, 600 individuals including 422 students, 132 instructors and Master instructors, 11 administrators, 27 private sector and government partners, and eight USAID and EDCs stakeholders took part in data collection. Of these 600, 207 were females, 389 were males, and four chose not to disclose sex. TABLE 2. DATA COLLECTION PARTICIPANTS BY TYPE AND SEX TOOL RESPONDENT TYPE AND SEX (MALE/FEMALE/DID NOT DISCLOSE) TOTAL Student/ Alumni Instructor Master Instructor Administrator Local Industry Partner Government/ Multinational Partner USAID/ EDC M F DND M F M F M F M F M F M F KII 10 5 N/A 16 2 N/A 7 4 13 4 7 3 2 6 79 FGD N/A 27 18 16 4 N/A N/A N/A N/A 65 Observations 196 82 N/A N/A N/A N/A N/A N/A N/A 278 Survey 66 59 4 29 20 N/A N/A N/A N/A N/A 178 Total 272 146 4 72 40 16 4 7 4 13 4 7 3 2 6 600 USAID.GOV USAID-LMI COMET EVALUATION REPORT | 8 As detailed in Table 2 above, approximately 35% of data collection respondents were female and 65% male, respectively. Although the ET aimed to achieve a more balanced male to female respondent ratio, this was not possible due to the structure of USAID-LMI COMET. The Activity focuses on STEM+AT faculties, which are notoriously gender un-balanced, and as a result includes more male than female participants. For example, in classrooms observed by the ET there were 82 female and 196 male students; the gender balance of the evaluation is reflective of the program at the time of the evaluation. ANALYSIS Prior to fieldwork, the ET reviewed key program documents and organized relevant information into a summary Excel spreadsheet. For qualitative primary data collected during fieldwork, the ET conducted iterative analysis, holding internal working sessions to triangulate new data and discuss emerging findings. Raw notes were first input into an aggregated Excel spreadsheet and then organized in a matrix categorizing findings, conclusions, and recommendations by EQ. Following fieldwork, the ET reviewed all data from the desk review, KIIs, FGDs, and observations to identify response categories and patterns, which were then used to generate a codebook. All primary qualitative data were entered and coded using the analysis software Dedoose. Qualitative data were disaggregated by stakeholder type, sex, age, country, institution, role, and location to capture any divergences. If differences were found, these are presented in the relevant findings sections; for sex specifically, disaggregated data are included in Annex F: Data Disaggregation from Surveys but are discussed in the findings only when variations in results were seen. Data received from the web-based survey were analyzed using Excel to tabulate responses, produce descriptive statistics, and disaggregate along the same lines as the qualitative data. The ET employed data triangulation, a data analysis strategy in which qualitative and quantitative data are first analyzed independently, in parallel, and then cross-verified for validity. This allowed the ET to thoroughly assess which elements of USAID-LMI COMET contributed to the achievement of expected outcomes, which elements did not, and most importantly, why. The themes generated through this analysis were used to identify and present the key findings highlighted in the evaluation report. In reporting qualitative findings from KIIs and FGDs in the sections below, the ET discusses both the frequencies and range of the ideas, themes, and topics produced by respondents. In order to make clear that these frequencies are not representative of the broader population, data are commonly presented in raw numbers out of the total number of KIIs and FGDs conducted (77, including 65 KIIs and 12 FGDs). The exception to this is reporting of USAID-LMI COMET activity selection results from the FGD outcome mapping exercise (see “Focus Group Discussions” above). In discussions of these findings, individual participant voting data are presented. This approach for analysis and reporting was taken in response to the design of the evaluation, which included both in-person and remote data collection that necessitated holding KIIs and FGDs with the same respondent groups (specifically, instructors participated in FGDs at locations of in-person data collection, while KIIs were held with instructors at remote data collection sites). Thus, findings from KIIs and FGDs are reported together, with triangulation occurring among different respondent groups (instructors versus administrators, industry partners, etc.), and with survey, observation and desk review data. 9 | USAID-LMI COMET EVALUATION REPORT USAID.GOV LIMITATIONS Main limitations to the evaluation include: ● Timeframe: The timing for the evaluation, specifically the data collection period, was compressed to avoid conflicts with end of semester exams. This necessarily limited the amount of data collection that could be conducted at each MLC; however, the ET aimed to construct a sample that included a variety of locations, roles, types, etc., to reflect the range of USAID-LMI COMET participants. ● Remote fieldwork: KIIs in Lao PDR and Myanmar were conducted remotely, over the phone or Skype. This can alter the dynamic of the interview, as the ET was unable to ascertain body language and other non-verbal factors during KIIs. Additionally, FGDs and observations were not held in Lao PDR and Myanmar, limiting the sample size and representation for these tools. The selection of a sample that included a wide variety of locations, roles, types, etc., of data sources helped to mitigate the limits imposed by remote fieldwork. ● Sampling: Sample selection was based on consultation with EDC and key points of contact within MLCs. This can lead to selection bias, as stakeholders might be more likely to guide evaluators towards those who have had positive experiences with the Activity. The ET took steps to mitigate this bias by using desk review findings to ensure that MLCs that have experienced both successes and challenges in institutionalizing the MS2W approach were included in data collection. The ET also used triangulation to combat selection bias; by combining information from multiple sources of data, any one piece of information would be balanced against findings and analysis across multiple sources. ● Lack of testing data: This evaluation does not include test results, rather relying on participant self￾reporting of learning outcomes. To address any biases this may create, the ET employed rigorous qualitative coding and triangulation of data between various respondent groups. USAID.GOV USAID-LMI COMET EVALUATION REPORT | 10 FINDINGS AND CONCLUSIONS APPROACH EQ1 To what extent is the MS2W approach meeting the needs of today’s workforce and is the approach likely to meet workforce needs in the next five to ten years in the targeted countries? FINDINGS The focused approach on soft skills is well-aligned with labor market needs. USAID-LMI COMET participants and stakeholders noted that while students are receiving adequate training in their technical fields, it is also critical that they learn soft skills—teamwork, communication, leadership, public speaking, problem solving, time management, etc. The importance of soft skills was highlighted in 54 of 77 (70%) KIIs and FGDs conducted with instructors, administrators, students, partners, and EDC and USAID staff (of these 77, 65 were KIIs and 12 FGDs, with responses given in each equally weighted). Non￾technical (e.g., English, IT) and hard skills were mentioned in 17 of 77 (22%) and 12 of 77 (15%) of discussions, respectively. Similarly, when asked what skills are most important for finding and keeping a job over the long-term, both student and instructor survey respondents indicated a variety of soft skills (see Figure 2). For qualitative data no discernible differences between male and female responses were observed. However, in the survey given to students, significantly more females (65%, or 26 of 40 female respondents) noted the importance of presentation skills and public speaking versus male respondents (41%; 18 of 44 male respondents). Writing skills were also more frequently emphasized by females (mentioned in 38% of female surveys, or 15 of 40, versus 14%, or six of 44, male surveys). This aligns with longstanding feminist theories of women being presented in subordination to others3 and women’s socialized belief that they need to present themselves in particular ways4–thus leading to an interest in learning to present well in public spaces and in writing. Table 3 and Table 4 in Annex F provide further information on sex-disaggregated student and instructor responses for this question. Despite the necessity of soft skill development, this is often missing from student’s educational training. One male instructor in Thailand reflected that school projects teach the basics of teamwork, but that 3 See, for example: Beauvoir, S.D. (1974). The Second Sex, New York: Vintage Books. 4 See, for example: Goffman, E. (1979). Gender Advertisements. London: Macmillan. Figure 2: Importance of Soft Skills for Employment 21 29 26 43 44 42 56 57 66 25 28 34 30 37 33 39 40 40 Writing skills Ability to use social media Networking and job search skills Technical skills for the position Presentation and public speaking Leadership English language skills Communication Teamwork and/or ability to… Teamwork, communication and English were highlighted by students and instructors as critical to career development # of selections; participants able to choose multiple responses Students (n=85) Instructors (n=43) Teamwork/groupwork 78% 93% 67% 93% 66% 91% 49% 77% 52% 86% 51% 70% 31% 79% 34% 65% 25% 58% 11 | USAID-LMI COMET EVALUATION REPORT USAID.GOV students are working “with other engineering students, not people at different levels–management, workers, etc.… those are the skills they need.” Students similarly felt that soft skills were important because with skills like leadership “we have more of a chance to be promoted” (Student, Cambodia). These findings were confirmed in a 2016 labor market assessment, which supports that generally, technical skills of graduates are acceptable to employers while soft skills are lacking. 5 Soft skill development leads to greater employability, a key concern for students, instructors and employers. Employability is targeted through a variety of USAID-LMI COMET programmatic features, including improving instructor teaching methods; better student learning; industry partners’ assistance in developing relevant curriculum; and providing work-based learning opportunities for students. Improved employability of students was explicitly discussed in 55% (42 of 77) of KIIs and FGDs conducted by the ET. Of these 42, 22 (52%) linked employability to improved student learning, 12 (29%) to improved teaching practices, nine (21%) to industry input into curriculum, and four (9%) to better work-based learning opportunities. Instructor FGD and KII participants praised the USAID-LMI COMET approach, saying it “helps students to get the right work because they learn through site visits, work experience, and internships” (Instructor, Thailand). Employability was also discussed frequently (in 12 of 22, or 56%, of partner FGDs and KIIs) by local industry, government and multinational partners. These respondents noted the benefits of hiring students taught by USAID-LMI COMET instructors, who adopt methods for developing soft skills in their teaching. For some companies with long-standing relationships with local colleges and universities, the need to re￾train new hires has historically been a concern. For example, one local industry partner described how new hires often do not have the teamwork and problem-solving skills needed to perform their jobs well; this employer therefore appreciated that USAID-LMI COMET equips students with soft skills while they are still in school. Hiring recent USAID-LMI COMET graduates, who need less re-training on problem solving and teamwork versus other entry-level workers, has saved his company time and money. Likewise, among instructors surveyed “students graduate with the skills needed to find a job” was cited frequently (40 mentions out of 43 respondents to this question) as a key objective of USAID-LMI COMET, second only to “improved teaching methods” (43 mentions among 44 instructor surveys). Employability was also noted as an important issue among student survey respondents, the majority of whom feel somewhat prepared to find a job but desire more technical, non-technical and soft skills training. Figure 3 details student survey responses regarding work readiness, disaggregated by male (M) and female (F) participants. Thus, while the evaluation was not designed to measure changes in longer-term youth employment outcomes, both qualitative and quantitative data from administrators, instructors, partners, students and other stakeholders show that employability is a critical component of the USAID-LMI COMET approach. 5 EDC (2016). Regional STEM+AT Labor Market Report, USAID. USAID.GOV USAID-LMI COMET EVALUATION REPORT | 12 Without dedicated activities to promote female engagement in STEM+AT, USAID-LMI COMET could not address this specific workforce need. Though greater inclusion of females was identified in the Labor Market Reports as a critical workforce need, the evaluation did not find evidence of systematic USAID-LMI COMET curricula specifically designed to promote their engagement. Participation of females was only informally supported; for example, teams were “strongly encouraged” to have gender diversity for the “Innovation Challenge,” an international competition hosted by USAID-LMI COMET that brought together students to present their unique innovation concepts to solve complex global issues. According to EDC, the Activity “looked at ways [it] could partner with groups, entities, companies, individuals, etc., that had gender programs, or provided support for females in the workplace, and their inclusion especially in STEM+AT” (EDC, Thailand). However, the ET found little evidence of gender-specific programming as reported by MLCs. While USAID-LMI COMET endeavored to support women who were already present in the teaching and learning spaces in which the program operated, the reality of this context is one of unequal access; for example, there were significantly more male (196) than female (82) students present in classrooms observed by the ET. Furthermore, misconceptions on women in STEM+AT persist: “Male students prefer to go out for field trips because they are more curious. Nowadays, [there are] more women in [STEM] but still less than males” (Student, Vietnam). Absent specific initiatives, the program did not affect current gender dynamics in MLCs, either in terms of enrollment numbers or increasing opportunities for women. Since local industry partners, MLCs, instructors and students were not pushed by the Activity to consider and adapt their practices to ensure women were afforded equal consideration across all aspects of school and work (i.e. from acceptance into STEM+AT departments, to work-based learning opportunities, to Figure 3. Factors for Student Employability USAID-LMI COMET students feel “somewhat prepared” to find a job in their desired field, but still require soft skill, technical and non-technical skill-building (n=86) M, 11 F, 8 22% (19/86 respondents) feel "completely prepared" M, 19 F, 15 40% (34/86) feel somewhat prepared, but would like to learn more soft skills M, 13 F, 14 No … 33% (28/86) feel somewhat prepared, but would like to learn more technical skills M, 14 F, 13 31% (27/86) feel somewhat prepared, but would like to learn more non-technical skills M, 5 F, 2 8% (7/86) do not feel prepared at all Did not respond, 1 13 | USAID-LMI COMET EVALUATION REPORT USAID.GOV hiring practices in industry, etc.), there was a default to existing gender norming rather than a space where different realities might be considered. Less evidence for broader, long-term impact on workforce readiness. The stated goal of USAID-LMI COMET is to “improve the employability of youth in the Lower Mekong partner countries by strengthening the skillsets of current and upcoming workers in STEM+AT.”6 However, the Activity reached 12 universities and vocational colleges in total across the five targeted countries, with 10 of these continuing to engage in the program at the time of the evaluation. Within the 10 MLCs, the Activity was not implemented in all academic departments. Thus, the ET found uneven application of the MS2W approach. In several MLCs, USAID-LMI COMET was part of a larger trend towards active learning; as a result, many students, industry partners, and even some instructors could not distinguish what activities were implemented through USAID-LMI COMET versus what came from other administrative or government initiatives, and could not identify direct linkages between the Activity and changes to employability at a regional level. Among students surveyed for the evaluation, 41% (41 of 99 total responses to this question) reported “no awareness” of the Activity (see Figure 4 and Table 5 in Annex F for sex￾disaggregated results). Given this and the above factors, it was difficult for the ET to identify specific programmatic contributions to overall workforce readiness in the Lower Mekong. Though data collection revealed positive impacts on instructor teaching and student learning (see “Teaching and Learning”), there is not yet enough evidence to determine whether USAID-LMI COMET would have the same results at a broader scale and over a longer period of time. CONCLUSIONS Based on the above findings, the ET concludes that the USAID-LMI COMET approach can be categorized into the following: • USAID-LMI COMET is highly responsive to labor market needs. Specifically, it endeavors to bolster students’ soft skills to close the gap between skills attained at graduation and those demanded by employers. This focus on soft skills resulted in participants and stakeholders identifying a strong link between the Activity and employability. 6 EDC (2017) Connecting the Mekong Through Education and Training, Monitoring and Evaluation Plan, USAID. 16% 41% 42% Very aware No awareness Some awareness 41% of students surveyed are not aware of the USAID-LMI COMET program (n=99) Figure 4: Student Activity Awareness (42/99) (41/99) (16/99) USAID.GOV USAID-LMI COMET EVALUATION REPORT | 14 • While USAID-LMI COMET engaged females who were “already in the room,” it did little to create new opportunities that might have encouraged greater access to opportunities in STEM+AT. Gender￾specific approaches were ad hoc, and when incorporated, relied on partnering with other organizations. Overall, the program could do more to support increased engagement of and opportunities for women in STEM+AT. • It is difficult to identify any direct linkages between USAID-LMI COMET and broader, regional-level workforce impacts or predict future effects over the long-term. The Activity approach was implemented through a select few colleges and universities, and not all departments within those. Therefore, it was difficult to distinguish—for the ET as well as for many participants, who had uneven knowledge of the program—which changes were the result of USAID-LMI COMET and what was already happening as part of a larger "active learning" movement. 15 | USAID-LMI COMET EVALUATION REPORT USAID.GOV TEACHING AND LEARNING EQ2 To what extent has the MS2W approach affected classroom practices and teaching strategies among instructors within the MLCs? What is the instructors’ level of awareness of, commitment to, and implementation of the approach? EQ3 To what extent has the MS2W approach affected student learning? What is students’ level of awareness, receptivity to, and perceived usefulness of the approach as implemented by their instructors? EQ4 What are, if any, unintended (positive and negative) outcomes on (1) the students, instructors, and administrators that have received direct assistance through the program, and (2) other players within the partner institutions? EQ7 What are key factors for the success and/or the challenges to date? FINDINGS Improvement in instructor teaching was a key success of USAID-LMI COMET. In 84% (65 of 77) KIIs and FGDs with instructors, administrators, students, partners, EDC and USAID, respondents noted the positive effects of USAID-LMI COMET on instructor teaching. Changes cited include greater use of learner-centered methods (e.g., group work, student presentations, etc.), as well as better organization of industry visits and interactions with local companies. Instructors described the program’s methodology as “new and different from traditional teaching. It puts the instructor more in the facilitator position” (Instructor, Myanmar). Survey participants confirmed these findings, with the majority (notably 70%, or 31 of 44 instructor respondents) indicating that USAID-LMI COMET positively impacted teaching and classroom practices (see Figure 5). Table 6 and Table 7 in Annex F provide further information on sex-disaggregated student and instructor responses for this question. 0% 3% 3% 18% 28% 49% 5% 5% 2% 0% 18% 70% Negatively impacted Somewhat negatively impacted Neutral/no impact Unsure Somewhat positively impacted Positively impacted 70% of USAID LMI-COMET instructors surveyed agreed that the Activity positively impacted teaching and classroom practices Instructors (n=44) Figure 5: Impact on Teaching Practices Students (n=68) (38/68) (31/44) (19/68) (8/44) (12/68) (0/44) (2/68) (1/44) (2/68) (2/44) (0/68) (2/44) USAID.GOV USAID-LMI COMET EVALUATION REPORT | 16 Though many instructors were previously familiar with active learning, the training they received through USAID-LMI COMET empowered them to implement these methods in their classrooms. Of the 28 KIIs and FGDs conducted with instructors, 25 (89%) highlighted how USAID-LMI COMET has led to improved classroom practices. Specifically, instructors self￾reported two features that significantly impacted their teaching: 1. The materials provided through USAID-LMI COMET included practical, concrete examples of how to apply active learning; and 2. Instructors were taught how to use these activities to better assess students—a typically challenging task at all levels of teaching. An administrator from Vietnam shared from his experience: “Before COMET, we had traditional teaching methods. When COMET was introduced, we changed our mindset about how to teach and how to help students. I like work-based learning and partnership with industry. For example, we send students to industry to work, we have documentation, detailed plans for the internship students. Students can keep records of their activities, industries comments, teachers comment with clear details. That documentation is like a toolkit. That helps us know what level students are at, how and what to improve. Previously, we taught only theories, making it hard to assess our students. Now, we have clear in our mind how to assess them (with their practice through work-based learning and industry partnership), thereby we can help our students better.” (Administrator, Vietnam). Teacher-focused activities such as the Sourcebook, Online Training Courses and Professional Development Workshops were effective and well-received. As detailed in Figure 6 below, instructors participating in the FGD outcome mapping exercise most frequently selected “online certification course” (chosen by 34 among 65 instructor FGD participants), Sourcebook for teachers (29 selections), and Professional Development Workshops (27 selections) as the USAID-LMI COMET activities that most strongly link to improved teaching. Commentary on Online Training Courses highlighted several useful elements: “new teaching techniques (e.g., student-led presentations, learner-centered assessment, etc.), how to study via online [tools]…improving my English because the course is in English” (Instructor, Vietnam). Instructors appreciated the ability to learn at their own pace through an online platform. “One week in Bangkok helped me change my mindset about teaching methodologies. I learned them before, but I didn’t understand much. Before becoming a university lecturer, I had to attend teaching methodology course, but it was theoretical, no practice. I could tell you about the methods but couldn’t apply them. Now can really do and apply them…I know how to organize class activities to engage students.” (Instructor, Vietnam) 17 | USAID-LMI COMET EVALUATION REPORT USAID.GOV Instructor KII and FGD participants provided additional positive feedback on the Sourcebook, which is comprised of multiple toolkits. These resources were described as a “more systematic” guide to active learning methods. The Sourcebook made it “easier to train others” and a resource that “[instructors] can refer to when they are going to apply activities for their students” (FGD instructors, Thailand). The Sourcebook was explicitly mentioned in 21 of 77 KIIs and FGDs (27%); it was referenced concurrently with improved teaching 17 times (of 21 mentions in KIIs and FGDs; 81%) and improved student learning eight times (of 21 mentions; 38%). Lastly, Professional Development Workshops were noted as beneficial for teachers. Seventeen of 77 KIIs and FGDs (22%) highlighted workshops; of these 17, 12 (70%) described improvements in instructor teaching. Respondents commented that the workshops helped introduce instructors to the principles of the MS2W approach, laying a foundation for engagement in the Activity through continued independent study via the online portal. After the introduction afforded by the workshop, instructors could then use the online course and Sourcebook to deepen their learning and apply techniques in the classroom. Students are receptive to and benefit from active learning. Classroom observations, KIIs, FGDs and surveys clearly demonstrate a positive response among students to the application of active learning methods. Students are highly engaged in their courses; group work, presentations, and integration of technology were frequently seen in the universities and colleges visited by the ET. Figure 7 below details student and teacher participation documented in an observation checklist during classroom observation sessions. Additionally, improvement in student learning was frequently discussed during KIIs and FGDs, with 80% (62 of 77) of respondents noting this as a key success of USAID-LMI COMET. Similarly, a majority of both students (52%; 34 of 65 responses to this question) and instructors (61%; 27 of 44 responses) surveyed agreed that the Activity has positively impacted learning and classroom experiences. Figure 6: Factors for Improved Teaching 1 1 1 3 7 10 11 22 27 29 34 Leadership Summits for MLCs Work-Based Learning Youth Initiated Activities Innovation Challenge Institutionalization Mentoring and Site Visits by EDC Staff MS2W Network Portal Application of Sourcebook Toolkit Professional Development Workshops Sourcebook for Teachers Online Instructor Certification Course When asked to designate which USAID-LMI COMET activities best contribute to improved teaching, instructors most frequently selected Online Instructor Certification Courses, Sourcebook for Teachers, and Professional Development Workshops (n=65) # and % of times activity was selected in FGD (52%) (45%) (42%) (34%) (17%) (15%) (11%) (5%) (2%) (2%) (2%) USAID.GOV USAID-LMI COMET EVALUATION REPORT | 18 As described by one instructor, due to USAID-LMI COMET training: “I know new techniques to teach my students, which makes learning enjoyable” (Instructor, Lao PDR). During KII and FGD discussions, participants noted how active learning methods encourage students to work with their peers and problem solve, making classroom sessions more interesting and engaging. Innovation Challenges were also cited as providing a unique value-add to learning, since these require students to work closely together in teams (Innovation Challenge was the second-most frequently discussed student activity, mentioned in 31 of 77 KIIs and FGDs versus 35/77 mentions of work-based learning). At the same time, it is important to note that the Activity approach focused on engaging instructors who would then impact students through their improved teaching methods (with some exceptions, including the Innovation Challenges). As discussed above (see “Approach”), in many colleges and universities USAID-LMI COMET was part of a larger active learning movement, with students having little to no awareness of program initiatives; those who were aware generally had specific references (e.g., participation in the Innovation Challenge, served as student representative, etc.). Thus, while students remarked positively on the use of active learning methods in their classrooms, these statements cannot all be explicitly tied to USAID-LMI COMET. Workload, language barriers and lack of incentives presented challenges for some instructors. Data collection revealed several key challenges specifically impacting teachers. 56% (43 of 77) of FGDs and KIIs discussed challenges, including workload (noted in 21 FGDs and KIIs), materials in English (13), lack of monetary, human resource or other incentives to adopt MS2W methods (9), large classroom size (6) and lack of administrative support (3). As detailed in Figure 8 below (also see Table 8 in Annex F: Data Disaggregation from Surveys for sex-disaggregated data), instructor survey respondents highlighted similar challenges, with the notable exception of industry partnership management—this issue was named more frequently in surveys versus FGDs and KIIs. Figure 7. Student and Instructor Engagement 89% 89% 44% 78% 89% 22% 11% 11% 56% 11% Do instructors use materials like visual aids? Are instructors engaged? Are lessons interactive? Do students work in groups? Are students engaged? Yes No Somewhat Classroom observation results demonstrate active engagement among both students and instructors (n=9) (8/9) (1/9) (7/9) (2/9) (4/9) (5/9) (8/9) (1/9) (8/9) (1/9) 19 | USAID-LMI COMET EVALUATION REPORT USAID.GOV Figure 8: Instructor Challenges Approximately half (47%; 18 of 38 conducted) of instructor and administrator KIIs and FGDs explained the difficulty of balancing a teaching workload with USAID-LMI COMET requirements, particularly the time needed to complete online trainings. Instructors interviewed by the ET discussed their existing teaching and research obligations, questioning how they can add new active learning methods to their curriculum while complying with various USAID-LMI COMET programmatic and administrative requests (e.g., filling out surveys, supporting other instructors in completing online training modules, etc.). Time spent supporting peer instructors was a more significant barrier in cases when an instructor was proficient in English and others were not; the English speaker was then asked to translate and help others in completing online training modules. As noted above, 13 KIIs and FGDs discussed challenges tied to accessibility of USAID-LMI COMET materials, which were made available in English only. While instructors initially accepted into trainings were required to have English proficiency, as the Activity expanded to new departments linguistic difficulties arose. For those without a basic command of English, online tools were challenging and required them to seek out language assistance; assignments were only accepted in English and timelines were tight for those who struggled to produce English materials and sample lesson plans. Some MLCs were better positioned to deal with this challenge—either their staff had stronger English skills overall, or an English faculty member was able to support. Nevertheless, certain instructors “could not adopt some techniques into their classes” (Instructor, Thailand) due to English-language barriers. Finally, the ET found that universities and colleges unevenly provided incentives (e.g. monetary compensation, reduced teaching workloads, etc.) to support instructors who took on the extra work of integrating USAID-LMI COMET methods in their classrooms. This challenge was of particular concern among instructors; of the nine FGDs and KIIs that highlighted incentives, five (55%) were with instructors. Despite the value of the Sourcebook, Online Training Courses and other USAID-LMI COMET resources, it is difficult to achieve systematic adoption of the MS2W approach without meaningful institutional incentives (see further discussion in “Institutionalization”). USAID.GOV USAID-LMI COMET EVALUATION REPORT | 20 CONCLUSIONS The below conclusions can be drawn regarding USAID-LMI COMET’s impact on teaching and learning: • Improvement in instructor teaching is one of the most reported successes of the Activity. USAID-LMI COMET has helped instructors shift away from traditional rote learning methods towards integrating more active learning into the classroom. Relatedly, the instructor training materials—Sourcebook, Professional Development Workshops, Online Certification Course, etc.—have filled a gap. Although instructors have long been requested to implement active learning, before USAID-LMI COMET there was a lack of resources providing concrete examples of how to do this. • Students are benefitting from the application of active learning methods in the classroom and from Innovation Challenges. However, this key segment of participants was not consistently and directly engaged by USAID-LMI COMET, and most had little knowledge of program activities. • Due to the nature of the Activity and key decisions made during program design (e.g., materials in English only, Activity implementers cannot control whether and how MLC administrations provide incentives to teachers, etc.), instructors faced several barriers to engagement. Language issues, while not unexpected, as the toolkit was only available in English, did inhibit the reach of the Activity to new instructors who had varying English language abilities—in addition to placing a burden on English￾speaking instructors who were asked to translate for others. Overall, these barriers were more prohibitive in some universities and colleges than others, depending on whether instructors felt supported to undergo training and implement the materials. “One difficulty is no control over incentives in the colleges… if people are getting performance raises due to research but not use of [the USAID-LMI COMET] model, that changes things. Some colleges were able to increase pay or increase the percent credit hours [of required teaching] (to allow instructors to adopt/follow the curriculum) and that was great, but these are independent institutions.” (EDC, Thailand) 21 | USAID-LMI COMET EVALUATION REPORT USAID.GOV INSTITUTIONALIZATION EQ5 What are the key factors that support institutionalization and the sustained implementation of the MS2W approach? What are the key challenges? EQ4 What are, if any, unintended (positive and negative) outcomes on (1) the students, instructors, and administrators that have received direct assistance through the program, and (2) other players within the partner institutions? EQ7 What are key factors for the success and/or the challenges to date? FINDINGS Adoption of USAID-LMI COMET was reported in several participating MLCs. 7As documented in “USAID-LMI COMET Lessons Learned and Proposed Adjustments,” strengthening uptake of the Activity within participating universities and colleges was prioritized following the second year of implementation. The ET found indications of success of this “institutionalization” approach; for example, 57% (25 of 44 total responses) of instructors surveyed stated that some elements of USAID￾LMI COMET have been adopted in their university or college, while 39% (17 of 44) described the Activity as “well adopted.” Notably, while only 4.5% (two of 44) indicated little or no institutionalization, no instructors reported full adoption. As seen in Figure 9 these results were relatively consistent across MLCs, with “some elements adopted” and “well adopted” constituting most survey responses. Likewise, approximately half (48%, or 37 of 77) of participants discussed improved institutionalization of USAID-LMI COMET during FGDs and KIIs. 7 Results from two respondents who did not indicate MLC affiliation omitted from this chart; no instructor survey responses received from UTYCC or NUOL. Figure 9: Adoption of USAID-LMI COMET 33% (1/3) 33% (1/3) 33% (1/3) 100% (5/5) 71% (5/7) 67% (2/3) 50% (5/10) 50% (4/8) 50% (1/2) 50% (1/2) 50% (1/2) 29% (2/7) 33% (1/3) 50% (5/10) 50% (4/8) 50% (1/2) 50% (1/2) 50% (1/2) At the majority of MLCs, instructors reported medium to high levels of USAID-LMI COMET adoption (n=44) No elements adopted Few elements adopted Some elements adopted Well adopted Mahidol MTC UD HCMUTE HueIC DUE USEA HUST ITC USAID.GOV USAID-LMI COMET EVALUATION REPORT | 22 Administration alignment to the goals of the Activity is critical to institutionalization. Data collection revealed differences in adoption of USAID-LMI COMET depending on the position of the university or college administration. Institutionalization is generally most robust at MLCs where the administration supports the Activity through concrete policies or incentives. For example, the Rector of HCMUTE is publicly committed to USAID-LMI COMET and has enacted policies to encourage use of active learning methods and work-based learning projects, tying these to instructor’s professional reviews and bonuses. At HueIC, every hour that an instructor teaches using active learning methods counts as one and a half official teaching hours. HCMUTE and HueIC were among the four MLCs where KII and FGD respondents most frequently cited improved institutionalization; 56% (five of nine total held) of KIIs and FGDs conducted at HueIC explicitly discussed successful adoption of the Activity approach, as well as 57% (four of seven) at HCMUTE, 100% (four of four) at Mahidol and 63% (five of eight) at DUE. Participants at these universities and colleges with strong administration alignment correspondingly indicated relatively fewer challenges in instructor teaching. At HueIC in particular, there were infrequent or no reports of issues with instructor workload, lack of incentives, classroom size or materials being inaccessible due to English language barriers—all challenges that arose in other MLCs. Only 22% (two of nine) of HueIC FGDs and KIIs mentioned instructor challenges, versus 90% (nine of 10) of those at NUL, 80% (four of five) at MTC and 71% (five of seven) at UTYCC. KII and FGD participants who noted better institutionalization of USAID-LMI COMET linked this to administration alignment more often than other factors, including government or instructor buy-in to the approach. Twenty-five of 77 unique KIIs and FGDs (32%) cited administration support as contributing to institutionalization, versus seven (9%) mentions of government alignment and nine (12%) of instructors. Among individual MLCs, this was again discussed most frequently at HueIC (56%, or five of nine KIIs and FGDs held), DUE (63%; five of eight) and Mahidol (100%; four of four). Thus, actions taken by the administration are a major key to successful institutionalization of USAID-LMI COMET. Government support also contributes to successful institutionalization. Like the university or vocational college administration, the government sets policies that can either encourage or dissuade MLCs from promoting active learning, connections with industry, and other core USAID-LMI COMET components. Although governments may support active learning generally, if their policies and directives to MLCs are misaligned it can mitigate the positive effects of administration support. For example, Mahidol has been designated by the Thai government as a research university, shifting emphasis away from teacher training. This impacts the university’s administrators as well as instructors, many of whom are “New [methods] always receive resistance…but the university has policies of applying blended learning and work￾based learning to encourage [teachers]. The university also gives bonuses (money, rewards) to encourage them. The university’s policy is in line with the project.” (Administrator, Vietnam) “It would be good if EDC is still implemented in ten years, [though] without EDC/USAID-LMI COMET it is unclear if the university can sustain this level of training because university is focused on science and education… Mahidol is one of nine universities that the government lists as research universities, so mainly it’s just research.” (Instructor, Thailand) 23 | USAID-LMI COMET EVALUATION REPORT USAID.GOV Use of active learning methods by new instructors Use of active learning methods by current instructors Definitely 23 Somewhat likely 17 Definitely 29 Somewhat likely 11 not “bought into” the Activity and do not see the value of changing their teaching methods. Consequently, these instructors are unwilling to dedicate time towards training. As noted above, 100% (four of four KIIs and FGDs conducted) of participants from Mahidol discussed administration alignment with USAID-LMI COMET; however, an equally high percentage (100%; four of four) mentioned institutionalization challenges. Of these, two highlighted lack of government support, and all four discussed the resulting unwillingness among some instructors to adopt the MS2W approach. Evidence points to USAID-LMI COMET sustainability, though administration and government policies can help ensure this occurs on an institutional versus individual level. There are several indications of sustainability of USAID-LMI COMET interventions, particularly among instructors already engaged in the Activity. The majority (57%, or 16 of 28 unique instructor KIIs and FGDs conducted) of instructors affirmed that they will continue to draw from the Sourcebook to apply active learning and work-based learning methods; having seen improvements in their teaching and in students’ engagement in the classroom, they are committed to implementation of MS2W methodology regardless of external support. This feedback was also reflected in instructor survey responses, most of which indicated continued use of active learning methods as well as the spread of these methods to new instructors (see Figure 10 below and Table 9/Table 10 in Annex F for sex disaggregated results). As highlighted by an instructor in Thailand, there is growing interest in USAID-LMI COMET approaches among new instructors beyond the original STEM+AT scope, which can bolster sustainability: “During my time, electrical, chemical and civil engineering have adopted [USAID-LMI COMET] most, because we are the master trainers who then train others. Other departments [have] no clear teaching techniques – faculty questioned why you’re not teaching engineering, why [teach] teaching techniques. But now some interest. Because now there is some [support for active learning], but no model for how to do this.” (Instructor, Thailand) Nevertheless, in many places—specifically in the absence of administration or government support—this is individual rather than systemic, and dependent on an instructor’s personal motivation. For MLCs seeking to maintain the program over the long-term, administration and government alignment remains critical: “[sustainability] depends on the university’s policy. If the university organizes many competitions and technology-based activities, students and teachers will apply the approach” (Instructor, HCMUTE). Figure 10: Sustainability of Active Learning Methods in MLCs 29 of 44 instructors surveyed (66%) agreed that use of active learning would “definitely” continue among current participants, while 23 of 44 (52%) believe that new instructors will “definitely” adopt these methods. (66%) (25%) (52%) (40%) USAID.GOV USAID-LMI COMET EVALUATION REPORT | 24 CONCLUSIONS Based on the above findings regarding institutionalization of USAID-LMI COMET within participating universities and colleges, the following conclusions can be drawn: • Support of the university or college administration is necessary for successful institutionalization because it helps mitigate common barriers to engagement (e.g. instructor workload, English language barriers, classroom size, etc.; see “Teaching and Learning”). While these challenges appear across participating MLCs to some degree, they are much less prohibitive in institutes with strong administrative engagement. • USAID-LMI COMET had limited engagement (with some exceptions, including participation of officials in Leadership Summits and a week-long training in Singapore) of local, regional or national governments within the five target countries. Although the lack of linkages to government did not necessarily prevent short-term adoption of the MS2W approach, it may become an issue for long￾term sustainability. MLC staff are unlikely to dedicate time and resources to activities that do not fit within broader educational directives. • Instructor-focused activities (e.g. application of the Sourcebook in classrooms) are the most sustainable element of USAID-LMI COMET for two main reasons: 1) These have had a positive effect on teaching and learning, generating enthusiasm for their use within MLCs; and 2) Implementation in classrooms where these are already in place is dependent on individual teachers, and is thus less affected by inconsistent administrative or government support. • One positive unintended outcome observed by the ET was the potential for growth of the Activity outside of the original STEM+AT scope; if properly managed (e.g., incentives provided, sufficient support for instructors with varying levels of English language proficiency, etc.), this could contribute significantly to the sustainability of USAID-LMI COMET. 25 | USAID-LMI COMET EVALUATION REPORT USAID.GOV PARTNERSHIPS EQ6 To what extent has the partnerships approach – including both multinational and local, and both personal engagement with companies and the use of online toolkits – been effective in strengthening the relationships between industry and institutions to increase the use of technology in the classroom and increase WBL opportunities, so as to close the gap between demand and supply in skilled human resources in the region? EQ7 What are key factors for the success and/or the challenges to date? EQ8 What evidence exists that the partnerships will likely be sustained – both at the multinational and at the local level? FINDINGS Despite inconsistent local industry partner knowledge of USAID-LMI COMET, the Activity helped MLCs better manage these relationships. USAID-LMI COMET included a variety of local industry, multinational and government partners, each interacting differently with participating colleges and universities. Local industry partners, who were generally located near MLCs, collaborated most frequently with students and instructors via work-based learning, internships and field trips. Multinational partners, including several US private-sector companies and academic institutions (e.g. Cisco, Amazon, Texas A&M University, etc.), engaged in larger activities and events, such as Innovation Challenges or providing cloud computing training to MLCs. Government partners also participated in activities such as Leadership Summits and leadership training, though the depth and frequency of engagement was limited. While most multinational partnerships began at the start of USAID-LMI COMET, MLCs generally relied on longstanding relationships with local companies for work-based learning, internships and field trips. KII and FGD respondents in all but one MLC (Mahidol) noted their long-term local industry partnerships, which existed years before USAID-LMI COMET. Thus, events carried out with local industry did not necessarily depend on Activity resources, and many partners had limited or uneven knowledge of USAID￾LMI COMET. In 75% (nine of 12) of local industry KIIs, participants were unclear as to whether and how USAID-LMI COMET related to the partnership between their companies and the university or college. This made it difficult for them to assess to what extent the Activity has contributed to strengthening the relationship. However, administrator and instructor KII participants highlighted the impact of USAID-LMI COMET on the overall management of local industry partnerships (see more details in the section below). 79% (30 of 38) of KIIs and FGDs held with instructors and administrations discussed usefulness of the partnerships approach; of these, 40% (12 of 30) mentioned improved organization of partnerships, second only to “Before, we had partnership[s]. [USAID-LMI] COMET helps reinforce and do it more systematically…from [the Activity’s] suggestions, we founded the Industry Advisory Committee. Before, we didn’t document the meetings with companies. Now we have a tool to do that. We have a template to document internships.” (Instructor, Vietnam) USAID.GOV USAID-LMI COMET EVALUATION REPORT | 26 better opportunities for students (53%; 16 of 30). The Sourcebook, which includes a work-based learning toolkit, and Industry Advisory Committees, which support MLCs in building partnerships, were both highlighted as useful tools for making industry relationships more organized and professional. below details results of partnerships cited by all KII and FGD participants (students, instructors, partners, etc.). Better managed local industry partnerships have led to more opportunities for students. The USAID-LMI COMET partnerships approach increased access to opportunities for students, tied particularly to work-based learning and internships. Improved organization of local industry partnerships allowed for activities to be implemented more frequently and at a higher quality, as noted by several FGD and KII respondents. At NUOL for example, instructors highlighted the usefulness of field trips to factories. At DUE, one local industry partner discussed his participation in creating curriculum as well as teaching courses for third- and fourth-year students. While access to technology and expertise, greater market entry for companies, and the development of linkages between multinational companies and the ASEAN region were also named as benefits of partnerships, these were mentioned less frequently than student opportunities (as seen in Figure 11 below, opportunities for students was cited in 29 of 77 FGDs and KIIs as a benefit of partnerships versus 10 citations for access to technology, 10 for regional and international linkages, 6 for access to expertise and 6 for company market access). Figure 11: Results of Partnership Approach Multinational partnerships had less of an impact on MLCs. Due to geographic distance and the fact that most relationships with multinational companies were managed directly by USAID and/or EDC rather than MLCs, multinational organizations were largely disconnected from core student, instructor and administrator programming. In 5 of 11 (45%) KIIs held with USAID, EDC and multinational organization staff, respondents remarked on a lack of relationships between institutes and multinational partners: “there is no direct involvement with [my company] and the MLCs, level of engagement was with USAID through a global MOU (memorandum of understanding)” (Partner, International). Aside from technology donations (e.g. Intel smart boards for all MLCs and Amazon Web Services training on and access to cloud computing resources for UTYCC and ITC) and support for Innovation Challenges, multinational partners worked less directly with participants versus local industry. 27 | USAID-LMI COMET EVALUATION REPORT USAID.GOV Thus, partnerships with local industry are the most likely to be sustained. Out of 53 KIIs and FGDs that talked about sustainability, a third (17) specifically cited continuation of partnerships—second only to sustainability of active learning, which was highlighted in almost half (24) of these discussions. Similarly, 27 of 44 (61%) instructor survey respondents stated that work-based learning and internships with local industry partnerships “definitely” will continue over the long term without external support (see Figure 12). Local industry partnerships existed before the launch of USAID-LMI COMET and will continue beyond the end of the Activity. The sustainability of these relationships hinges on market demand rather than specific USAID-LMI COMET activities; as long as universities and colleges continue to produce highly-skilled graduates, local industries will be incentivized to partner: “We’ll strengthen our relationship [with the MLC] in not only mechanics but also in sales. Due to the high demand of high-quality workforce, we’ll work closely with the college. And because of our close relationship with the college so far” (Industry Partner, Vietnam). CONCLUSIONS The following can be concluded from findings on the effectiveness and sustainability of partnerships: • Private sector partnerships are a central feature of USAID-LMI COMET, as they create important opportunities for students to engage in internships and work-based learning. Relationships between MLCs and local industry partners are deeper (i.e., they have more frequent, direct contact and take part in ongoing activities) and therefore more sustainable than those with multinational companies. Unless MLCs and multinationals can identify mutual interest in a specific project (e.g., Amazon cloud curriculum), these relationships will likely not continue after the end of the Activity. • While there are some cases of technology donations and collaboration between MLCs and multinational organizations, USAID-LMI COMET has not significantly affected use of technology in the classroom. Rather, a much greater impact is seen on management of existing local industry partner relationships and access to work-based learning opportunities for students. • One key challenge identified by the ET centered on partners’ uneven knowledge of the Activity. This resulted in a missed opportunity to build connections among various local industry partners, and between local and multinational companies. For example, local industry partners may find value in recruiting from the USAID-LMI COMET online platform YouRock but were largely unaware of this and other initiatives not directly tied to work-based learning and internships. Figure 12: Sustainability of Partnerships 1 4 12 27 Definitely will not continue Unlikely to continue Somewhat likely to continue Definitely will continue The majority of instructor survey respondents agree that USAID-LMI COMET will "definitely continue" over the long-term (n=44) (61%) (27%) (9%) (2%) USAID.GOV USAID-LMI COMET EVALUATION REPORT | 28 NETWORK EQ9 To what extent has the Network been – or is likely to be – effective in: a. Having early adopters --including instructors and institutions-- teach/support later adopters? b. Broadening its reach across the region, including in harder-to-reach areas? c. Gaining the buy-in and support of (1) the MLC administration and (2) the government? FINDINGS Instructors benefit from having a means of sharing best practices. During FGDs and KIIs, instructors in particular highlighted the usefulness of the network in enabling them to share experiences, address successes and challenges, and learn from teachers in other institutes via in￾person workshops, the MS2W portal and online professional development. Of the 34 FGDs and KIIs that discussed the effectiveness of the network, 18 (53%) of these were with instructors, seven (21%) with administrators, six (18%) with local industry, government and multinational partners, and three (9%) with other respondents (e.g. students, EDC, etc. see Figure 13 below). Instructors found online interactions via the MS2W Portal and the online certification courses helpful; these were complemented by in-person interactions during workshops or regional events such as the Leadership Summit. As one instructor in Lao PDR reflected: “So far, I have participated in the training in Thailand twice [and] attended three online courses of all three levels. I think it is a very good opportunity to have participants from many countries because we can share homework and learn from each other. We do not hesitate to provide the feedback to each other…I think it is good opportunity to learn from them.” Instructor survey participants echoed this feedback, with 91% (39 of 43 respondents to this question) indicating that the network impacts the adoption of USAID-LMI COMET, particularly among those who have trained or otherwise interacted with “early adopter” peer instructors from other institutions (of these 39 respondents, 20, or 51%, reported a “moderate effect” on Activity adoption, and 19, or 49%, a “significant effect”). See Table 12 in Annex F for sex-disaggregated results for this survey question. For administrators and partners, the network was not a major driver of their participation in USAID-LMI COMET. While a few administrators and partners noted the benefits of the network, it was not a central feature of their experience in USAID-LMI COMET—particularly in comparison to instructors. Although there were only a small number (eight of 77; 10%) of KIIs and FGDs that described the network as not useful or not a motivating factor for joining the Activity, this feedback came primarily from administrators and local industry or government partners. Of the eight KIIs and FGDs remarking negatively on the network, four (50%) were with partners, three with administrators (38%), and one (12%) with an instructor. Survey data from instructors generally upheld this finding, particularly in regard to partner engagement in the network; 53% (23 of 43 responses to this question) of instructors reported that the network had little or no effect on government support for USAID-LMI COMET. Notably, instructors do believe that the network influences administrators, with 32% (14 of 44 responses) indicating a “moderate effect” and 45% Figure 13: Effectiveness of the Network Responses 29 | USAID-LMI COMET EVALUATION REPORT USAID.GOV (20 of 44) a “significant effect” of the network on adoption of the Activity by MLC administrators. Table 11 and Table 13 in Annex F provide sex-disaggregated results for these survey questions; female instructors more frequently cited a “significant effect” of the network versus males, though these differences were not observed during qualitative data collection. Nevertheless, administrators themselves largely described the network as a positive though not essential feature of USAID-LMI COMET: “When we applied [to the Activity] we didn’t know what countries and institutes are in the project. We just know that it involves the Lower Mekong” (Administrator, Vietnam). For private sector partners, many of whom had limited knowledge of the various USAID-LMI COMET activities (see “Partnerships”), the network was less significant because they were unaware of what it did and how it functioned. The network did not broaden the reach of the Activity to rural or hard-to-access areas. The competitive application process for admission into USAID￾LMI COMET did not consider location or institute type, resulting in unbalanced demographics. In addition, application materials were available in English only, with all reporting and requirements (e.g., participation in the online certification course) also completed in English (see “Teaching and Learning”). 90% of currently active MLCs are universities, and the majority are located in capital or other large cities—where institute administrators and staff are more likely to speak English. When asked whether the Activity adopted strategies to facilitate engagement of participants from rural areas, EDC and USAID KII respondents noted while their inclusion was encouraged, there were no specific or policies in place. During the first four years of implementation of USAID-LMI COMET, the reach of the Activity did not broaden to include a larger number of institutes; instead of expansion, staff focused on institutionalization among participating universities and colleges. Key program documents, including the “USAID-LMI COMET Lessons Learned and Proposed Adjustments,” developed by EDC and approved by USAID/RDMA, outline the reasoning for this decision (e.g., uneven material mastery by Lead Instructors, insufficient institutionalization within each MLC, etc.) Nevertheless, this inhibited any possibility of growth to rural or hard-to-access areas beyond the originally selected institutions. CONCLUSIONS The below conclusions are based on evaluation findings regarding instructor, administrator and partner engagement in the USAID-LMI COMET network, as well as reach of the Activity to rural areas: • The network facilitated sharing of best practices and advice. For instructors in particular, online professional development, the MS2W portal, and in-person workshops and events significantly enhanced their experience in USAID-LMI COMET. The key value-add was not that they came from different countries, but that they taught in universities and colleges implementing the Activity. Thus, the same benefits of regional networking can also likely be achieved through in-country interactions. • For administrators and partners the network was generally not a motivator—either positively or negatively—of participation in USAID-LMI COMET. When most administrators and partners first engaged in the program, they were unaware or marginally aware of network activities. This is potential “The way that grants are done, institutes have to comply in English, making it hard for rural institutes.” (USAID/RDMA) USAID.GOV USAID-LMI COMET EVALUATION REPORT | 30 missed opportunity for the Activity; greater connections with government in particular may have strengthened the reach and sustainability of the network. • USAID-LMI COMET favored urban institutions due to the MLC selection process (i.e., through competitive procurement that did not factor in location or institute type); by the time this evaluation took place, the Activity had not expanded beyond the original 12 MLCs to include new institutes.8 This necessarily limited the network, preventing engagement of a diverse array of universities and colleges, including those in rural or hard-to-access areas. 8 After data collection was completed, USAID reported an expansion of COMET in its final year to 20 additional universities of technology in Myanmar, undertaken via direct engagement from the Burmese government. 31 | USAID-LMI COMET EVALUATION REPORT USAID.GOV RECOMMENDATIONS USAID-LMI COMET demonstrated several clear successes, particularly in the high quality and usability of training materials among the Activity’s target audience of English-speaking instructors. This allowed university and college professors to effectively implement innovative teaching methods in the classroom, enhancing students’ learning overall as well as their development of soft skills—which is critical to employability. In addition, USAID-LMI COMET’s work-based learning toolkit and other industry-focused activities (e.g. Industry Advisory Committees) enabled participating MLCs to better manage their existing relationships with local industry partners, leading to increased opportunities for students. However, alongside these core successes, several challenges ultimately limited the reach of the Activity. USAID-LMI COMET was adopted in 12 universities and colleges and—despite a shift in focus from growth to institutionalization in the third year of implementation—only in particular departments within these. This uneven application of the Activity made it difficult to identify broader programmatic contributions to workforce readiness in the Lower Mekong, particularly over the long-term. Implementation success depended heavily on the support of administrators (and to a lesser extent government); in cases where this was absent, instructors consequently faced barriers to adoption of the MS2W approach such as workload or English-language difficulties. Furthermore, MLCs did not play a central role in the management of most multinational industry partnerships, hindering the sustainability of these relationships once EDC and USAID support is concluded. Moving forward, to ensure a stronger and more sustained delivery of USAID-LMI COMET and similar Activities, the ET recommends the following: APPROACH 1) USAID should continue commissioning labor market assessments or similar needs assessment studies to inform Activity design and implementation: The labor market assessment conduced at the start of USAID-LMI COMET (and updated periodically throughout implementation) helped to ensure that the Activity approach was well-tailored to labor market needs, specifically the development of soft skills among STEM+AT students. 2) Through discussions internally and with implementing partners, USAID must ensure that Activity design aligns with the scope of intended impacts: USAID-LMI COMET key documents indicate a goal of increasing workforce readiness in the Lower Mekong, yet the Activity was not designed to achieve these broader impacts. First and most importantly, this is a difficult goal for a regional education program to attain in the allotted five years; secondly, uneven application of the approach within MLCs (see “Approach”) as well as shifts in Activity focus from expansion to institutionalization (see “Network”) limited the reach of the Activity and thus the potential for more widespread workforce impacts. Nevertheless, key documents (e.g. MEL Plans) reviewed by the ET describe intended changes to regional-level workforce readiness. This resulted in tension between USAID-LMI COMET’s stated goals and what unfolded on the ground. 3) USAID-LMI COMET must integrate more comprehensive and targeted initiatives to support female inclusion in STEM+AT: To meet this workforce need, USAID-LMI COMET must include specific, gender-focused programming. For example, the Activity can incorporate gender considerations into all teaching and learning materials, including the online professional development USAID.GOV USAID-LMI COMET EVALUATION REPORT | 32 courses or the Sourcebook toolkits (e.g., curricula with examples of women engaged in industries stereotypically considered “male”). Data collection revealed a demand among female USAID-LMI COMET participants to further hone their presentation, public speaking and writing skills (see “Approach”); supplemental programming focused on these and other relevant areas can help ensure that the Activity is responsive to this participant group. TEACHING AND LEARNING 4) USAID should continue to invest in high-quality teacher training materials and activities: The Sourcebook, Online Training Courses and Professional Development Workshops were critical in enabling instructors to introduce active learning methods in the classroom. Implementation and dissemination of these should be at the forefront of USAID-LMI COMET expansion or the development of similar education and workforce readiness initiatives. 5) USAID-LMI COMET should dedicate budget towards translation and training: Language represented a major barrier to engagement among some MLCs. USAID-LMI COMET materials (e.g. application materials, teacher training courses, etc.) need to be accessible in local languages if the Activity aims to be inclusive and to broaden its reach to rural areas. Alternatively, funding could be set aside for additional staff to train and interpret for instructors with low English skills. INSTITUTIONALIZATION 6) When universities and colleges apply to join USAID-LMI COMET, administration alignment should be a key criterion for admission: The evaluation clearly demonstrates that programming is most successful in universities and colleges with concrete administration backing for USAID-LMI COMET (see “Institutionalization”). While the Activity has little control over administration policies among current MLCs, for future initiatives a thorough review of policies towards active learning (e.g., Does the institute have written policies on active learning? Are instructors provided incentives to learn about and use these methods?) during the application stage will help ensure selection of institutes that support implementation of USAID-LMI COMET activities. 7) USAID-LMI COMET should continue to support expansion and marketing of the Activity to additional academic departments within existing MLCs: The evaluation found interest in USAID-LMI COMET outside of the targeted STEM+AT fields; continued expansion to additional departments would help broaden the reach of the Activity. As this expansion occurs, USAID-LMI COMET should be intentionally marketed among participants to raise awareness of activities and to distinguish from—or potentially find synergies with—other active learning initiatives. 8) For future initiatives, USAID and the implementing partner must aim to work more closely with government from the beginning: While USAID-LMI COMET invited participation of government officials in events and activities, it did not explicitly work towards integrating its approach in education policies or curricula. Though partnering with governments is complex and can aggravate political issues and slowdown implementation, USAID-LMI COMET and similar programs cannot affect long-term change in education systems without engagement from the Ministry of Education or similar institutions. 33 | USAID-LMI COMET EVALUATION REPORT USAID.GOV PARTNERSHIPS 9) USAID should continue to invest in effective local industry partnership activities, specifically the work-based learning toolkit and Industry Advisory Committees: Along with teacher training materials, these represented a key success of USAID-LMI COMET, as they enabled MLC administrators and instructors to better manage local industry relationships—ultimately enhancing student learning. 10) USAID-LMI COMET should reexamine staff priorities and/or hire additional dedicated staff as needed to help MLCs manage complex multinational and government partnerships: While several Activity initiatives targeted teachers and students (e.g. creation of the Sourcebook, implementation of Innovation Challenges, etc.), there was less dedication of staff and resources towards partnerships, particularly with government and multinational companies (and these relationships were generally owned by USAID and EDC rather than MLCs, making them less sustainable over the long-term; see “Partnerships”). While USAID-LMI COMET did engage a partnerships manager, further support (via a reprioritization of current staff responsibilities along with the hiring of additional staff where appropriate) is needed to better connect MLCs with government and multinational partners, and to build relationships between various partners. For example, local industry and multinational partners were largely disconnected from each other; stronger relationships among these groups may have resulted in additional ways to support work-based learning and other student opportunities, and enhanced overall partnership sustainability. NETWORK 11) USAID must amend or introduce alternatives to the competitive procurement process in order to reach a more diverse range of institutes: The geographic and educational diversity of USAID-LMI COMET was limited by the competitive procurement process, which favored institutes with English-speaking faculty located in urban areas— i.e., those best equipped to complete a U.S. government application form. If the Activity hopes to achieve widespread impact and reach institutes in rural areas, this process must be revisited. For example, USAID can set concrete targets for accepting a certain number of institutes in rural or hard-to-access areas, and provide support during the application process (e.g., translation support or making materials available in local languages). Collaborating more closely with government ministries of education during the procurement stage can also help expose a broader range of universities and colleges to USAID-LMI COMET. 12) USAID-LMI COMET should transition resources dedicated to the regional network towards strengthening national networks in each participating country: Many of the benefits of the USAID-LMI COMET network—such as the sharing of best practices with colleagues—can be attained through in-country activities. This is both cost-efficient and eliminates potential language barrier issues. In addition, the network should be national rather than exclusively tied to specific affiliation with a university or college; for example, there are a few cases of instructors transferring to different institutes after receiving USAID-LMI COMET training (although this is currently rare, these numbers are likely to grow as the Activity expands). A strong in-country network could leverage this mobility among instructors, supporting them to introduce MS2W methods into their new institutions, thereby expanding the Activity’s reach. USAID.GOV USAID-LMI COMET EVALUATION REPORT | 34 ANNEX A: SCOPE OF WORK C.3.1 USAID-LOWER MEKONG INITIATIVE CONNECTING THE MEKONG THROUGH EDUCATION AND TRAINING (USAID-LMI COMET) ACTIVITY C.3.1.1 TITLE OF EVALUATION Final performance evaluation of the USAID-Lower Mekong Initiative Connecting the Mekong through Education and Training (USAID-LMI COMET) Activity. C.3.1.2 PURPOSE OF EVALUATION This final evaluation will be used to take stock of lessons learned on the Activity (October 2014 to present) and to inform future USAID and State Department programming priorities. The purpose is to evaluate Activity performance and effectiveness against stated goals and objectives. The evaluation is being conducted a year before the end of the Activity so that findings can be used to inform future project design. The primary audiences are the USAID program manager and other USAID and State Department decision-makers, as well as the implementing partner. Other audiences may include educational institutions in the region, the private sector, host country governments, other USAID Missions implementing or planning workforce development programming, as well as other international development practitioners. The evaluation will benefit these stakeholders through improved USAID and State Department programming, both current and future, as well as improved sustainability of U.S. Government investments. Evaluation results may inform the design of future USAID and State Department regional workforce development and economic growth activities. C.3.1.3 BACKGROUND a) Award Information Activity Title Connecting the Mekong through Education and Training (AID- 486- C-14-00001) Life of Activity From October 6, 2014 to October 5, 2019 Total Estimated Funds $12,397,433 Implementer Education Development Center, Inc. b) Background and Purpose of the Award Skilled workers in the Lower Mekong countries (Cambodia, Lao PDR, Myanmar, Thailand and Vietnam) are in high-demand but in short supply: currently, 70 percent of employers are looking to hire – but only 16 percent find that recent graduates have the skills they need. 35 | USAID-LMI COMET EVALUATION REPORT USAID.GOV With the Association of Southeast Asian Nations (ASEAN) Economic Community comes regional mobility of skills and free flow of workers – an opportunity for businesses, as they gain access to a wider pool of skilled labor, but also greater challenges for graduates, as they enter an increasingly competitive labor market. To truly reap the benefits of ASEAN integration, Lower Mekong job￾seekers need job-relevant education and training. While industry in the region has been moving up the value chain toward higher-margin products, higher-education programs have struggled to keep up. Students lack multidisciplinary skill sets, and businesses are having difficulty locating talent in managerial and marketing/sales positions. This suggests that the “specialist” engineering graduates produced by the educational system may in fact be overly specialized. At the same time, although courses in statistics, probability, physics, chemistry, and other sciences are required parts of the curriculum, many new graduates lack applied knowledge in these areas -- due to the limited use of technology (for example, hands-on software such as simulations) in the classroom, and limited opportunities for internships and other work- based learning. Finally, employers indicate soft skills, such as leadership, interpersonal skills, the ability to constructively challenge authority, and problem solving, are difficult to find. The latter— lack of problem-solving skills—is particularly concerning since problem solving and creativity drive innovation, which in turn, drives growth. The lack of a systemic relationship between educational institutions and businesses limits the ability of the institutions to respond to market needs. This includes a resistance by educational institutions to update curricula, a limited role for the private sector in the curriculum development process, limited interaction between faculty and private enterprises, and limited opportunity for quality internships. USAID-LMI COMET is a five-year workforce development program that enhances ASEAN’s ability to improve economic integration and global competitiveness by equipping Lower Mekong youth with the skills they need to succeed in the 21st century marketplace. The Activity uses online learning through vocational centers and higher education institutions to increase the number of current and upcoming workers in Science, Technology, Engineering, Mathematics, Accounting and Tourism (STEM+AT). By providing students and job seekers with relevant skills – and opportunities for students to practice those skills in a real workplace setting – the Activity strengthens workforce competitiveness. The specific goal of the contract is to bridge the gap between the skills employers need and what traditional universities and vocational centers teach in the Lower Mekong sub-region. Outputs include (1) strengthening online information sharing and job forecasting to support institutional capacity development and the strengthening of networks of education providers; and (2) using distance-learning platforms and traditional workshops to increase the quality of education in high￾demand skills in STEM+AT professions. Public-private partnerships, in the form of durable and effective linkages between education providers and employers, are a key component of USAID-LMI COMET, as a way to close the gap between the demand and supply in skilled human resources. The Activity collaborates with leading multinational technology companies including Cisco, Intel, and Amazon Web Services to leverage technology and innovative instruction in classrooms, as well as to strengthen the role of women in STEM. USAID.GOV USAID-LMI COMET EVALUATION REPORT | 36 Collaboration with local industry partners is growing through the Activity’s leadership institutions – the Mekong Learning Centers (MLCs). Twelve MLCs (two in Cambodia, two in Lao PDR, one in Myanmar, two in Thailand, and five in Vietnam) are the primary nodes in USAID-LMI COMET ’s MekongSkills2Work (MS2W) Network, which connects jobseekers, academia, and industry partners. By 2019, USAID-LMI COMET will help the MLCs equip 80,000 youth with the high-demand workplace skills – including hands-on work and technology experience as well as soft skills like adaptability, teamwork, and communication – that are in demand by businesses. As the first-ever workforce development project implemented at USAID/RDMA, and one of the first regional workforce development programs at USAID, USAID-LMI COMET has adopted a learning/adaptive management approach to implementation, and has adjusted activities many times based on the feedback that is regularly solicited from program stakeholders (MLC students and instructors, the private sector, USAID, and the project team). Key achievements to date include: • Through over 400 trainings and events, and a vast array of publicly-available resources on the MS2W web portal, USAID-LMI COMET has trained over 600 instructors with cutting-edge instructional approaches and has reached over 44,000 students. • The project has adopted “innovation challenges” as an interdisciplinary approach to foster technical, soft and entrepreneurial skills in youth as they identify and solve problems in their communities or global issues. U.S. companies including Cisco and Intel have provided mentorship and technology to aid teams in refining their innovative solutions. Several teams have gone on to win international innovation challenges. • Sustaining innovation: the MS2W model is deepening its roots within the MLCs as well as branching out across the region. Instructors at nine institutions are expanding training to more instructors across their campuses, while government and industry are looking at expanding more broadly across the region. For example, based on popular demand, the MLC in Myanmar is planning a country-wide MS2W training for Technology universities in November 2018. c) Development Context Young people are growing up in a world changing faster and more fundamentally than ever before. Advances in technology are bringing forth great opportunities. But they also present challenges, as industries undergo major transformations. According to the International Labor Organization, half of all jobs in Southeast Asia may be lost to automation in the next 20 years. At the same time, given strong economic growth, new types of jobs are being created. In 2015, 70 percent of employers in the Lower Mekong sub-region were looking to hire, but only 16 percent found that recent graduates had the skills they needed. To fill the jobs that are in demand in the marketplace, and to reap the benefits of labor mobility under Association of Southeast Asian Nations (ASEAN) integration, Lower Mekong job-seekers need job-relevant education and training in key professions. In a quickly evolving marketplace, soft skills become increasingly important - even more so than technical skills. These are the skills that will allow employees to adapt to quickly changing job requirements, to think critically, analyze effectively, work together, and create new ideas. 37 | USAID-LMI COMET EVALUATION REPORT USAID.GOV d) Link to USAID/RDMA Strategy USAID/RDMA’s Regional Development Cooperation Strategy (RDCS) includes a focus on strengthening regional integration (Intermediate Result 1.1). Regional integration plays an important role in Asia’s development for a variety of reasons. While fostering the more efficient movement of goods, services, and people, it also provides an opportunity to establish common standards that promote inclusive and sustainable growth practices, as well as fostering a sense of regional identity and connectivity that can assist in addressing regional issues. Regional institutions play a critical role in integration by establishing the framework for cooperation, dialog and decision-making. This includes ASEAN as well as the Asia-Pacific Economic Cooperation. In addition, U.S. Government initiatives such as LMI and the Young Southeast Asian Leaders Initiative play important roles in promoting economic and social regional integration by focusing U.S. development efforts around themes that align with regional priorities. LMI is a multilateral partnership initiated by the U.S. in 2009 to promote cooperation in the Mekong sub-region, in areas that include connectivity and education. The priorities outlined in the LMI Plan of Action align with those of the Master Plan on ASEAN Connectivity and those of the Initiative for ASEAN Integration, which stresses the immediate need to accelerate the development of Myanmar, Cambodia, Lao PDR, and Vietnam for the successful realization of the ASEAN Economic Community. USAID-LMI COMET falls under Sub-Intermediate Result 1.2.2 of USAID/RDMA’s RDCS, which calls for enhanced movement and protection of people and knowledge. The Activity aims to improve the reach and quality of post-secondary education in STEM+AT, to strengthen youth development and economic growth in the least developed countries in ASEAN. Intended Results USAID-LMI COMET enhances ASEAN’s ability to improve economic integration and global competitiveness through human resource development in the Lower Mekong sub-region. The Activity bridges the gap between the skills employers need and what traditional universities and vocational centers teach in the Lower Mekong sub-region by supporting institutional capacity development and the strengthening of networks of education providers, as well as using technology and online learning to increase the quality of education in high-demand skills. By 2019, USAID-LMI COMET will help 12 post-secondary education institutions in Lower Mekong countries equip 80,000 youth with the high-demand workplace skills – including hands-on work and technology experience as well as soft skills like adaptability, teamwork, and communication – demanded by businesses in Science, Technology, Engineering, Math, Accounting and Tourism (STEM+AT) sectors. The key Activity outcome indicator is “Number of persons receiving new or better employment as a result of participation in U.S. Government-funded workforce development activity”. Approach & Implementation Through its signature MekongSkills2Work approach, the Activity: USAID.GOV USAID-LMI COMET EVALUATION REPORT | 38 • empowers university and vocational school instructors with the tools they need to create active, student-centered, learning environments in and outside the classroom; and, • provides tools to strengthen industry partnerships so as to increase work exposure, for example through internships. The MekongSkills2Work Sourcebook is a collection of publicly available online toolkits that instructors can use to transform their classrooms into dynamic learning environments. The USAID￾LMI COMET project team, and now a core group of high-achieving MekongSkills2Work instructors throughout the region, provide extensive coaching to support these instructors in changing from (passive) lecturing to (active) facilitation. The Activity also supports educational institutions in building and strengthening their relationships with industry, to increase employers’ input into the curriculum, and improve hands-on learning through innovation challenges and work-based learning activities including internships. Finally, the Activity collaborates with leading American multinational technology companies (Cisco, Intel, Amazon Web Services) and universities (Texas A&M) to leverage cutting edge educational approaches and technologies to facilitate educational opportunities in the Lower Mekong countries while improving workforce capacity for international and U.S. companies. Geographical Coverage & Targeted Beneficiaries As an LMI signature Activity, USAID-LMI COMET focuses on the five countries of the Lower Mekong. A network of competitively-selected higher-education institutions, known as “Mekong Learning Centers” (two in Cambodia, two in Lao PDR, one in Myanmar, two in Thailand, and five in Vietnam), form the core of the MekongSkills2Work Network, which connects jobseekers, academia, and industry partners. The targeted beneficiaries are university and vocational college students. Available Documents The following documents will be shared with the Contractor: • USAID-LMI COMET contract and modifications • Annual work plans • Quarterly and annual reports • Monitoring, Evaluation and Learning Plan, with indicator data • Data Quality Assessment Report • USAID-LMI COMET lessons learned and proposed adjustments document • Institutionalization maps, grant reviews, and partnerships tracker • Factsheets, social media coverage, USAID-approved success stories • Relevant data from M&E surveys conducted by the USAID COMET project team, such as the youth employment survey, online learning team survey, Sourcebook standards self-assessment survey, portal user experience survey, survey to Mekong Learning Center instructors and administrators in late 2016, etc. • Suggested Informant List • Social media coverage • USAID-approved success stories 39 | USAID-LMI COMET EVALUATION REPORT USAID.GOV C.3.1.4 EVALUATION QUESTIONS The evaluation questions below are listed in the order of priority. All key and sub-questions must be answered. USAID-LMI COMET ’s signature approach includes innovative instruction (through teaching tools housed online in the MekongSkills2Work Sourcebook and accompanied by in-person and online training and coaching), workplace exposure through private sector partnerships, and regional learning and sharing through participation in the MekongSkills2Work Network. 1. To what extent is the MekongSkills2Work approach meeting the needs of today’s workforce and is the approach likely to meet workforce needs in the next five to ten years in the targeted countries? 2. To what extent has the MekongSkills2Work approach affected classroom practices and teaching strategies among instructors within the Mekong Learning Centers? What is the instructors’ level of awareness of, commitment to, and implementation of the approach (disaggregated and analyzed by sex, country, type of institution, and type of intervention/service or other variables as appropriate)? 3. To what extent has the MekongSkills2Work approach affected student learning? What is students’ level of awareness, receptivity to, and perceived usefulness of the approach as implemented by their instructors (disaggregated and analyzed by sex, country, type of institution, and type of intervention/service or other variables as appropriate)? 4. What are, if any, unintended (positive and negative) outcomes on (1) the students, instructors, and administrators that have received direct assistance through the Activity, and (2) other players within the partner institutions? 5. What are the key factors that support institutionalization and the sustained implementation of the MekongSkills2Work approach? What are the key challenges? Industry-academic partnerships are a key part of the long-term view for USAID-LMI COMET, to help close the human resources gap between employers and education providers. 1. To what extent has the partnerships approach – including both multinational and local, and both personal engagement with companies and the use of online toolkits – been effective in strengthening the relationships between industry and institutions to increase the use of technology in the classroom and increase work-based learning opportunities, so as to close the gap between demand and supply in skilled human resources in the region? 2. What are key factors for the success and/or the challenges to date? 3. What evidence exists that the partnerships will likely be sustained – both at the multinational and at the local level? The MekongSkills2Work Network is led by the Mekong Learning Centers, who serve as MekongSkills2Work leadership institutions in the sub-region and includes private sector partners and job-seekers. 1. To what extent has the Network been – or is likely to be – effective in: USAID.GOV USAID-LMI COMET EVALUATION REPORT | 40 a. Having early adopters --including instructors and institutions-- teach/support later adopters? b. Broadening its reach across the region, including in harder-to-reach areas? c. Gaining the buy-in and support of (1) the Mekong Learning Center administration and (2) the government? C.3.1.5 EVALUATION DESIGN AND SUGGESTED METHODOLOGIES a) Study Design, Data Collection and Analysis Methods The evaluation will utilize mixed methods. The methodology should be collaborative and participatory. It should include USAID, the implementing partner, and key populations benefitting from or otherwise involved in the Activity interventions, as much as possible, in evaluation planning and implementation. USAID suggests interviews, focus group discussions, and surveys as primary data collection methods. The evaluation team will design appropriate tools for collecting data from various units of analysis. Any data collection tools developed will be submitted to USAID/RDMA for approval prior to fielding. It is anticipated that the field data collection will be conducted in a combination of English and local languages. In addition to these suggestions, the Contractor will provide a more detailed explanation of the proposed methodology for carrying out the work as laid out in the matrix below in the design phase. USAID requests that the Contractor proposes the areas of inquiry and methods that it believes, based on its evaluation and sectoral expertise, are best suited to answering the research questions and achieving the stated purpose. The evaluation team will be required to perform the field research in a minimum of four Mekong Learning Centers in three Lower Mekong countries (to be determined in collaboration with USAID and the USAID-LMI COMET team). The three countries for the field research will likely be Thailand, Cambodia and Vietnam. In addition, the evaluation team will be required to conduct a virtual data collection among beneficiaries in any country and Mekong Learning Center it does not physically travel to. The team will also conduct interviews and gather information from USAID and USAID-LMI COMET project staff working in Bangkok The information collected will be analyzed by the evaluation team to establish credible answers to the questions and provide major trends and issues. The proposal should detail any data analysis software or specialized coding systems to be used in the analysis. 41 | USAID-LMI COMET EVALUATION REPORT USAID.GOV Research Questions Suggested Data Sources Suggested Data Collection Methods (including sampling methodology, where applicable) Suggested Data Analysis Methods The MekongSkills2Work approach. 1. To what extent is the MekongSkills2Work approach meeting the needs of today’s workforce --and is likely to meet workforce needs in the next five to ten years-- in the targeted countries? To what extent has the MekongSkills2Work approach affected classroom practices and teaching strategies among instructors within the Mekong Learning Centers? What is the instructors’ level of awareness of, commitment to, and implementation of the approach (disaggregated and analyzed by sex, country, type of institution, and type of intervention/service or other variables as appropriate)? To what extent has the MekongSkills2Work approach affected student learning? What is students’ level of awareness, Primary: • Mekong Learning Center administrators, instructors and students • USAID-LMI COMET team • USAID Secondary: • Other staff at the Mekong Learning Centers Desktop review Online questionnaire (survey) Interviews and focus group discussions at a minimum of four Mekong Learning Centers in three countries Complementary information gathered by email or telephone, including potentially at additional Mekong Learning Centers Observations, as time permits and deemed necessary/appropriate , of USAID- LMI COMET events, such as classes led by MekongSkills2Work instructors. To be determined by the evaluation team. Disaggregate and analyze data by sex, and by other variables such as countries, type of institution, and type of intervention /service as appropriate. USAID.GOV USAID-LMI COMET EVALUATION REPORT | 42 receptivity to, and perceived usefulness of the approach as implemented by their instructors (disaggregated and analyzed by sex, country, type of institution, and type of intervention/service or other variables as appropriate)? 4. What are, if any, unintended (positive and negative) outcomes on (1) the students, instructors, and administrators that have received direct assistance through the Activity, and (2) other players within the partner institutions? What are the key factors that support institutionalization and the sustained implementation of the MekongSkills2Work approach? What are the key challenges? classes led by MekongSkills2Work instructors. 43 | USAID-LMI COMET EVALUATION REPORT USAID.GOV Partnerships. 6. To what extent has the partnerships approach – including both multinational and local, and both personal engagement with companies and the use of online toolkits – been effective in strengthening the relationships between industry and institutions to increase the use of technology in the classroom and increase work-based learning opportunities, so as to close the gap between demand and supply in skilled human resources in the region? What are key factors for the success and/or the challenges to date? What evidence exists that the partnerships will likely be sustained – both at the multinational and at the local level? Primary: • Mekong Learning Center administrators and instructors • Private sector partners, including multinationals and local businesses • USAID-LMI COMET team • USAID Secondary: • Other staff at the Mekong Learning Centers • Other businesses Desktop review Online questionnaire (survey) Interviews and focus group discussions Email or telephone interviews To be determined by the evaluation team. USAID.GOV USAID-LMI COMET EVALUATION REPORT | 44 The Network. To what extent has it been – or is likely to be – effective in: (a) Having early adopters -- including instructors and institutions-- teach/support later adopters? (b) Broadening its reach across the region, including in harder-to- reach areas? (c) Gaining the buy-in and support of (1) the Mekong Learning Center administration and (2) the government? Primary: ● Mekong Learning Center administrators and instructors ● Government officials affiliated with the Activity ● USAID-LMI COMET team ● USAID Secondary: ● Other staff at the Mekong Learning Centers and other government officials Desktop review Online questionnaire (survey) Interviews and focus group discussions at a minimum of four Mekong Learning Centers in three countries Email or telephone. To be determined by the evaluation The Contractor will start the evaluation by reviewing documentation provided by USAID and the USAID-LMI COMET team, and any relevant secondary research they collect. The Contractor is expected to construct the overall theory of change for this evaluation during the desk review period, as well as to present initial findings from the document review against the evaluation questions in the evaluation design and implementation plan. b) Limitations or constraints of the design and methodologies Given the time and cost implications of evaluating an Activity spread out over five countries with five different languages, the evaluation team will likely be required to perform the field research in three Lower Mekong countries only (Cambodia, Thailand and Vietnam), with virtual data collection in the other two countries (Burma and Lao PDR). To compensate for this limitation, the countries and educational institutions will be selected in collaboration with USAID and the USAID-LMI COMET team to represent the spectrum of institutions involved in the Activity, including but not limited to both university and technical school(s), various STEM+AT fields, and level of adoption of the MekongSkills2Work approach. 45 | USAID-LMI COMET EVALUATION REPORT USAID.GOV Language is another limitation. While the majority of MekongSkills2Work Network members speak English, it is not their native tongue, so communication can be an issue. The methodology and data collection instruments, in particular survey design and interview questions guide, should take this into account and be as clear and concise as possible. In addition, the evaluation team may engage the help of a translator to increase access to non-English speakers. c) USAID-LMI COMET Documents for Review 1. USAID-LMI COMET Contract and Modifications (ATTACHMENT J.3, the remaining documents will be shared upon the award issuance) 2. Annual work plans 3. Monitoring & Evaluation Plan with indicator data 4. Quarterly and annual reports 5. USAID COMET lessons learned and proposed adjustments document 6. Social media coverage 7. USAID-approved success stories 8. Factsheets 9. Relevant data from the MLC surveys such as the youth employment survey, the online learning team survey, the Sourcebook standards self-assessment survey, the portal user experience survey, etc. USAID.ORG USAID-LMI COMET EVALUATION REPORT | 46 ANNEX B: DATA COLLECTION PROTOCOLS 1.KEY INFORMANT INTERVIEW PROTOCOLS PROTOCOL A: STUDENTS/ALUMNI I. Interview Tracking Information Interview ID: Date: Interviewer: Notetaker: Translator: MLC Affiliation (if applicable), Location: Length of Recording (hours/minutes): File Name: 2. Introductions and Informed Consent Interviewer Thank you for coming here today. We are grateful that you are giving us your time. First, let me introduce the team: [The facilitator, translator and notetaker should introduce themselves]. We are here on behalf of Social Impact and USAID to conduct research on the effects of development projects funded by the United States government on your community. Before we begin, you should know that neither this research nor your answers to our questions will in any way determine if a project will be implemented here, continue to be here, or effect your ability to access or receive any services. The benefit of this research is to ensure that projects are well designed, and work well together based on local people’s input. This interview will take approximately one hour. With your permission, we would like to audio record our conversation and take notes. The recordings and the notes will not be shared with anyone outside the research team and your name will be kept confidential. We hope this makes you feel comfortable to express your ideas freely. Your participation in this discussion is voluntary and so if you do not feel comfortable, then you are welcome to excuse yourself at any time during the research without any consequences to you. 47 | USAID-LMI COMET EVALUATION REPORT USAID.ORG Do you have any questions? Do you agree to participate? Will you allow us to record the conversations? Thank you! 3. Participant Demographic Information Please ask the interview participant to provide the following information: Name: □ Student □ Alumni Sex: □ Male □ Female Age Range: □ 18-19 □ 20-24 □ 25-29 □ 30-35 □ 35+ Current or previous area of study: Year of study at institute: □ First □ Second □ Third □ Fourth □ Fifth □ Other: ___________ Actual or anticipated graduation date (Month; Year): __________________________ Employment status: □ Unemployed □ Full-time, one employer □ Full-time, multiple employers □ Part-time, one employer □ Part-time, multiple employers □ Self-employed/Entrepreneur Current occupation (if employed): ____________________________ LMI-COMET Participation (Years): □ 0-6 months □ 6 months – 1 year □ 1 – 2 years □ 2 – 3 years □ 4+ years 4. Interview Questions No. Question (Students / Alumni) EQ 1 Are you aware of the LMI-COMET program being implemented in your institute? If yes, in your opinion please describe the key characteristics and objectives of the program. N/A 2 Has the program had any effects (positive or negative) on instructor teaching and classroom practices? Do these effects differ by group (females versus males; subject area)? Can you think of any stories or examples that illustrate these effects? 2 USAID.ORG USAID-LMI COMET EVALUATION REPORT | 48 3 Has the program had any effects (positive or negative) on student learning and experiences in the classroom? Do these effects differ by group (females versus males; subject area)? Can you think of any stories or examples that illustrate these effects? 3 4 Have you observed or experienced anything (positive or negative) about the LMI-COMET program that you did not expect or that surprised you? 4 5 What do you think have been the most successful aspects of the LMI-COMET program? What do you think have been the major challenges? Are these successes and challenges the same for all groups: females versus males; subject area? 7 6 What skills do students need in order to find a position in the STEM+AT sectors and succeed over the long-term in that position? Do these needs differ by group (females versus males; subject area)? Do you think that the LMI-COMET program equips students with these skills? Why or why not? 1 7 Do you think that the LMI-COMET program has helped prepare you for your current (alumni) or desired future (students) career? Why or why not? 1 8 Please provide any additional thoughts or feedback on the LMI-COMET program. Would you recommend that the program continues and/or expands either within existing institutes or externally to other institutes? Why or why not? In the same capacity or in an altered capacity (probe: such as)? N/A 49 | USAID-LMI COMET EVALUATION REPORT USAID.ORG PROTOCOL B: INSTRUCTORS AND/OR TECH SUPPORT I. Interview Tracking Information Interview ID: Date: Interviewer Notetaker: Translator: MLC Affiliation (if applicable), Location: Length of Recording (hours/minutes): File Name: 2. Introductions and Informed Consent Interviewer Thank you for coming here today. We are grateful that you are giving us your time. First, let me introduce the team: [The facilitator, translator and notetaker should introduce themselves]. We are here on behalf of Social Impact and USAID to conduct research on the effects of development projects funded by the United States government on your community. Before we begin, you should know that neither this research nor your answers to our questions will in any way determine if a project will be implemented here, continue to be here, or effect your ability to access or receive any services. The benefit of this research is to ensure that projects are well designed, and work well together based on local people’s input. This interview will take approximately one hour. With your permission, we would like to audio record our conversation and take notes. The recordings and the notes will not be shared with anyone outside the research team and your name will be kept confidential. We hope this makes you feel comfortable to express your ideas freely. Your participation in this discussion is voluntary and so if you do not feel comfortable, then you are welcome to excuse yourself at any time during the research without any consequences to you. Do you have any questions? Do you agree to participate? Will you allow us to record the conversations? Thank you! USAID.ORG USAID-LMI COMET EVALUATION REPORT | 50 3. Participant Demographic Information Please ask the interview participant to provide the following information: Name: Sex: □ Male □ Female Area of instruction: Years of employment at the institute: □ 1-2 □ 3-5 □ 6-9 □ 10-14 □ 15-19 □ 20+ LMI-COMET Participation (Years): □ 0-6 months □ 6 months – 1 year □ 1 – 2 years □ 2 – 3 years □ 4+ years LMI-COMET Participation (Type): □ Level 1 □ Level 2 □ Level 3 (Master) □ Other/No Level 4. Interview Questions No. Questions (Instructors and/or Tech Support) EQ 1 Please describe your classroom arrangement: seating, equipment, number of students, etc. What is your approach to lesson planning and classroom instruction? 2 2 Are you aware of the LMI-COMET program being implemented in your institute? If yes, in your opinion please describe the key characteristics and objectives of the program. N/A 3 Has the program had any effects (positive or negative) on instructor teaching and classroom practices? Do these effects differ by group (females versus males; subject area)? Can you think of any stories or examples that illustrate these effects? 2 4 Has the program had any effects (positive or negative) on student learning and experiences in the classroom? Do these effects differ by group (females versus males; subject area)? Can you think of any stories or examples that illustrate these effects? 3 5 Have you observed or experienced anything (positive or negative) about the LMI-COMET program that you did not expect or that surprised you? 4 6 Which departments (subject areas) or staff (administrators, instructors, etc.) have been the strongest promoters of the LMI￾COMET program in your institute, and why? Who has been more reluctant to adopt the program, and why? 5 51 | USAID-LMI COMET EVALUATION REPORT USAID.ORG 7 In your opinion, will the following continue over the long-term in your institute without external support from EDC? Why or why not?: • Use of active-learning methods by instructors currently engaged in LMI-COMET • Spread of active-learning methods to new instructors in your institute who are not currently engaged in LMI-COMET • Work-based learning and internships with local industry partners 5 8 The LMI-COMET program includes participants from multiple countries across the Lower Mekong region, including instructors in other institutes who you may have interacted with online or in workshops. Has this regional participation impacted your engagement in the program in any way? Why or why not? 9 9 What do you think have been the most successful aspects of the LMI-COMET program? What do you think have been the major challenges (probe: has your institute experienced technology challenges that impacted implementation of the program)? Are these successes and challenges the same for all groups: females versus males; students, instructors, administrators or others at the institute; subject area? 7 10 What skills do students need in order to find a position in the STEM+AT sectors and succeed over the long-term in that position? Do these needs differ by group (females versus males; subject area)? Do you think that the LMI-COMET program equips students with these skills? Why or why not? 1 11 Please provide any additional thoughts or feedback on the LMI-COMET program. Would you recommend that the program continues and/or expands either within existing institutes or externally to other institutes? Why or why not? In the same capacity or in an altered capacity (probe: such as)? N/A USAID.ORG USAID-LMI COMET EVALUATION REPORT | 52 PROTOCOL C: ADMINISTRATORS I. Interview Tracking Information Interview ID: Date: Interviewer Notetaker: Translator: MLC Affiliation (if applicable), Location: Length of Recording (hours/minutes): File Name: 2. Introductions and Informed Consent Interviewer Thank you for coming here today. We are grateful that you are giving us your time. First, let me introduce the team: [The facilitator, translator and notetaker should introduce themselves]. We are here on behalf of Social Impact and USAID to conduct research on the effects of development projects funded by the United States government on your community. Before we begin, you should know that neither this research nor your answers to our questions will in any way determine if a project will be implemented here, continue to be here, or effect your ability to access or receive any services. The benefit of this research is to ensure that projects are well designed and work well together based on local people’s input. This interview will take approximately one hour. With your permission, we would like to audio record our conversation and take notes. The recordings and the notes will not be shared with anyone outside the research team and your name will be kept confidential. We hope this makes you feel comfortable to express your ideas freely. Your participation in this discussion is voluntary and so if you do not feel comfortable, then you are welcome to excuse yourself at any time during the research without any consequences to you. Do you have any questions? Do you agree to participate? Will you allow us to record the conversations? Thank you! 53 | USAID-LMI COMET EVALUATION REPORT USAID.ORG 3. Participant Demographic Information Please ask the interview participant to provide the following information: Name: Sex: □ Male □ Female Years of employment at the institute: □ 1-2 □ 3-5 □ 6-9 □ 10-14 □ 15-19 □ 20+ Years of employment at the administrator position: □ 1-2 □ 3-5 □ 6-9 □ 10-14 □ 15-19 □ 20+ LMI-COMET Participation (Years): □ 0-6 months □ 6 months – 1 year □ 1 – 2 years □ 2 – 3 years □ 4+ years 4. Interview Questions No. Questions (Administrators) EQ 1 In your opinion, please describe the key characteristics and objectives of the LMI-COMET program currently being implemented by your institute. N/A 2 Have you observed or experienced anything (positive or negative) about the LMI-COMET program that you did not expect or that surprised you? 4 3 Which departments (subject areas) or staff (administrators, instructors, etc.) have been the strongest promoters of the LMI￾COMET program in your institute, and why? Who has been more reluctant to adopt the program, and why? 5 4 In your opinion, will the following continue over the long-term in your institute without external support from EDC? Why or why not?: • Use of active-learning methods by instructors currently engaged in LMI-COMET • Spread of active-learning methods to new instructors in your institute who are not currently engaged in LMI-COMET • Work-based learning and internships with local industry partners 5 5 Prior to the start of the LMI-COMET program, did your institute have relationships with local industry partners? With multinational partners? If yes, please describe these relationships (probe: student internships; work-based learning; other activities). 6 USAID.ORG USAID-LMI COMET EVALUATION REPORT | 54 6 Since the start of the LMI-COMET program, has there been a change in the relationships between your institute and local industry partners? Has there been a change in the relationships between your institute and multinational partners? If yes, please describe these changes (probe: increased access to and use of technology; increased opportunities for work-based learning and other professional development, both generally and for females) 6 7 In your opinion, will the relationships between your institute and local industry partners continue over the long-term without external support from EDC? Will relationships between your institute and multinational partners continue? Why or why not? 8 8 The LMI-COMET program includes participants from multiple countries across the Lower Mekong region. Has this regional participation impacted the adoption of the program by the administration of your institute? Why or why not? 9 9 What do you think have been the most successful aspects of the LMI-COMET program? What do you think have been the major challenges? Are these successes and challenges the same for all groups: females versus males; students, instructors, administrators or others at the institute; subject area? 7 10 What skills do students need in order to find a position in the STEM+AT sectors and succeed over the long-term in that position? Do these needs differ by group (females versus males; subject area)? Do you think that the LMI-COMET program equips students with these skills? Why or why not? 1 11 Please provide any additional thoughts or feedback on the LMI-COMET program. Would you recommend that the program continues and/or expands either within existing institutes or externally to other institutes? Why or why not? In the same capacity or in an altered capacity (probe: such as)? N/A 55 | USAID-LMI COMET EVALUATION REPORT USAID.ORG PROTOCOL D: PARTNER-LOCAL INDUSTRY/PARTNER-GOVERNMENT/PARTNER-MULTINATIONAL I. Interview Tracking Information Interview ID: Date: Interviewer Notetaker: Translator: MLC Affiliation (if applicable), Location: Length of Recording (hours/minutes): File Name: 2. Introductions and Informed Consent Interviewer Thank you for coming here today. We are grateful that you are giving us your time. First, let me introduce the team: [The facilitator, translator and notetaker should introduce themselves]. We are here on behalf of Social Impact and USAID to conduct research on the effects of development projects funded by the United States government on your community. Before we begin, you should know that neither this research nor your answers to our questions will in any way determine if a project will be implemented here, continue to be here, or effect your ability to access or receive any services. The benefit of this research is to ensure that projects are well designed and work well together based on local people’s input. This interview will take approximately one hour. With your permission, we would like to audio record our conversation and take notes. The recordings and the notes will not be shared with anyone outside the research team and your name will be kept confidential. We hope this makes you feel comfortable to express your ideas freely. Your participation in this discussion is voluntary and so if you do not feel comfortable, then you are welcome to excuse yourself at any time during the research without any consequences to you. Do you have any questions? Do you agree to participate? Will you allow us to record the conversations? Thank you! USAID.ORG USAID-LMI COMET EVALUATION REPORT | 56 3. Participant Demographic Information Please ask the interview participant to provide the following information: Name: Sex: □ Male □ Female Years of employment at (organization or office name): □ 1-2 □ 3-5 □ 6-9 □ 10-14 □ 15-19 □ 20+ LMI-COMET Participation (Years): □ 0-6 months □ 6 months–1 year □ 1–2 years □ 2–3 years □ 4+ years 4. Interview Questions No. Questions (Partners: Local Industry, Government, Multinationals) EQ 1 In your opinion, please describe the key characteristics and objectives of the LMI-COMET program currently being implemented by (institute name) N/A 2 Prior to the start of the LMI-COMET program, did your office have a relationship with (institute name). If yes, please describe this relationship (probe only for local industry and multinational partners: student internships; work-based learning; other activities). 6 3 Has this changed since the start of LMI-COMET? If yes, please describe these changes (probe: increased access to and use of technology; increased opportunities for work-based learning and other professional development, both generally and for females). In your opinion, will this continue over the long-term without external support from EDC? Why or why not? 8 4 [Local industry/multinational only] What issues or factors can help ensure the continuation of partnership activities with (institute name) over the long-term? [Government only] What issues or factors can help strengthen government support for the LMI-COMET program over the long-term? 5,8 5 Have you observed or experienced anything (positive or negative) about the LMI-COMET program that you did not expect or that surprised you? 4 57 | USAID-LMI COMET EVALUATION REPORT USAID.ORG 6 The LMI-COMET program includes participants from multiple countries across the Lower Mekong region. Has this regional participation impacted your office’s support of the program? Why or why not? 9 7 What do you think have been the most successful aspects of the LMI-COMET program? What do you think have been the major challenges? Are these successes and challenges the same for all groups: females versus males; different institutes [government partners only]; subject area 7 8 What skills do students need in order to find a position in the STEM+AT sectors and succeed over the long-term in that position? Do these needs differ by group (females versus males; subject area)? Do you think that the LMI-COMET program equips students with these skills? Why or why not? 1 9 Please provide any additional thoughts or feedback on the LMI-COMET program. Would you recommend that the program continues and/or expands either within existing institutes or externally to other institutes? Why or why not? In the same capacity or in an altered capacity (probe: such as)? N/A USAID.ORG USAID-LMI COMET EVALUATION REPORT | 58 PROTOCOL E: EDC/USAID STAFF I. Interview Tracking Information Interview ID: Date: Interviewer Notetaker: Translator: MLC Affiliation (if applicable), Location: Length of Recording (hours/minutes): File Name: 2. Introductions and Informed Consent Interviewer Thank you for coming here today. We are grateful that you are giving us your time. First, let me introduce the team: [The facilitator, translator and notetaker should introduce themselves]. We are here on behalf of Social Impact and USAID to conduct research on the effects of development projects funded by the United States government on your community. Before we begin, you should know that neither this research nor your answers to our questions will in any way determine if a project will be implemented here, continue to be here, or effect your ability to access or receive any services. The benefit of this research is to ensure that projects are well designed, and work well together based on local people’s input. This interview will take approximately one hour. With your permission, we would like to audio record our conversation and take notes. The recordings and the notes will not be shared with anyone outside the research team and your name will be kept confidential. We hope this makes you feel comfortable to express your ideas freely. Your participation in this discussion is voluntary and so if you do not feel comfortable, then you are welcome to excuse yourself at any time during the research without any consequences to you or your families. Do you have any questions? Do you agree to participate? Will you allow us to record the conversations? Thank you! 59 | USAID-LMI COMET EVALUATION REPORT USAID.ORG 3. Participant Demographic Information Please ask the interview participant to provide the following information: Name: □ USAID □ EDC Sex: □ Male □ Female Title; Role with LMI-COMET Program: LMI-COMET Participation (Years): □ 0-6 months □ 6 months – 1 year □ 1 – 2 years □ 2 – 3 years □ 4+ years Years of employment at (organization or office name): □ 1-2 □ 3-5 □ 6-9 □ 10-14 □ 15-19 □ 20+ 4. Interview Questions No. Questions (EDC/USAID staff) EQ 1 In your opinion, please describe the key characteristics and objectives of the LMI-COMET program 1 2 Please describe any gender analysis activities conducted during the course of the LMI-COMET program 1 3 Did the LMI-COMET program adopt any specific strategies to facilitate engagement of females and/or other marginalized groups? If yes, please describe. Did a technical expert or other staff support in implementing these strategies? 1 4 What skills or attributes do workers need to be successful, particularly in the STEM+AT sectors? Do these needs differ by group (females versus males; subject area)? 1 5 Do you think that the LMI-COMET program currently addresses those needs? Why or why not? What about a year from now? In 5 years? In 10 years? 1 6 What could the LMI-COMET program do to better address those needs, particularly for females? Please provide specific examples if possible. 1 7 To what extent do you think the LMI-COMET program has been adopted and incorporated by administrators, instructors, and others in the participating institutes? Why or why not has the program been adopted? Does this differ by group (female versus male; subject area)? 5 USAID.ORG USAID-LMI COMET EVALUATION REPORT | 60 8 Are there particular departments or staff that tend to be strong promoters of the LMI-COMET program within their institutes? Are there others that generally face difficulties? Why? 5 9 In a year, what do you think the LMI-COMET program will look like? What about in 5 or 10 years? 5 10 How, if at all, have relationships between institutes and (local industry, multinational, government) partners changed (positively or negatively) since the start of the LMI-COMET program? 6 11 Since the start of the LMI-COMET program, have the relationships between institutes and (local industry, multinational, government) partners affected the following in any way: • Access to and use of technology by participating institutes • Opportunities for work-based learning • Female access to workforce development opportunities 6 12 What issues or factors could weaken partnerships between institutes and (local, government, multinational) partners over the long-term? What can help ensure that they remain strong over the long-term? 8 13 As you are aware, the LMI-COMET program includes participants from multiple countries and types of institutes across the Lower Mekong region. What, if any, have been the effects of this regional participation? Can you think of any stories or examples that illustrate these effects? 9 14 To what extent has this regional participation in the LMI-COMET program influenced the following: ● The adoption and incorporation of the program by administrators ● The adoption and incorporation of the program by instructors, particularly those who have trained or otherwise interacted with instructors from other institutes ● The participation of (local industry, government and multinational) partners in the LMI-COMET program ● Reach of the program in areas that are rural or otherwise harder to access 9 15 What do you think have been the most successful aspects of the LMI-COMET program? What do you think have been the major challenges? What led or contributed to these? 7 16 Do these successes and challenges differ by group (females versus males; students, instructors, administrators or others at the institute; subject area) If so, how were these differences addressed during the course of the program? 7 17 If you could adjust one aspect of the LMI-COMET program, what would it be, and why? 7 18 Please provide any additional thoughts or feedback on the LMI-COMET program. Would you recommend that the program continues and/or expands either within existing institutes or externally to other institutes? Why or why not? In the same capacity or in an altered capacity (probe: such as)? 5 61 | USAID-LMI COMET EVALUATION REPORT USAID.ORG 2. FOCUS GROUP DISCUSSION PROTOCOLS FOCUS GROUP ACTIVITY & DISCUSSION PROTOCOL – INSTRUCTORS I. FGD Tracking Information FGD Group: □ Instructors □ Master/Lead Instructors FGD ID: Date: Facilitator: Notetaker: Translator: MLC Affiliation (if applicable), Location: Length of Recording (hours/minutes): File Name: 2. Introductions and Informed Consent Facilitator: Thank you for coming here today. We are grateful that you are giving us your time. First, let me introduce the team: [The facilitator, translator and notetaker should introduce themselves]. We are here on behalf of Social Impact and USAID to conduct research on the effects of the USAID- LMI-COMET project on your community. Before we begin, you should know that neither this research nor your answers to our questions will in any way determine if a project will be implemented here, continue to be here, or effect your ability to access or receive any services. The benefit of this research is to ensure that projects are well designed and work well together based on local people’s input. This discussion will take approximately one hour and a half. With your permission, we would like to audio record our conversation and take notes. The recordings and the notes will not be shared with anyone outside the research team and your name will be kept confidential. We hope this makes you feel comfortable to express your ideas freely. Your participation in this discussion is voluntary and so if you do not feel comfortable, then you are welcome to excuse yourself at any time during the research without any consequences to you. Do you have any questions? Do you agree to participate? Will you allow us to record the conversations? Thank you! USAID.ORG USAID-LMI COMET EVALUATION REPORT | 62 3. Participant Demographic Information Please ask each FGD participant to provide the following information: No. Name (First) Sex Area of instruction Years of employment at the institute LMI-COMET Participation (Type): LMI-COMET Participation (Years) 1 □ M □ F □ 1-2 □ 3-5 □ 6-9 □ 10-14 □ 15-19 □ 20+ □ Level 1 □ Level 2 □ Level 3 (Master) □ Other/No Level □ 0-6 months □ 6 months – 1 year □ 1 – 2 years □ 2 – 3 years □ 4+ years 2 □ M □ F □ 1-2 □ 3-5 □ 6-9 □ 10-14 □ 15-19 □ 20+ □ Level 1 □ Level 2 □ Level 3 (Master) □ Other/No Level □ 0-6 months □ 6 months – 1 year □ 1 – 2 years □ 2 – 3 years □ 4+ years 3 □ M □ F □ 1-2 □ 3-5 □ 6-9 □ 10-14 □ 15-19 □ 20+ □ Level 1 □ Level 2 □ Level 3 (Master) □ Other/No Level □ 0-6 months □ 6 months – 1 year □ 1 – 2 years □ 2 – 3 years □ 4+ years 4 □ M □ F □ 1-2 □ 3-5 □ 6-9 □ 10-14 □ 15-19 □ 20+ □ Level 1 □ Level 2 □ Level 3 (Master) □ Other/No Level □ 0-6 months □ 6 months – 1 year □ 1 – 2 years □ 2 – 3 years □ 4+ years 5 □ M □ F □ 1-2 □ 3-5 □ 6-9 □ 10-14 □ 15-19 □ 20+ □ Level 1 □ Level 2 □ Level 3 (Master) □ Other/No Level □ 0-6 months □ 6 months – 1 year □ 1 – 2 years □ 2 – 3 years □ 4+ years 6 □ M □ F □ 1-2 □ 3-5 □ 6-9 □ 10-14 □ 15-19 □ 20+ □ Level 1 □ Level 2 □ Level 3 (Master) □ Other/No Level □ 0-6 months □ 6 months – 1 year □ 1 – 2 years □ 2 – 3 years □ 4+ years 7 □ M □ F □ 1-2 □ 3-5 □ 6-9 □ 10-14 □ 15-19 □ 20+ □ Level 1 □ Level 2 □ Level 3 (Master) □ Other/No Level □ 0-6 months □ 6 months – 1 year □ 1 – 2 years □ 2 – 3 years □ 4+ years 8 □ M □ F □ 1-2 □ 3-5 □ 6-9 □ 10-14 □ 15-19 □ 20+ □ Level 1 □ Level 2 □ Level 3 (Master) □ Other/No Level □ 0-6 months □ 6 months – 1 year □ 1 – 2 years □ 2 – 3 years □ 4+ years 9 □ M □ F □ 1-2 □ 3-5 □ 6-9 □ 10-14 □ 15-19 □ 20+ □ Level 1 □ Level 2 □ Level 3 (Master) □ Other/No Level □ 0-6 months □ 6 months – 1 year □ 1 – 2 years □ 2 – 3 years □ 4+ years 10 □ M □ F □ 1-2 □ 3-5 □ 6-9 □ 10-14 □ 15-19 □ 20+ □ Level 1 □ Level 2 □ Level 3 (Master) □ Other/No Level □ 0-6 months □ 6 months – 1 year □ 1 – 2 years □ 2 – 3 years □ 4+ years 11 □ M □ F □ 1-2 □ 3-5 □ 6-9 □ 10-14 □ 15-19 □ 20+ □ Level 1 □ Level 2 □ Level 3 (Master) □ Other/No Level □ 0-6 months □ 6 months – 1 year □ 1 – 2 years □ 2 – 3 years □ 4+ years 12 □ M □ F □ 1-2 □ 3-5 □ 6-9 □ 10-14 □ 15-19 □ 20+ □ Level 1 □ Level 2 □ Level 3 (Master) □ Other/No Level □ 0-6 months □ 6 months – 1 year □ 1 – 2 years □ 2 – 3 years □ 4+ years 63 | USAID-LMI COMET EVALUATION REPORT USAID.ORG 4. FGD Exercise: Outcome Mapping Materials needed: 10-12 large sheets of paper (one per program change/outcome, with extra), pre-cut labels (each participant should have a group of all activities, with extra) Preparation: Hang 7 large sheets of paper, one per programmatic “change” 1. Improved Student Learning 2. Students Graduate with the Skills Needed to Work in STEM+AT 3. Improved Instructor Teaching 4. Improved Teacher Ability to Train Others in MS2W Approach 5. MS2W Approach Adopted by (Institute Name) Instructors and Administration 6. Improved Connections with Industry Partners 7. Increased Connections Among Regional Institutes Activity Introduction: The focus of our discussion today will be on teacher instruction and student learning within your institute. We will ask you questions on these topics directly and through an activity that we will do with the papers you see around the room. We will guide you through this activity, but if you have any questions or need clarification along the way, please do not hesitate to ask! First, I want to begin by asking you a few general questions: Questions 1. Please describe your classroom arrangement: seating, equipment, number of students, etc. What is your approach to lesson planning and classroom instruction? 2. Are you aware of the LMI-COMET program being implemented in your institute? If yes, please describe the key characteristics and objectives of the program. USAID.ORG USAID-LMI COMET EVALUATION REPORT | 64 As you can see, we have placed around the room 7 pieces of paper, each labeled with a “change” that the LMI-COMET program aims to create. For the next 5 minutes, please walk around the room and look at each of these intended changes. After 5 minutes, bring the group back together to discuss the following: Questions 1. Do you agree with all of the changes listed? Is there anything listed that you did not see or experience? 2. Do these changes differ by group (females versus males; students, instructors, administrators or others at the institute; subject area)? 3. Is there anything missing? Are there changes that you have seen or experienced that are not listed around the room? *If yes, write these on extra paper and hang on the wall 4. Are there any negative changes that you have seen or experienced since the start of the LMI-COMET program? *If yes, write these on extra paper and hang on the wall Following this discussion, distribute colored labels with LMI-COMET activities: a. Online Instructor Certification Course b. Application of Sourcebook Toolkit c. Sourcebook for Teachers d. Professional Development Workshops for Teachers e. Leadership Summits for MLCs f. Mentoring and Site Visits by EDC Staff g. Innovation Challenge h. Youth Initiated Activities i. Institutionalization j. Work-Based Learning k. MS2W Network Portal 65 | USAID-LMI COMET EVALUATION REPORT USAID.ORG Each label you have received names an activity of the LMI-COMET program. Please read through the activities, and for each one think about what “change” you feel it is most effective in achieving. For the next 10 minutes, please walk around the room and place each activity on the change you think it links most directly with. If you feel strongly that a certain activity links to more than one change, you may pick up extra labels in the front of the room. After 10 minutes, bring the group back together. Taking each piece of paper from the wall, look at the most frequently placed activities and discuss with participants why they feel these activities are effective in achieving the particular change: Outcome/Change Questions Improved Student Learning 5. Why do you think (x activity) helped to improve student learning? 6. Has the program had any effects (positive or negative) on student learning and experiences in the classroom? Do these effects differ by group (females versus males; subject area)? Can you think of any stories or examples that illustrate these effects? Students Graduate with the Skills Needed to Work in STEM+AT 7. Why do you think (x activity) helped students graduate with the skills needed to work in STEM+AT? 8. What skills do students need in order to find a position in the STEM+AT sectors and succeed over the long-term in that position? Do these needs differ by group (females versus males; subject area)? Do you think that the LMI-COMET program equips students with these skills? Why or why not? Improved Instructor Teaching 9. Why do you think (x activity) helped to improve instructor teaching? 10. Has the program had any effects (positive or negative) on instructor teaching and classroom practices? Do these effects differ by group (females versus males; subject area)? Can you think of any stories or examples that illustrate these effects? Improved Teacher Ability to Train Others in MS2W Approach 11. Why do you think (x activity) helped enable teachers to have the knowledge to train others in the MS2W approach? 12. In your opinion, will the following continue over the long-term in your institute without external support from EDC? Why or why not?: a. Use of active-learning methods by instructors currently engaged in LMI-COMET b. Spread of active-learning methods to new instructors in your institute who are not currently engaged in LMI-COMET USAID.ORG USAID-LMI COMET EVALUATION REPORT | 66 MS2W Approach Adopted by (Institute Name) Instructors and Administration 13. Why do you think (x activity) helped to improve instructor and administrator adoption of the LMI-COMET program? 14. Which departments (subject areas) or staff (administrators, instructors, etc.) have been the strongest promoters of the LMI-COMET program in your institute, and why? Who has been more reluctant to adopt the program, and why? Improved Connections with Industry Partners 15. Why do you think (x activity) helped to improve connections with industry partners? 16. Prior to the start of the LMI-COMET program, did your institute have relationships with local industry partners? If yes, please describe these relationships (probe: student internships; work-based learning; other activities). Have these relationships changed since the start of the program? If yes, please describe these changes (probe: increased access to and use of technology; increased opportunities for work-based learning and other professional development, both generally and for females) Increased Connections Among Regional Institutes 17. Why do you think (x activity) helped to increase connections among institutes across the Lower Mekong? 18. Has this regional participation impacted your engagement in program in any way? Have there been other results of these connections? 5. Closing Discussion Ask these final questions after each outcome/change paper is individually discussed: Questions 19. What do you think have been the most successful aspects of the LMI-COMET program? What do you think have been the major challenges (probe: has your institute experienced technology challenges that impacted implementation of the program)? Are these successes and challenges the same for all groups: females versus males; students, instructors, administrators or others at the institute; subject area? 20. Please provide any additional thoughts or feedback on the LMI-COMET program. Would you recommend that the program continues and/or expands either within existing institutes or externally to other institutes? Why or why not? In the same capacity or in an altered capacity (probe: such as)? Thank you! 67 | USAID-LMI COMET EVALUATION REPORT USAID.ORG 3.DIRECT OBSERVATION PROTOCOLS Observation Tracking Information Observation ID: Date: Facilitator: Translator: Institute Name, Location: Demographic Information Instructor Name(s): Class subject: Course level: ____________ Number of Females: Number of Males: Total students: ______________ Observation Information Observation Prompt Response Comments/Notes 1. How much of the class time is teacher-led versus student￾led? Total (min.) Teacher: Total (min.) Student: Teacher-led (minutes): Student-led (minutes): 2. Are students engaged in the material and classroom? (e.g. are students raising their hands to participate/respond) □ Yes □ Somewhat □ No 2a. Does this differ by group (female versus male) □ Yes □ Somewhat □ No Number of times female students called on: Number of times male students called on: 3. Do students appear bored or anxious? (i.e. identify active listeners – eye contact on instructor, not playing with personal items, e.g. cell phone) □ Yes □ Somewhat □ No 3a. Does this differ by group (female versus male) □ Yes □ Somewhat □ No At minute 20 of the class: At minute 45 of the class: Females actively listening = Females actively listening = Males actively listening = Males actively listening = USAID.ORG USAID-LMI COMET EVALUATION REPORT | 68 Females NOT actively listening = Females NOT listening = Males NOT actively listening = Males NOT listening = 4. Is the instructor physically engaged in the classroom? (e.g. does the instructor move around the classroom? Lecture from behind a desk?) □ Yes □ Somewhat □ No 5. Does the instructor make use of classroom materials? (e.g. use visual aids? Point out or pass round materials for students to see/review?) □ Yes □ Somewhat □ No 6. Are instructors clear in their teaching lessons and activities? (e.g. after an activity is explained, or command given, do students follow the prompt uniformly – implying comprehension)? □ Yes □ Somewhat □ No 7. Do instructors reference or use MS2W materials (e.g., MS2W portal)? □ Yes □ Somewhat □ No 8. Are computers or other technologies present in the classroom? If so, how (if at all) does the instructor incorporate technologies into their lesson? □ Yes □ Somewhat □ No 9. Do students use computers or other technologies as part of the class? □ Yes □ Somewhat □ No 9a. Does this differ by group (female versus male)? That is, do all students have equal access to and use of technologies during the class? □ Yes □ Somewhat □ No 10. Are lessons physically interactive? Are students moving around, talking, etc., as part of learning during the class? □ Yes □ Somewhat □ No 10a. Does this differ by group (female versus male) □ Yes □ Somewhat □ No 11. Do students work in groups? □ Yes □ Somewhat □ No 11a. If yes, what can you observe about the balance of who is most active within small groups? 69 | USAID-LMI COMET EVALUATION REPORT USAID.ORG 12. Does the instructor have an organized, detailed lesson plan (ask to review)? Does the lesson plan contain learning objectives? Does it differentiate based on learner needs? □ Yes □ Somewhat □ No 13. To what extent does the MS2W approach appear integrated in the classroom under observations? Does the approach appear to equally incorporate all students? Noting: gender, disability, markers of social class (e.g. items of value such as expensive phones, watches, apparel, etc.), ethnicity, etc. STUDENT QUESTIONS b. Is this your first time taking a class with (professor name)? c. Please describe some of the activities that you are asked to do in this class. Do you think that the activities help you to learn the course material? Do you think they help you to learn or develop any other skills? d. Do you do similar activities in other classes? (within this department? In classes in other departments?) If you could choose, would you want more of your classes taught using similar activities? (across all subject areas?) e. Do you think that male or female instructors are more likely to teach using activities? Why? Do you think male or female students enjoy learning through classroom activities more? Why? f. In your opinion, can classroom activities build skills that are important to the work you want to do in the future? If so, which skills? If not, what is most important to developing skills for work? USAID.ORG USAID-LMI COMET EVALUATION REPORT | 70 4.ONLINE SURVEY PROTOCOLS STUDENT/ ALUMNI SURVEY Thank you for visiting our survey page, we are grateful that you are giving us your time! This survey was created by Social Impact, a program evaluation firm, working with the United States Agency for International Development (USAID), the Education Development Center (EDC), and your university or college to better understand your experiences and the things you have learned in school. In particular, this evaluation is focused on the use of active learning methods in the classroom, and how this might affect job skills and employability. Your feedback is very important to us and will help us ensure that future university and college projects are well-designed and take into account student’s opinions. It will take approximately 10 minutes to complete the survey. This is not a test, so there no right or wrong answers. Your answers will remain confidential, which means that they will not be shared with other students, instructors, or others outside the evaluation team. Your personal information will also remain confidential. We hope this makes you feel comfortable to express your ideas freely. Thank you for completing our survey! Sincerely, Social Impact Evaluation Team 71 | USAID-LMI COMET EVALUATION REPORT USAID.ORG No. Students/Alumni i. Introduction: • At which university or college are you currently a student: o Select from [drop-down menu]: University of Economics—The University of Da Nang/ Hanoi University of Science and Technology/ Ho Chi Minh City University of Technology and Education/ Hue Industrial College/ Institute of Technology Cambodia/ Laos National Institute of Tourism and Hospitality/ Mahidol University/ Maptaphut Technical College/ National University of Laos/ University of Da Nang/ University of South-East Asia/ University of Technology Yatanarpon Cyber City/ Other- fill-in • What is your current age? o Select from: 18-19/20-24/25-29/30-35/35+ • What is your gender? o Select from: Female/ Male/ Prefer not to disclose/ Identify other than female or male • Which most accurately describes your field of study? o Select from: Accounting/ Business, Finance and Banking/ Chemistry/ Communications/ Economics/ Education/ Engineering/ Information Technology/ Statistics and Informatics/ Tourism, Hospitality and Food Technology/ Other- fill-in • Are you currently employed? o Select from: Yes/No • If you are employed, which of the following most accurately describes your employment status? o Select from: Full-time, one employer/ Full-time, multiple employers/ Part-time, one employer/ Part-time, multiple employers/ Self-employed/entrepreneur • If you are employed, what is your current field or occupation? o Select from: Accounting/ Business, Finance and Banking/ Chemistry/ Communications/ Economics/ Education/ Engineering/ Information Technology/ Statistics and Informatics/ Tourism, Hospitality and Food Technology/ Other- fill-in USAID.ORG USAID-LMI COMET EVALUATION REPORT | 72 1 • Are you aware of the USAID-LMI COMET program being implemented in your institution? o Select from: No awareness / Some awareness / Very aware • If you selected “some awareness” or “very aware” above, please indicate what you believe are the key objectives of the program (select all that apply): o Select from: Improved student learning/ Greater work experience and internship opportunities for students/ Students graduate with the skills needed to find a job/ Improved teaching methods/ Instructors and administrators promote active learning methods throughout the university or college/ Increased connections with industry partners/ Increased connections with other universities and colleges in the Lower Mekong region/ Other – fill in • If you selected “some awareness” or “very aware” above, please indicate what you believe are the key activities of the program (select all that apply): o Select from: Online instructor certification course/ Sourcebook for instructors/ Active learning methods applied in the classroom/ Innovation Challenge/ Work-based learning and/or internships/ MekongSkills2Work (MS2W) network portal/ YouRock/ Edmodo/ Other – fill in • If you selected “some awareness” or “very aware” above, please indicate if, in your opinion, the program has impacted instructor teaching and classroom practices at your university or college: o Select from: Negatively impacted / Somewhat negatively impacted / Neutral/No impact / Somewhat positively impacted / Positively impacted / Unsure • If you selected “some awareness” or “very aware” above, please indicate if, in your opinion, the program has impacted student learning and experiences in the classroom at your university or college: o Select from: Negatively impacted / Somewhat negatively impacted / Neutral/No impact / Somewhat positively impacted / Positively impacted / Unsure 2 • Thinking about the classes you take at your university or college, would you describe them as “active” (for example, the class uses technology, you work in groups, you give presentations, you engage in work-based learning, etc.) or “traditional” (for example, the instructor lectures and the students take notes)? o Select from: All of my classes are active, and none are traditional/ Most of my classes are active, and a few are traditional/ About half of my classes are active, and the other half traditional/ Most of my classes are traditional, and a few are active/ All of my classes are traditional, and none are active / Other -fill in 3 • What, if any, skills do active learning classes help you develop? (select all that apply): o Select from: Teamwork and/or ability to work in groups/ Communication/ Leadership/ Presentation and public speaking/ Networking and job search skills/ Technical skills for the position/ Writing skills/ English language skills/ Ability to use social media/ None of the above/ Other -fill in 73 | USAID-LMI COMET EVALUATION REPORT USAID.ORG SURVEY FOR INSTRUCTORS Thank you for visiting our survey page, we are grateful that you are giving us your time! This survey was created by Social Impact, a program evaluation firm, working with the United States Agency for International Development (USAID), the Education Development Center (EDC), and your university or college to better understand your experiences as an instructor. In particular, this evaluation is focused on a program known as USAID-LMI COMET, which promotes the use of active learning methods in the classroom, partnerships with industry, and other activities for students and instructors. The evaluation seeks to better understand how USAID-LMI COMET might impact instructor teaching, student learning, and employability of students. Your feedback is very important to us and will help us ensure that future university and college projects are well-designed and take into account instructor’s opinions. It will take approximately 10 minutes to complete the survey. This is not a test, so there no right or wrong answers. Your answers will remain confidential, which means that they will not be shared with students, instructors, administrators or others outside the evaluation team. Your personal information will also remain confidential. We hope this makes you feel comfortable to express your ideas freely. Thank you for completing our survey! Sincerely, Social Impact Evaluation Team 4 • In your opinion, which of these skills are most important for finding a job in your desired field and succeeding over the long-term in that job? (select all that apply): o Select from: Teamwork and/or ability to work in groups/ Communication/ Leadership/ Presentation and public speaking/ Networking and job search skills/ Technical skills for the position/ Writing skills/ English language skills/ Ability to use social media/ Other -fill in 5 • In your opinion, do you feel prepared to find a job in your desired field and succeed in that job over the long-term? (select all that apply): o Yes, I feel completely prepared / I feel somewhat prepared, but I would like to learn more soft skills (for example, teamwork, communication, leadership, etc.)/ I feel somewhat prepared, but I would like to learn more technical skills for my desired position/ I feel somewhat prepared, but I would like to learn more English language, social media and other non-technical skills/ No, I do not feel prepared at all USAID.ORG USAID-LMI COMET EVALUATION REPORT | 74 No. Instructors i Introduction: • At which university or college are you currently an instructor: o Select from [drop-down menu]: University of Economics—The University of Da Nang/ Hanoi University of Science and Technology/ Ho Chi Minh City University of Technology and Education/ Hue Industrial College/ Institute of Technology Cambodia/ Laos National Institute of Tourism and Hospitality/ Mahidol University/ Maptaphut Technical College/ National University of Laos/ University of Da Nang/ University of South-East Asia/ University of Technology Yatanarpon Cyber City/ Other- fill-in • What is your gender? o Select from: Female/ Male/ Prefer not to disclose/ Identify other than female or male • Which most accurately describes the field in which you primarily teach? o Select from: Accounting/ Business, Finance and Banking/ Chemistry/ Communications/ Economics/ Education/ Engineering/ Information Technology/ Statistics and Informatics/ Tourism, Hospitality and Food Technology/ Other- fill-in • How long have you been employed at the college or university where you are currently an instructor?: o Select from: 1-2 years/ 3-5 years/ 6-9 years/ 10-14 years/ 15-19 years/ 20 or more years • How long have you participated in the USAID-LMI COMET program?: o Select from: 0-6 months/ 6 months-1 year/ 1 – 2 years/ 2 – 3 years/ 4 or more years • What stage or level of training for USAID-LMI COMET have you attained?: o Select from: Level 1/ Level 2/ Level 3 (Master or Lead Instructor)/ Other -fill in 1 • Which of the following describes your classroom arrangement? (select all that apply): o Select from: Technology (for example, computers, projector, etc.) is present in the classroom/ Students are seated in rows/ Students are seated in groups/ Student seating is adjustable/ Class includes less than 20 students/ Class includes 21-49 students/ Class includes 50 or more students • How often do you use the following methods? o Select frequency using the following scale: Never/ Rarely/ Sometimes/ Often  Lecture/ Individual work/ Group work/ Project-based learning/ Work-based Learning/ Student presentations/ In-class work using text-based materials/ In-class work using online materials/ Other— fill in 2 • Are you aware of the USAID-LMI COMET program being implemented in your institution? (Skip logic to #2 if no awareness here) 75 | USAID-LMI COMET EVALUATION REPORT USAID.ORG o Select from: No awareness / Some awareness / Very aware • If you selected “some awareness” or “very aware” above, please indicate what you believe are the key objectives of the program (select all that apply): o Select from: Improved student learning/ Greater work experience and internship opportunities for students/ Students graduate with the skills needed to find a job/ Improved teaching methods/ Instructors and administrators promote active learning methods throughout the university or college/ Increased connections with industry partners/ Increased connections with other universities and colleges in the Lower Mekong region/ Other – fill in • If you selected “some awareness” or “very aware” above, please indicate what you believe are the key activities of the program (select all that apply): o Select from: Online instructor certification course/ Sourcebook for instructors/ Active learning methods applied in the classroom/ Professional Development Workshops for Teachers/ Leadership Summits for universities and colleges/ Mentoring and site visits by EDC staff/ Innovation Challenge/ Work-based learning and/or internships/ Youth-initiated activities/ Institutionalization/ MekongSkills2Work (MS2W) network portal/ YouRock/ Edmodo/ Other – fill in 3 • In your opinion, has the USAID-LMI COMET program impacted instructor teaching and classroom practices at your university or college?: o Select from: Negatively impacted / Somewhat negatively impacted / Neutral/No impact / Somewhat positively impacted / Positively impacted / Unsure 4 • In your opinion, has the USAID-LMI COMET program impacted student learning and experiences in the classroom at your university or college?: o Select from: Negatively impacted / Somewhat negatively impacted / Neutral/No impact / Somewhat positively impacted / Positively impacted / Unsure 5 • To what extent do you think the USAID-LMI COMET program has been adopted and incorporated by administrators, instructors, and others in your institution? o Select from: Not at all/ Few elements adopted / Some elements adopted / Well adopted/ Fully institutionalized 6 • In your opinion, will the following continue over the long-term in your university or college without external support (for example, USAID or EDC support)? o Select from the following scale: Definitely will continue/ Somewhat likely to continue/ Unlikely to continue / Definitely will not continue  Use of active-learning methods by instructors currently engaged in USAID-LMI COMET USAID.ORG USAID-LMI COMET EVALUATION REPORT | 76  Spread of active-learning methods to new instructors in your university or college who are not currently engaged in USAID-LMI COMET  Work-based learning and internships with local industry partners 7 • In five years, what do you think the USAID-LMI COMET program will look like at your university or college? o Select from: Will be less visible at my university or college / Will operate about as it does today / Will be adopted by more departments/ Will be adopted across all departments at my university or college 8 • The USAID-LMI COMET program includes participants from multiple countries and types of institutions across the Lower Mekong region, who you may have interacted with online or in workshops. Has this regional participation impacted your engagement in the program in any way? (select all that apply): o Select from: Regional network did not influence my decision to participate in USAID-LMI COMET / It was useful to share best practices with other instructors/ It was useful to practice English with other instructors/ It would be better to interact in my native language instead of English/ Regional network helped foster solidarity among instructors/ Receiving training by other instructors motivated me to participate in USAID￾LMI COMET 9 • In your opinion, to what extent has this regional participation in USAID-LMI COMET influenced the following: o For each response choose along the following scale: No effect/ A small effect/ Moderate effect/ Significant effect  The adoption and incorporation of the program by the administration of your university or college  The adoption and incorporation of the program by the instructors of your university or college, particularly those who have trained or otherwise interacted with instructors from other institutions  Support of the program by local government  Reach of the program in areas that are rural or otherwise harder to access 10 • What do you think have been the most successful outcomes of the USAID-LMI COMET program? o Please evaluate the following outcomes of the program based on the following scale: (1 = Very Successful; 2 = Somewhat successful; 3 = A little successful 4 = Not successful; 5 = Do not know)  Improved Student Learning  Students Graduate with the Skills Needed to Work in STEM+AT  Improved Instructor Teaching  Improved Instructor Ability to Train Others in MS2W Approach  MS2W Approach Adopted by Instructors at your University or College  MS2W Approach Adopted by the Administration at your University or College  Increased Connections Among Regional Institutions 77 | USAID-LMI COMET EVALUATION REPORT USAID.ORG 11 • What do you think have been the major challenges of the USAID-LMI COMET program? (select all that apply): o Select from: Difficult to complete trainings due to high workload/ Lack of support from the administration of my university or college/ Lack of incentives to engage in trainings/ Difficult to change methods of teaching to comply with MS2W approach/ Difficult to work with industry partners/ Materials not accessible because they are in English/ Lack of technology or materials needed to implement USAID-LMI COMET program/ Trainings not useful/ More trainings needed/Other—fill in 12 • In your opinion, what skills do students need to find a job and succeed in that job over the long-term, particularly in the STEM+AT sectors? (select all that apply): o Select from: Teamwork and/or ability to work in groups/ Communication/ Leadership/ Presentation and public speaking/ Networking and job search skills/ Technical skills for the position/ Writing skills/ English language skills/ Ability to use social media/ Other -fill in 13 • In your opinion, does USAID-LMI COMET equip students with the skills needed to find a job and succeed in that job over the long-term, particularly in the STEM+AT sectors? From the list below, select the skills that you believe the program currently addresses (select all that apply): o Select from: Teamwork and/or ability to work in groups/ Communication/ Leadership/ Presentation and public speaking/ Networking and job search skills/ Technical skills for the position/ Writing skills/ English language skills/ Ability to use social media/ Other -fill in 14 • Would you recommend that your university or college continues to implement the USAID-LMI COMET program? o Select from: Yes, the program should continue in the same capacity/ Yes, the program should expand within my college or university/ Yes, the program should expand to other colleges and universities/ Yes, the program should continue, but only with changes to how it is implemented/No, the program should not continue/ Other—fill in USAID.ORG USAID-LMI COMET EVALUATION REPORT | 78 ANNEX C: LIST OF RESPONDENTS MEKONG LEARNING CENTERS Country Institution Position Category Number of Respondents in Position Category Vietnam Ho Chi Minh City University of Technology and Education (HCMUTE) Administrator (Point of contact) 1 Vietnam Ho Chi Minh City University of Technology and Education (HCMUTE) Master Instructor 2 Vietnam Ho Chi Minh City University of Technology and Education (HCMUTE) Instructor Innovation Lead 2 Vietnam Ho Chi Minh City University of Technology and Education (HCMUTE) Work-Based Learning coordinator 1 Vietnam Ho Chi Minh City University of Technology and Education (HCMUTE) Technical Support 1 Vietnam Ho Chi Minh City University of Technology and Education (HCMUTE) Student Innovation Lead 1 Vietnam Ho Chi Minh City University of Technology and Education (HCMUTE) Youth Representative 1 Vietnam Hanoi University of Science and Technology (HUST) Administrator (Point of contact) 2 Vietnam Hanoi University of Science and Technology (HUST) Master Instructor 2 Vietnam Hanoi University of Science and Technology (HUST) Work-Based Learning coordinator 1 Vietnam Hanoi University of Science and Technology (HUST) Instructor Innovation Lead 2 Vietnam Hanoi University of Science and Technology (HUST) Student Innovation Lead 1 Vietnam Hanoi University of Science and Technology (HUST) Youth Representative 1 Vietnam Hue Industrial College (HueIC) Administrator (Point of contact) 1 Vietnam Hue Industrial College (HueIC) Administrator Assistant 1 Vietnam Hue Industrial College (HueIC) Master Instructor 5 Vietnam Hue Industrial College (HueIC) Instructor Innovation Lead 2 Vietnam Hue Industrial College (HueIC) Work-Based Learning Coordinator 1 Vietnam Hue Industrial College (HueIC) Technical Support 1 Vietnam Hue Industrial College (HueIC) Student Innovation Lead 1 79 | USAID-LMI COMET EVALUATION REPORT USAID.ORG Vietnam Hue Industrial College (HueIC) Youth Representative 1 Vietnam University of Economics - University of Danang (DUE) Administrator (Point of contact) 1 Vietnam University of Economics - University of Danang (DUE) Master Instructor 2 Vietnam University of Economics - University of Danang (DUE) Work-Based Learning Coordinator 1 Vietnam University of Economics - University of Danang (DUE) Instructor Innovation Lead 2 Vietnam University of Economics - University of Danang (DUE) Student Innovation Lead 1 Vietnam University of Economics - University of Danang (DUE) Technical Support 1 Vietnam University of Economics - University of Danang (DUE) Youth Representative 1 Vietnam University of Danang (UD) Administrator (Point of contact) 1 Vietnam University of Danang (UD) Admin Support 1 Vietnam University of Danang (UD) Lead Instructor 2 Vietnam University of Danang (UD) Work-Based Learning Lead 2 Vietnam University of Danang (UD) Work-Based Learning Coordinator 1 Vietnam University of Danang (UD) Instructor Innovation Lead 2 Vietnam University of Danang (UD) Student Innovation Lead 1 Vietnam University of Danang (UD) Technical Support 1 Vietnam University of Danang (UD) Youth Representative 1 Laos National University of Laos (NUOL) Administrator (Point of contact) 1 Laos National University of Laos (NUOL) Master Instructor 4 Laos National University of Laos (NUOL) Instructor Innovation Lead 2 Laos National University of Laos (NUOL) Work-Based Learning Coordinator 2 Laos National University of Laos (NUOL) Technical Support 2 Laos National University of Laos (NUOL) Student Innovation Lead 1 Laos National University of Laos (NUOL) Youth Representative 1 Thailand Mahidol University (MU) Administrator (Point of contact) 1 USAID.ORG USAID-LMI COMET EVALUATION REPORT | 80 Thailand Mahidol University (MU) Master Instructor 2 Thailand Mahidol University (MU) Instructor Innovation Lead 2 Thailand Mahidol University (MU) Work-Based Learning Coordinator 1 Thailand Mahidol University (MU) Technical Support 1 Thailand Mahidol University (MU) Student Innovation Lead 1 Thailand Mahidol University (MU) Youth Representative 1 Thailand Maptaphut Technical College College Advisor (Point of contact) 2 Thailand Maptaphut Technical College Administrator 1 Thailand Maptaphut Technical College Lead Instructor 2 Cambodia Institute of Technology Cambodia (ITC) Administrator (Point of contact) 1 Cambodia Institute of Technology Cambodia (ITC) Former Administrator* 1 Cambodia Institute of Technology Cambodia (ITC) Master Instructor 2 Cambodia Institute of Technology Cambodia (ITC) Work-Based Learning Coordinator 1 Cambodia Institute of Technology Cambodia (ITC) Instructor Innovation Lead 2 Cambodia Institute of Technology Cambodia (ITC) Student Innovation Lead 1 Cambodia Institute of Technology Cambodia (ITC) Technical Support 1 Cambodia Institute of Technology Cambodia (ITC) Youth Representative 3 Cambodia University of South-East Asia (USEA) Administrator (Point of contact) 1 Cambodia University of South-East Asia (USEA) Master Instructor 2 Cambodia University of South-East Asia (USEA) Work-Based Learning Lead 1 Cambodia University of South-East Asia (USEA) Instructor Innovation Lead 2 Cambodia University of South-East Asia (USEA) Technical Support 1 Cambodia University of South-East Asia (USEA) Youth Representative 1 Cambodia University of South-East Asia (USEA) Student Innovation Lead 1 Myanmar University of Technology (Yatanarpon Cyber City) (UTYCC) Administrator (Point of contact) 1 81 | USAID-LMI COMET EVALUATION REPORT USAID.ORG Myanmar University of Technology (Yatanarpon Cyber City) (UTYCC) Master Instructor 3 Myanmar University of Technology (Yatanarpon Cyber City) (UTYCC) Lead Instructor 1 Myanmar University of Technology (Yatanarpon Cyber City) (UTYCC) Work-Based Learning Coordinator 1 Myanmar University of Technology (Yatanarpon Cyber City) (UTYCC) Instructor Innovation Lead 2 Myanmar University of Technology (Yatanarpon Cyber City) (UTYCC) Student Innovation Lead 1 Myanmar University of Technology (Yatanarpon Cyber City) (UTYCC) Technical Support 1 Myanmar University of Technology (Yatanarpon Cyber City) (UTYCC) Youth Representative 2 LOCAL INDUSTRY PARTNERS COUNTRY INSTITUTION INDUSTRY PARTNER POSITION Vietnam University of Economics - University of Danang (DUE) CIMA Chief Representative Vietnam University of Economics - University of Danang (DUE) CPA Australia Senior Business Development Manager Vietnam University of Economics - University of Danang (DUE) Fast Accounting Representative Vietnam University of Danang (UD) Axxon Active Director Danang branch Vietnam University of Danang (UD) Code Engine Studio CEO Vietnam University of Danang (UD) Global Cybersoft Representative Laos National University of Laos (NUOL) Mitr Phol Group Company Manager of Human Recource Development Division Laos National University of Laos (NUOL) Lao Agro Tech co., Ltd Manager Laos National University of Laos (NUOL) Honda Laos Manager Thailand Mahidol University (MU) Dutchmill Corporation Associate Professor Cambodia Institute of Technology Cambodia (ITC) EZECOM Representative Cambodia Institute of Technology Cambodia (ITC) Denso Representative Cambodia Institute of Technology Cambodia (ITC) SCG Representative USAID.ORG USAID-LMI COMET EVALUATION REPORT | 82 Cambodia University of South-East Asia (USEA) Esoftix Director Cambodia University of South-East Asia (USEA) ACLEDA Plc. Representative Cambodia University of South-East Asia (USEA) Sokha Siem Reap Resort Representative Myanmar University of Technology (Yatanarpon Cyber City) (UTYCC) Marter Solution Representative Myanmar University of Technology (Yatanarpon Cyber City) (UTYCC) Unified Tech Representative Myanmar University of Technology (Yatanarpon Cyber City) (UTYCC) Koe Koe Tech Representative GOVERNMENT PARTNERS COUNTRY INSTITUTION INDUSTRY PARTNER POSITION Vietnam National Institute for Vocational Education and Training Vice Director Cambodia Directorate General of Higher Education, Cambodia Deputy Director General Myanmar Department of Higher Education, Myanmar Deputy Director Thailand National Science Technology and Innovation Policy Office (STI) Policy Researcher Thailand Office of the Vocational Education Commission (OVEC), Ministry of Education Senior Advisor Thailand Petroleum Institute of Thailand Senior Advisor MULTINATIONAL PARTNERS PARTNER POSITION Intel Technology Asia Pte Ltd. Former Director, Innovation Programs Corporate Affairs – Asia Cisco Systems (Thailand) Ltd. Program Manager The Asia Foundation Policy and Strategy Advisor Former Cisco Point of Contact Former POINT OF CONTACT 83 | USAID-LMI COMET EVALUATION REPORT USAID.ORG AWS ASEAN Technical Program Manager Texas A&M University Director, Engineering Entrepreneurship and Associate Professor of the Practice World Economic Forum Former USAID-LMI COMET Partnerships Manager, Project Lead, Future of Education, Gender, and Work YouRock.Jobs CEO USAID Former USAID-LMI COMET Contract Officer Representative USAID.ORG USAID-LMI COMET EVALUATION REPORT | 84 ANNEX D: DOCUMENTS REVIEWED 1. USAID/RDMA, COMET Contract AID-486-C-14-00001, October 2014 2. USAID/RDMA, COMET Contract AID-486-C-14-00001 Mod 1, December 2014 3. USAID/RDMA, COMET Contract AID-486-C-14-00001 Mod 2, April 2015 4. USAID/RDMA, COMET Contract AID-486-C-14-00001 Mod 3, September 2016 5. USAID/RDMA, COMET Contract AID-486-C-14-00001 Mod 4, June 2017 6. USAID/RDMA, COMET Contract AID-486-C-14-00001 Mod 5, September 2017 7. USAID/RDMA, COMET Contract AID-486-C-14-00001 Mod 6, June 2018 8. USAID COMET, Year 1 FY15 Annual Work Plan, February 2015 9. USAID COMET, Year 2 FY16 Annual Work Plan, August 2015 10. USAID COMET, Year 3 FY17 Annual Work Plan, September 2016 11. USAID COMET, Year 4 FY18 Annual Work Plan, June 2017 12. USAID COMET, Year 1 FY16 MEL Plan, July 2015 13. USAID COMET, Year 2 FY16 MEL Plan, July 2016 14. USAID COMET, Year 3 FY17 MEL Plan, October 2017 15. USAID COMET, Year 1 Annual Report, November 2015 16. USAID COMET, Year 2 Annual Report, November 2016 17. USAID COMET, Year 3 Annual Report, November 2017 18. USAID COMET, Year 4 Annual Report, November 2018 19. USAID COMET, Year 1 Q1 Quarterly Report, January 2015 20. USAID COMET, Year 1 Q2 Quarterly Report, April 2015 21. USAID COMET, Year 1 Q3 Quarterly Report, August 2015 22. USAID COMET, Year 2 Q1 Quarterly Report, January 2016 23. USAID COMET, Year 2 Q2 Quarterly Report, April 2016 24. USAID COMET, Year 2 Q3 Quarterly Report, July 2016 25. USAID COMET, Year 3 Q1 Quarterly Report, January 2017 26. USAID COMET, Year 3 Q2 Quarterly Report, May 2017 27. USAID COMET, Year 3 Q3 Quarterly Report, July 2017 28. USAID COMET, Year 4 Q1 Quarterly Report, January 2018 29. USAID COMET, Year 4 Q2 Quarterly Report, April 2018 30. USAID COMET, Lessons Learned and Proposed Adjustments, October 2016 31. USAID COMET, Press Release: Instructor Training Pilot Success Story, Date Unknown 32. USAID COMET, Press Release: Pioneers, Part III Public Sector, p 420, International Partnerships. Date Unknown. 33. YSEALI/USAID, Press Release: USAID and YSEALI Promote Youth Innovation to Tackle ASEAN Food Security, July 2016 34. USAID COMET, Fact Sheet: Engaging Youth as Partners, May 2018 35. USAID COMET, Fact Sheet: USAID COMET, May 2018 36. USAID COMET, Fact Sheet: MekongSkills2Work Training Module, August 2016 37. USAID COMET, Fact Sheet: Industry Partnerships, September 2016 38. USAID LMI, Fact Sheet: Mekong Learning Centers, Date Unknown 39. USAID LMI, Fact Sheet: MekongSkills2Work Experiential Learning in the Workplace, January 2017 40. USAID LMI, Fact Sheet: USAID – LMI COMET Briefer, June 2018 85 | USAID-LMI COMET EVALUATION REPORT USAID.ORG 41. USAID LMI, Fact Sheet: USAID – LMI COMET Overview, January 2017 42. USAID COMET, GYES Youth Employment Survey, February 2016 43. USAID COMET, GYES Youth Employment Baseline Report, March 2018 44. USAID COMET, Online Learning Team Survey, November 2016 45. USAID COMET, Online Learning Team Response Summary, January 2017 46. USAID COMET, Portal User Experience Survey: Questions and Responses, February 2017 47. USAID COMET, Portal User Experience Survey: Results Summary, February 2017 48. USAID COMET, Quality Standards Self-Assessment Tool, February 2017 49. USAID COMET, Quality Standard Self-Assessment Survey Report, October 2017 50. USAID COMET, Infographic: List of Mekong Learning Centers, November 2017 51. USAID COMET, Regional STEM+AT Labor Market Assessment Report, February 2016 52. USAID COMET, Regional Baseline STEM+AT Labor Market Assessment, February 2015 53. USAID COMET, Grant Phase 1 Review Summary, Date Unknown USAID.ORG USAID-LMI COMET EVALUATION REPORT | 86 ANNEX E: DISCLOSURE OF CONFLICT OF INTEREST FORMS 87 | USAID-LMI COMET EVALUATION REPORT USAID.ORG USAID.ORG USAID-LMI COMET EVALUATION REPORT | 88 89 | USAID-LMI COMET EVALUATION REPORT USAID.ORG ANNEX F: DATA DISAGGREGATION FROM SURVEYS TABLE 3. IMPORTANCE OF SOFT SKILLS FOR EMPLOYMENT (STUDENT SEX DISAGGREGATION) SOFT SKILL FEMALE STUDENTS MALE STUDENTS DID NOT DISCLOSE # % # % # % Teamwork and/or ability to work in groups 33 83% 32 73% 1 100% 66 Communication 29 73% 27 61% 1 100% 57 English language skills 26 65% 30 68% 0 0% 56 Presentation and public speaking 26 65% 18 41% 0 0% 44 Technical skills for the position 17 43% 25 57% 1 100% 43 Leadership 18 45% 23 52% 1 100% 42 Ability to use social media 15 38% 14 32% 0 0% 29 Networking and job search skills 12 30% 14 32% 0 0% 26 Writing 15 38% 6 14% 0 0% 21 N= 85 students who responded to this survey question (44 male students, 40 female students, and 1 who did not disclose) TABLE 4. IMPORTANCE OF SOFT SKILLS FOR EMPLOYMENT (INSTRUCTOR SEX DISAGGREGATION) SOFT SKILL FEMALE INSTRUCTORS MALE INSTRUCTORS TOTAL # % # % Teamwork and/or ability to work in groups 14 82% 26 100% 40 Communication 15 88% 25 96% 40 English language skills 16 94% 23 88% 39 Presentation and public speaking 15 88% 22 85% 37 Technical skills for the position 12 71% 18 69% 30 Leadership 13 76% 20 77% 33 Ability to use social media 12 71% 16 62% 28 Networking and job search skills 14 82% 20 77% 34 Writing 11 65% 14 54% 25 N= 43 instructors who responded to this survey question (26 male instructors and 17 female instructors) USAID.ORG USAID-LMI COMET EVALUATION REPORT | 90 TABLE 5. STUDENT ACTIVITY AWARENESS (STUDENT SEX DISAGGREGATION) FEMALE STUDENTS MALE STUDENTS DID NOT DISCLOSE TOTAL # % # % # % No Awareness 15 34% 25 46% 1 100% 41 Some Awareness 22 50% 20 37% 0 0% 42 Very Aware 7 16% 9 17% 0 0% 16 N= 99 students who responded to this survey question (54 male students, 44 female students, and 1 who did not disclose) TABLE 6. IMPACT ON TEACHING PRACTICES (STUDENT SEX DISAGGREGATION) FEMALE STUDENTS MALE STUDENTS TOTAL # % # % Negatively Impacted 0 0% 0 0% 0 Somewhat Negatively Impacted 0 0% 2 5% 2 Unsure 6 19% 6 17% 12 Neutral / No Impact 2 6% 0 0% 2 Somewhat Positively Impacted 8 25% 11 31% 19 Positively Impacted 16 50% 17 47% 33 N = 68 students who responded to this survey question (36 male students and 32 female students) TABLE 7. IMPACT ON TEACHING PRACTICES (INSTRUCTOR SEX DISAGGREGATION) FEMALE INSTRUCTORS MALE INSTRUCTORS TOTAL # % # % Negatively Impacted 0 0% 2 7% 2 Somewhat Negatively Impacted 1 6% 1 4% 2 Unsure 0 0% 0 0% 0 Neutral / No Impact 1 6% 0 0% 1 Somewhat Positively Impacted 2 12% 6 22% 8 Positively Impacted 13 76% 18 67% 31 N = 44 instructors who responded to this survey question (27 male instructors and 17 female instructors) 91 | USAID-LMI COMET EVALUATION REPORT USAID.ORG TABLE 8. INSTRUCTOR CHALLENGES (INSTRUCTOR SEX DISAGGREGATION) CHALLENGES FEMALE INSTRUCTORS MALE INSTRUCTORS TOTAL # % # % Difficult to complete trainings due to high workload 12 71% 14 54% 26 Lack of incentives to engage in trainings 9 53% 15 58% 24 Difficult to work with industry partners 7 41% 13 50% 20 More trainings needed 5 29% 10 38% 15 Lack of technology or materials needed to implement USAID LMI-COMET program 4 24% 9 35% 13 Lack of support from the administration of my university or college 4 24% 7 27% 11 Difficult to change methods of teaching to comply with MS2W approach 2 12% 6 23% 8 Materials not accessible because they are in English 1 6% 3 12% 4 N = 43 instructors who responded to this survey question (26 male instructors and 17 female instructors) TABLE 9. SUSTAINABILITY OF ACTIVE LEARNING METHODS BY CURRENT INSTRUCTORS (INSTRUCTOR SEX DISAGGREGATION) Use of active-learning methods by instructors currently engaged in USAID LMI-COMET FEMALE INSTRUCTORS MALE INSTRUCTORS TOTAL # % # % Definitely will continue 10 59% 19 70% 29 Somewhat likely to continue 6 35% 5 19% 11 Unlikely to continue 0 0% 2 7% 2 Definitely will not continue 1 6% 1 4% 2 N = 44 instructors who responded to this survey question (27 male instructors and 17 female instructors) TABLE 10. SPREAD OF ACTIVE LEARNING METHODS TO NEW INSTRUCTORS (INSTRUCTOR SEX DISAGGREGATION) Spread of active-learning methods to new instructors in your university or college who are not currently engaged in USAID LMI-COMET FEMALE INSTRUCTORS MALE INSTRUCTORS TOTAL # % # % Definitely will continue 8 47% 15 56% 23 Somewhat likely to continue 7 41% 10 37% 17 Unlikely to continue 0 0% 2 7% 2 Definitely will not continue 2 12% 0 0% 2 N = 44 instructors who responded to this survey question (27 male instructors and 17 female instructors) USAID.ORG USAID-LMI COMET EVALUATION REPORT | 92 TABLE 11. EFFECTS OF REGIONAL PARTICIPATION ON PROGRAM ADOPTION BY INSTRUCTORS (INSTRUCTOR SEX DISAGGREGATION) Effect of regional participation on the adoption and incorporation of the program by the administration of your university or college FEMALE INSTRUCTORS MALE INSTRUCTORS TOTAL # % # % Significant Effect 10 59% 10 37% 20 Moderate Effect 4 24% 10 37% 14 A Small Effect 3 18% 7 26% 10 No Effect 0 0% 0 0% 0 N=44 instructors who responded to this survey question (27 male instructors and 17 female instructors) TABLE 12. EFFECTS OF REGIONAL PARTICIPATION ON PROGRAM ADOPTION BY INSTRUCTORS WHO TRAINED ELSEWHERE (INSTRUCTOR SEX DISAGGREGATION) Effect of regional participation on the adoption and incorporation of the program by the instructors of your university or college, particularly those who have trained or otherwise interacted with instructors from other institutions FEMALE INSTRUCTORS MALE INSTRUCTORS TOTAL # % # % Significant Effect 11 65% 8 31% 19 Moderate Effect 5 29% 15 58% 20 A Small Effect 0 0% 3 12% 3 No Effect 1 6% 0 0% 1 N=43 instructors who responded to this survey question (26 male instructors and 17 female instructors) TABLE 13. EFFECTS OF REGIONAL PARTICIPATION ON PROGRAM SUPPORT BY LOCAL GOVERNMENT (INSTRUCTOR SEX DISAGGREGATION) Effect of regional participation on support of the program by local government FEMALE INSTRUCTORS MALE INSTRUCTORS TOTAL # % # % Significant Effect 4 24% 0 0% 4 Moderate Effect 5 29% 11 42% 16 A Small Effect 4 24% 12 46% 16 No Effect 4 24% 3 12% 7 N=43 instructors who responded to this survey question (26 male instructors and 17 female instructors)