This publication was produced for the United States Agency for International Development. It was prepared by DevTech, Inc., under the LAC Education Support Contract (7200AA18M00024). This report was authored by Elizabeth Emanuel, William Dant, Georgia Sinclair, Travis Reid, Miguel Nunez, Carla Paredes, and Jose Pineda. USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION JULY 2022 DME PHOTOGRAPHY USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION LAC Education Support Contract, 7200AA18M00024 Evaluation duration: September 2021 - June 2022 First draft submitted June 6, 2022 Revised draft submitted July 8, 2022 DISCLAIMER This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents of this report are the sole responsibility of DevTech Systems, Inc. and do not necessarily reflect the views of USAID or the United States Government. i | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV TABLE OF CONTENTS ACRONYMS III ACKNOWLEDGMENTS III EXECUTIVE SUMMARY IV I. INTRODUCTION 1 1.1. EVALUATION PURPOSE 1 II. BACKGROUND 3 2.1. PROGRAM OVERVIEW 3 2.2. PROGRAM OBJECTIVES 3 2.3. THEORY OF CHANGE 4 III. EVALUATION DESCRIPTION 5 3.1. EVALUATION OBJECTIVES AND RESEARCH QUESTIONS 5 IV. METHODOLOGY 7 4.1. EVALUATION SYSTEMS APPROACH 7 4.2. VARIABLES 8 4.3. INSTRUMENTS 8 4.4. SAMPLE 10 4.5. DATA COLLECTION AND ANALYSIS 11 4.6. LIMITATIONS 13 V. EVALUATION FINDINGS 13 5.1. RQ1: CHANGE IN SYSTEMS WITHIN HEIS AND IMPROVEMENTS IN THEIR INSTITUTIONAL PERFORMANCE 16 5.2. SUSTAINABILITY AND INSTITUTIONALIZATION OF ADVANCE IN HEIS 30 5.3. ENABLING AND INHIBITING FACTORS 42 5.4. INDICATORS OF CHANGE 45 VI. DISCUSSION AND CONCLUSIONS 50 VII. RECOMMENDATIONS FOR FUTURE PROGRAMMING 52 ANNEXES 56 ANNEX 1. DETAILED PROGRESS MARKERS BY HEI AND KEY AREAS OF SUSTAINABILITY AND INSTITUTIONALIZATION 56 ANNEX 2. INDIVIDUAL OUTCOME MAP TABLES BY HEI 62 ANNEX 3. DESCRIPTION OF HIGHER EDUCATION INSTITUTIONS THAT PARTICIPATED IN THE EVALUATION 67 ANNEX 4. CONSENT FORM TO PARTICIPATE IN KIIS/FGDS 69 ANNEX 5. DATA COLLECTION INSTRUMENTS 70 USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | ii LIST OF FIGURES Figure 1: Summary of Level of Sustainability and Institutionalization Across HEIs viii Figure 2: Participants/Faculty and Administrators of HEIs Perception of Quality of Advance Services 14 Figure 3: Percentage of HEI Participants That Consider That Tools and Techniques Contributed the Curriculum Design 19 Figure 4: Percentage of HEI Participants That Consider That Advanced Positively Influenced Development of the Staff 20 Figure 5: Summary of Sustainability and Institutionalization of Key Elements of Advance 32 Figure 6: Level of Institutionalization and Sustainability by HEI and by Key Areas 33 LIST OF TABLES Table 1: Data Collection Participants 11 Table 2: HEI Self-Reported Average Scores for Labor-Bridging Services, Admissions, and Recruitment 15 Table 3: Associate Degree Programs Developed or Revised 16 Table 4: Advance Students’ Profile 24 Table 5: Scholarships Provided by Advance and Retention Rates 27 Table 6: Factors Influencing Retention Rates 28 Table 7: Progress Markers - Curriculum and Pedagogy Institutionalization and Sustainability 35 Table 8: Progress Markers - Professional Development Institutionalization and Sustainability 37 Table 9: Progress Markers - Labor-Bridging Services Institutionalization and Sustainability 38 Table 10: Progress Markers - Admissions and Recruitment Institutionalization and Sustainability 40 Table 11: Progress Markers - Access to Scholarships Institutionalization and Sustainability 41 Table 12: Key Intermediate Indicators of Change 47 iii | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV ACKNOWLEDGMENTS The “USAID/Jamaica Advance Final Performance Evaluation” is the product of a collaboration between USAID/Washington and DevTech Systems Inc. The following people played key roles in the provision of advice for this study: • Nathaniel Haight, Education Advisor/COR • Michael Lisman, Education Team Lead • Jamie Salazar, Youth and Workforce Development Specialist for USAID/LAC • Deborah Greebon, Senior Monitoring, Evaluation, and Learning Advisor for USAID/DDI • Olga Merchan, Senior Youth and Workforce Development Advisor for USAID/DDI The authors wish to thank USAID for commissioning the USAID/Jamaica Advance Final Performance Evaluation, implemented under the USAID LAC Support Contract GS-10F-0048L/7200AA18M00024. USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | iv ACRONYMS APL Accreditation of Prior Learning ASc. Associate Degree in Science CAP Career Advancement Program CAPE Caribbean Advanced Proficiency Examination CARICOM Caribbean Community and Common Market CCCJ Council of Community Colleges of Jamaica DDI/EDU Education Team of USAID’s Bureau for Development, Democracy, and Innovation EDPT Entertainment Design, Production, and Technology Program EEM Entertainment and Events Management ET Evaluation Team FGD Focus Group Discussion GOJ Government of Jamaica HE Higher Education HEI Higher Education Institution HOD Head of Department ICFS Image Consulting and Fashion Styling ICT Information and Communication Technology IR Intermediate Result KII Key Informant Interview KPC Knockalva Polytechnic College LAC USAID’s Bureau for Latin America and the Caribbean MOEYI Ministry of Education, Youth, and Information MOU Memorandum of Understanding NVQ-J Jamaican National Vocational Qualification of Jamaica Note: NVQ-J signals that the holder has a certain set of skilled competencies and is recognized across Jamaica and the Caribbean Community and Common Market. There are six levels of achievement for students in the National Vocational Qualification system NVQ-L 1-6. PATH Programme of Advancement Through Health and Education RQ Research Question RSD/EDU Regional Sustainable Development / Education Office RWDP Regional Workforce Development Program SEL Social and Emotional Learning SOW Statement of Work ToC Theory of Change TTE Tertiary Technical Education Note: 2-year technical course that results in NVQ-L levels 3 through 5, roughly equivalent to an associate degree. v | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV TVET Technical and Vocational Education and Training Note: training in the practical skills, attitudes, and knowledge relating to technical professions in various economic sectors. UCJ University Council of Jamaica UNESCO United Nations Educational, Scientific, and Cultural Organization U.S. United States USAID United States Agency for International Development USG U.S. Government UTECH University of Technology, Jamaica UWI University of the West Indies VTDI Vocational Training Development Institute USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | vi EXECUTIVE SUMMARY This document outlines the results of a final performance evaluation of the United States Agency for International Development (USAID) Office of Education in Latin America and the Caribbean (LAC/RSD/EDU) USAID-Advance Program in Jamaica, known in Spanish as Avanza. The evaluation was conducted by DevTech Systems Inc. over the period of September 2021 to May 2022. Implemented by FHI 360, USAID-Advance is a seven-year workforce development activity (September 2015 – September 2022) targeting the Dominican Republic, Guatemala, Honduras, and Jamaica to provide or improve local higher education institutions’ (HEIs) capacity to recruit, support, bring to graduation, and place in employment disadvantaged students in career sectors identified as open to further employment. The purpose of the evaluation was to analyze Jamaica as a case study to 1) assess the influence and long￾term sustainability of the Advance Program’s interventions on the participating Jamaican HEIs’ performance and the institutionalization of the various outputs of the program, including degree programs developed, approach to curriculum development among other areas; and 2) identify enabling and inhibiting contextual and institutional factors associated with the HEIs’ performance, using indicators of change. The evaluation analyzes both objectives focusing on five key areas of change for Advance: curriculum and pedagogy, professional development, labor bridging services, admissions and recruitment, and access to scholarships. The evaluation team (ET) conducted the study one year before the conclusion of the program in order to generate findings and identify lessons learned that can inform the design of further workforce readiness projects in the region and beyond. The primary audiences of the evaluation are the USAID LAC/RSD/EDU, the USAID Bureau for Development, Democracy, and Innovation Center for Education (DDI/EDU), and other USAID decision-makers, as well as USAID’s implementing partners. METHODOLOGY The evaluation team designed the study with a systems lens in mind. Using a systems approach, the ET focused on understanding the operation and performance of the HEIs as a system, including the actors involved, their roles and responsibilities, and what activities Advance has generated to influence their operation. The ET used outcome mapping (OM) as the main methodology to assess the changes in institutional performance across HEIs, focusing on identifying and assessing changes in the behavior of HEIs with which Advance works directly. OM does not seek to prove causality or attribution for changes but attempts to show logical linkages between the changes and Advance activities, thereby enabling the ET to understand the program’s contributions. The team utilized mixed-methods, including the following: 1) desk review; 2) key informant interviews (KIIs); 3) focus group discussions (FGDs); and 4) web-based surveys. The ET also led a two-hour theory of change (ToC) workshop with USAID and FHI 360 staff, with the aim of gaining a deeper understanding of Advance’s ToC, causal mechanisms underlying critical assumptions, expected results in the systems, and system-level interventions, as well as any discrepancies between planned actions and actual implementation. The team selected respondents using purposive as well as snowball sampling. The evaluation incorporated a gender responsive design that included data collection with men and women. The ET integrated informed consent protocols into all KII and FGD protocols. Participants included informants—leadership, faculty, administrators, and students—from the four targeted HEIs as well as vii | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV relevant representatives of Government of Jamaica (GOJ) ministry stakeholders and representatives from targeted employing industries. A total of 51 participants from four HEIs were interviewed in this study. The HEIs included the University of Technology (UTECH), Vocational Training Development Institute (VTDI), Knockalva Polytechnic College (KPC), and Knox Community College. To analyze results, the ET employed data triangulation with qualitative and quantitative data first analyzed independently, in parallel, and then cross-verified for validity. The themes generated through this analysis were then used to identify and present the key findings highlighted in the report. As with all evaluations, this report must be read with limitations in mind, specifically restrictions imposed by COVID-19. Due to the COVID-19 pandemic, the entire education system in Jamaica pivoted online from March 2020 to April 2022. During data collection, many HEIs were still operating virtually. All the field work for this evaluation was conducted through online surveys and virtual, Zoom-based FGDs and KIIs. 1 The evaluation generated several findings, conclusions, and recommendations across four major research questions, which are presented and discussed next. KEY FINDINGS AND CONCLUSIONS RQ1. What are the changes in systems within HEIs and improvements in their institutional performance? Advance strengthened HEIs though the development and improvement of six associate degree programs aligned to the needs of the labor market, and built the capacity of HEIs in the areas of curriculum design and pedagogy. HEIs are better positioned to provide market￾relevant education for youth. During the development of new courses, both UTECH and VTDI faculty and staff acquired curriculum development skills, competencies, and expertise in the development and strengthening of their programs. Advance did not work with the community college staff during the development of new courses, as Advance developed the programs for the Council of Community Colleges of Jamaica (CCCJ) directly. However, the CCCJ passed the associate degree programs to their colleges for delivery, which also led to the dissemination of training related specifically to the new curricula and approaches for teaching and experiential learning related to the curricula. Advance supported the development and review of curricula, and also introduced new and modern approaches to experiential learning—linking industry and academics with pedagogy focusing on hands-on learning, which had the added advantage of enabling students to interface directly with industry as part of their learning. Advance provided professional development opportunities for HEIs and reached faculty and staff beyond those assigned to the Advance Program, allowing HEIs to provide overall improved services for youth. Some of the workshops provided by Advance were open to all faculty and staff of the HEIs and were not limited to the faculty that taught Advance-supported courses. Lecturers considered that the training content was relevant to take back into the classroom and transfer to students. For example, the course in social and emotional learning (SEL) was new to all HEIs. Advance included a train-the-trainer component, which was key to institutionalizing this training across 1 Limitations and mitigation measures are described in the Methodology section. USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | viii various departments within the HEIs. These institutions are now better prepared to support students at the HEIs, and not only those that participate in Advance Programs. Following Advance support for professional development, VTDI faculty developed and implemented a student advisement center, designed as a safe space for students to allow them to engage in discussions with lecturers. Admissions processes did not change, as HEIs reported that they already had good recruitment and selection practices in place. However, Advance support better enabled HEIs to undertake assessments of vulnerable youth and determine their suitability for higher education. The program also influenced the profile of students for admissions, as in many cases some of the students accepted to the Advance Program would not have normally matriculated. HEIs also improved their marketing and targeting strategies to disadvantaged youth. The weakest area appeared to be in labor-bridging services. This area was new for all institutions in the study, so the learning curve may have been higher than in other targeted areas. This deficiency may also be due in part to the limitations imposed by a fragile labor market under the COVID-19 pandemic. RQ2. What is the level of sustainability and institutionalization2 of Advance in participating HEIs? Overall, as part of its key informant interviews the evaluation team found that all areas are, at least, at an emergent level of institutionalization. The majority of changes introduced by Advance in methodologies, approaches, and techniques are either established or advanced, with some still in early stages of planning. It is expected that some of these changes will continue after the program ends. The level of institutionalization and sustainability of Advance in participating HEIs varies both by key area and by HEI, with some common trends. • Advance made available six new or revised associate degree programs that are fully aligned with the needs of the labor market in areas where there were existing gaps. These programs are already fully institutionalized across HEIs in Jamaica, which means that even after the Advance Program has ended, these courses will continue to be offered. 2 The evaluation team classified institutionalization and sustainability by four levels, from latent to advanced, as described in Section V.2. FIGURE 1: SUMMARY OF LEVEL OF SUSTAINABILITY AND INSTITUTIONALIZATION ACROSS HEIS ix | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV • Training opportunities provided for staff have been passed on to students via course delivery in all evaluated HEIs. The community colleges created communities of practice among themselves to share information and best practices in course delivery. • All HEIs have already taken steps to couple academics with labor-bridging services in courses of study outside of Advance, although the level of institutionalization is still emergent, as interactions between HEIs and the private sector were limited, and more work is needed to link youths and HEIs with the private sector. • The system already had adequate admissions and recruitment processes in place that did not change. However, the system is now able to undertake assessments of vulnerable youth and determine their suitability for higher education. All HEIs have started to utilize the marketing and recruitment approaches of Advance to target disadvantaged youth in their programs. • Most HEIs recognized the value and benefit of providing access to scholarships, and a few HEIs are implementing mechanisms to continue providing financial aid using or adopting different approaches and developing their own scholarship programs. RQ3. What contextual and institutional factors enabled or inhibited the application of an institutional capacity-building/performance improvement approach? The enabling contextual factors influencing performance and improvement in the HEIs that the evaluation team identified are: 1. Enabling contextual factors. The main enabling contextual factor that the ET identified is Advance’s focus on labor market placement aligned with existing evidence of key areas with high demand for jobs. 2. Enabling institutional factors. HEIs were very appreciative of FHI 360’s consistent supervision and monitoring of program progress on-campus and the curriculum design and capacity-building initiatives—with their direct engagement of higher education (HE) faculty and staff, enhanced by qualified outside experts and study tours. Overall, HEIs mentioned teamwork between FHI 360 and participating HEIs as a valuable approach. Additionally, there was an overall commitment by HEIs to enable the Advance Program to be successful, which should not be taken for granted in a program. 3. Inhibiting contextual factors. Beyond the challenges created by the COVID-19 pandemic, selecting students for scholarships and reconciling the focus on disadvantaged youth and college readiness was also a challenge. Without the soft skills, self-awareness, self-confidence, and ability to communicate effectively in the HE settings, these students’ abilities to engage and learn successfully were hampered. Outreach to the labor market network was also challenging and found to be weaker than expected. 4. Inhibiting institutional factors. The scholarship and wraparound systems initially created for these socially and economically disadvantaged youth did not at times fully provide the support needed (expected student commitment, improved soft skills, realistic funding and disbursement USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | x of scholarship benefits, or resit testing fees). Once recognized, FHI 360 and the participating HEIs began to address these problems at the time of this study, and an improvement took place. RQ4. What are the indicators of change that can lead to anticipated long-term outcomes around institutional strengthening? The evaluation team looked at intermediate indicators across each of this study’s five key areas to measure change that can lead to anticipated long-term outcomes around institutional strengthening. Indicators of change analyzed include the percentage of students completing the degree program, the percentage of students requiring funding to complete their program due to failing one or more courses, the percentage of students dropping out of the program, the percentage increase in vulnerable students accessing jobs, and the percentage of students moving on to a degree program following completion of Advance, among others. However, HEIs were unable to provide sufficient data to fully undertake this assessment. Quantitative data such as change in career completion rates, increase in vulnerable students accessing jobs, or percentage of students moving to a degree program was limited across all HEIs primarily because HEIs did not collect data on the Advance Program in this manner. Whilst it was clear that Advance established reporting requirements, data available was activity-based as opposed to outcome-based. Notwithstanding, all HEIs indicated that there were increases in the number of disadvantaged students admitted to their degree programs, bolstered by the scholarship component of the Advance Program, enabling disadvantaged youth that would not have been able to access and afford the tuition and related cost for higher education—a key objective of the Advance Program. The completion rate of these students, however, was considered unsatisfactory by most participating HEIs in the study, as mentioned above. According to the ET findings described under RQ1 and RQ2, positive changes to curriculum and pedagogy occurred across all the HEIs in the study, particularly those that worked more closely with the FHI 360 team and experts, where needed. (Participating community colleges already had such courses developed through a previous initiative.) However, Advance’s outcomes were observed across all participating HEIs, as described above. The establishment of student advisement centers and more student-centric approaches that catered to the needs of their students was also noted as significant. All participating HEIs reflected evidence of new policies, guidelines, and procedures, thanks to the Advance Program. This was particularly observed in informal changes to curriculum and professional development and recruitment. The somewhat early timing of this evaluation limited data and information on the success of the labor-bridging activities since that component of Advance in Jamaica is still underway. Some HEIs also indicated the need for assistance in gathering data and information toward determining final program outcomes. All HEIs demonstrated challenges in evaluating student transitions to industry or entrepreneurship, another impact of the COVID-19 pandemic. Across the five areas of the Advance Program, there was clear evidence of the institutionalization and sustainability of various components and sub-components of the Advance Program, which is the main indicator of change observed during this evaluation, as evidenced by the findings under xi | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV RQ2 in Section V.2. When considering all sub-areas, the ET found that UTECH institutionalized3 13 sub￾components, VTDI 11, Knockalva 18, and Knox 14, out of a total of 18 components. RECOMMENDATIONS FOR FUTURE PROGRAMMING The following provides recommendations that could be useful for future programming in higher education, building on the lessons learned and best practices of the Advance Program in Jamaica: • Creating stronger linkages between high schools and tertiary-level institutions to better enable high school students to be motivated to attend tertiary level institutions and to be better prepared. This could also include working with HEIs to develop pre-university programs or summer programs to help youth transition into universities. • Fully understanding the disadvantaged youth in each country—be their challenges poverty, domestic or urban violence, racism, linguistic or ethnic minorities, etc.—to be able to anticipate issues and challenges early and include these in project design. • Strengthening the capacity of HEIs to better understand their roles in social and emotional learning and to better understand SEL approaches and how to apply them. • Establishing accountability mechanisms that extend between the program and the HEI, the program and the students, and the students and the HEIs. Establishing scholarship agreements with students would be key, and through the reporting requirements, could help the HEIs track these students’ progress upon graduating from the HEI. • Emphasizing the importance of professional development of faculty and staff in areas such as experiential learning—never underestimating the fact that this teaching approach is not widely known or utilized by some HEIs. • Utilizing experts and industry players to support curriculum development in HEIs. This was not a common approach in Jamaica but was one that was very much appreciated since it also provided faculty with several areas of new information and knowledge that enhanced their own teaching in the classroom. • Establishing communities of practice among lecturers and HEIs to share approaches to learning and teaching of disadvantaged youth as well as new curricula areas. 3 Sub-components are considered institutionalized if they are at least at the emergent level. UTECH and VTDI are more developed HEIs. Thus, they had better-defined processes, and adopted few approaches/methods. 1 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV I. INTRODUCTION This document presents the main findings from the Jamaica Regional Workforce Program performance evaluation, conducted under a Task Order from the USAID/LAC Education Support Contract. For more than three decades (1984-2015) USAID invested in the technical skill development and workforce preparation at the HE level of disadvantaged and/or minority youth from 12 countries across Central America, South America, and the Caribbean (including Jamaica) through an individual skill and leadership development model that brought these youth to United States (U.S.) community colleges for two years.4 This model’s theory of change was built on the premise that most in-country HE institutions did not have the capacity nor an interest in providing the access, resources, training, job-placement networks, or social-emotional support systems required to work successfully with this student profile. While this U.S.-based training model was proven to effectively affect the individual participants5—who did indeed return home and reintegrate into their societies’ workforces—the impact on their own countries’ capacity to train such youth remained underdeveloped and underequipped to accommodate this same youth profile that could result in skilled workforce training at home. Moreover, resources required for the U.S.-based model limited impact on the number of potential participants. Since 2015, the Agency’s focus and theory of change have pivoted to an integrated model that addresses institutional capacity in-country, alongside youth workforce development, addressing faculty capacity, updating curriculums to match workforce demands, and strengthening HEIs relationships to labor￾seeking industries in order to improve workforce access for the targeted student interns and graduates. 1.1. EVALUATION PURPOSE This evaluation will provide USAID’s LAC/RSD/EDU team with information, analysis, and recommendations to help guide future HE investments and funding decisions. It will also assist the DDI/EDU team to advance learning under the USAID Higher Education Learning Agenda to inform future activity designs and implementation. Given that the Latin America and the Caribbean (LAC) regional HE portfolio transitioned from an individual empowerment model (CASS and SEED scholarship programs6) to an institutional capacity development model (Advance Program), there is shared interest in assessing the effectiveness of this model and contributing to the evidence base on USAID, U.S. Government (USG), and other donor approaches to HE programming. The ET’s primary goal was to assess the changes within the targeted HEIs, including whether they were either fully or partially institutionalized, to ensure the sustainability of outcomes. However, the analysis also required that the team determine if there was any preliminary evidence that the outcomes of the Advance Program and promoted changes at the institutional level extended or can be extended to a broader national level, including the Ministry of Education, Youth and Information (MOEYI), CCCJ, and 4 Scholarship for Education and Economic Development (SEED), Georgetown University Center for Intercultural Education and Development. Accessed 05/16/22 at https://cied.georgetown.edu/scholarship-for-education-and-economic-development/. 5 Chesterfield, R. & Dant, W. (2013). Evaluation of LAC Higher Education Scholarships Programs. Accessed 05/16/22 at https://pdf.usaid.gov/pdf_docs/pdacx232.pdf. 6 USAID-funded Cooperative Association of States for Scholarships (CASS) program, later known as the Scholarship for Education and Economic Development (SEED) program. USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 2 within the wider context of developmental initiatives for youth. Analyzing these results serves as an opportunity to propose recommendations, tools, and strategies to adopt the best practices, lessons learned, and successes of Advance. The current section provides an overview of the evaluation design, implementation, and findings. Section II provides background information on Advance, the program’s objectives, and its ToC. Section III describes the evaluation and presents the evaluation questions. Section IV describes the evaluation methodology: systems approach, variables, instruments, sample, data collection, and analysis, as well as some limitations. Section V presents the main evaluation findings by research question and key variables. Section VI includes a discussion on higher level conclusions based on the evaluation findings; and, lastly, Section VII includes recommendations to USAID related to lessons learned and best practices that could be adopted and/or adapted in other locations around the LAC region and beyond. Annexes include the survey instruments, list of persons that participated in the evaluation, and progress markers. 3 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV II. BACKGROUND 2.1. PROGRAM OVERVIEW In LAC, one in five youth ages 15 to 24 are out of school and not working, and this number is projected to increase. Lack of access to youth empowerment interventions, including life skills-based education, training, and livelihood opportunities as well as psychosocial support is resulting in high rates of youth and adolescent violence, an increase in teenage pregnancies, and lack of interest and motivation of young persons to lead fulfilling lives, among others. The Advance Program was developed to respond to several of these issues facing youth by creating an enabling environment through HEIs to improve youth employability, by strengthening the capacity of two- and three-year technical tertiary education programs to meet the needs of disadvantaged youth in the Dominican Republic, Guatemala, Honduras, and Jamaica, the latter of which is the focus of this study. The project is funded by the USAID and implemented by FHI 360. Since its 2018 inception in Jamaica, the Advance Program has been partnering with the following tertiary education institutions across Jamaica: the University of Technology (UTECH), the Vocational Training Development Institute (VTDI), and the CCCJ that coordinates the work of eight Community Colleges. The CCCJ is statutory body of the MOEYI and was specifically established to coordinate the work of community colleges in Jamaica. By working with the CCCJ, the Advance Program was able to extend its reach and impact to these eight colleges through, for example, the development of new curricula and programs. The overall aim of the Advance Program is “to strengthen technical education programs that are able to equip students with skills that are in demand by the private sector and ensure that youth are ready to fill quality positions across these promising industries.” 7 The primary private sector employers targeted by Advance in Jamaica are in tourism (especially health and wellness tourism), agri-business, and the cultural and creative industries (e.g. digital animation, arts management). 2.2. PROGRAM OBJECTIVES The Advance Program works to build the capacity of targeted institutions by: • Improving curriculum design and pedagogy; • Providing professional development opportunities for faculty and staff; • Enhancing labor market bridging services to help students find gainful employment after graduating; • Strengthening recruitment and admissions processes within HEIs; and • Providing local scholarships for disadvantaged students to attend technical and post-secondary level programs. 7 Advance Jamaica Program’s official website: https://www.advanceprogram.org/about-us/, accessed 9/10/2021. USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 4 The program objectives defined by USAID and FHI 360 include the following long-term results and their corresponding intermediate results (IRs): Result 1 – Students in targeted academic units demonstrating market-relevant skills increased • IR 1.1 Curriculum design and pedagogy improved • IR 1.2 Teacher, faculty, and staff development refined • IR 1.3 Labor market bridging enhanced Result 2 – Equitable access for disadvantaged individuals to targeted programs of study increased • IR 2.1 Expanded access • IR 2.2 Provision of scholarships 2.3. THEORY OF CHANGE Advance’s ToC “Reach Back, Reach Out, and Reach In,” is further articulated as follows: If we reach back to upper secondary-aged youth in Dominican Republic, Guatemala, Honduras, and Jamaica to help them understand their post-secondary choices, And reach out to technical [vocational and other higher education] training institutions to build their capacity in specific programs and services, And reach into employers in these four countries to better understand their workforce needs and how these needs align with technical education programs, Then more disadvantaged youth will obtain better or new employment [opportunities].8 At USAID’s suggestion, the DevTech evaluation team organized and hosted a Theory of Change Review Workshop on March 23, 2022 to further frame and explore the components of Advance’s ToC that the evaluation team should focus on in its data analysis and findings. Key USAID and FHI 360 Advance staff members who were engaged in a discussion on the proposed study and areas that could be of particular focus participated in the workshop. Contributions made by participants at this workshop assisted the evaluation team in better understanding issues and challenges faced by FHI 360 (the Jamaica team) and HEIs in terms of targeted students’ (non-)preparedness for HE, the unanticipated support that HEIs needed, for example, to understand the reporting requirements of USAID, which took focus away from advancing some activities in a timely manner, such as labor bridging services. The ET also learned that the “systems approach” focus evolved through experience rather than having been intentionally highlighted in the original project design. Specifically, the ToC workshop provided insights on the following: 8 LAC Regional Workforce Development Program: Life of Agreement Strategic Plan. FHI 360, August 22, 2016. 5 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV • The extent to which Advance’s activities in Jamaica have been able to “reach out” to employers, and how this has compared to Advance projects in the other countries, given that outreach to labor institutions is one of the primary expected outcomes of the Advance Program. • Use of the systems approach in the evaluation to better identify and analyze existing processes that affect performance, generate and enable best practices, and/or aim to improve the experience of program beneficiaries. • The impact of COVID-19 on the program, specifically focusing on how the pandemic may have affected the processes related to institutionalization and sustainability of outcomes by the HEIs, even though the COVID-19 pandemic began two years after the start of Advance and several components of the project were already implemented and/or completed prior to the onset of the pandemic. • Aspects of the program they felt had the greatest impact, which in FHI 360’s estimation was “reach out,” further indicating that there was not a clear balance on all three inputs within the ToC. • The use of a labor market assessment for Jamaica, which was completed in 2016, and therefore indicated that there may be a need to assess whether labor market conditions changed or if assumptions made on labor market potential and value changes were no longer true. • The role that the COVID-19 pandemic played on the program. For example, FHI 360 said that while psychosocial support was made available to Advance students, the pandemic seemed to inject lower trust in, and uptake of, these support services that might have helped students better manage the psychosocial impacts associated with the pandemic—this attitude was unexpected. III. EVALUATION DESCRIPTION 3.1. EVALUATION OBJECTIVES AND RESEARCH QUESTIONS Overall, this study assesses Advance’s implementation outcomes across the following four broad research areas that were aligned to the study’s research questions. Research Areas Research Questions Change in the institutional performance of the HEIs in the study as systems in and of themselves, and changes in the overall HEI system RQ1. What are the changes in systems within HEIs and improvements in their institutional performance? According to the perspective of key stakeholders in HEIs, how have the institutions improved because of Advance, specifically in the areas of 1) curriculum and pedagogy, 2) professional development of faculty and staff, 3) labor-bridging services, 4) admissions and recruitment, and 5) access to scholarships? According to the perspective of key stakeholders within USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 6 Research Areas Research Questions HEIs, in government institutions and private sector counterparts, to what extent, and how, has Advance influenced changes in institutional performance and systems of HEIs to be better aligned and more relevant to Jamaica’s workforce needs? Provide examples of institutional performance and improvements as appropriate. Sustainability and institutionalization of Advance outcomes in participating HEIs RQ2. What is the level of sustainability and institutionalization of Advance in HEIs? Will your institution make further efforts to institutionalize and sustain Advance’s support? If so, how; if not, why not? Have there already been changed or amended processes or procedures to adopt the changes in the areas of 1) curriculum and pedagogy, 2) professional development of faculty and staff, 3) labor-bridging services, 4) admissions and recruitment, and 5) access to scholarships? What are these changes? And what elements within these changes can be defined as the contributors to sustainability? Are there any capacity gaps within each of the five areas that remain and need to be addressed? Do the HEIs need support to make these changes, or do they need directives to effect the changes by the head of the institution in the case of UTECH or VTDI, or the CCCJ in the case of community colleges? Enabling and inhibiting contextual and institutional factors influencing said performance improvement RQ3. What contextual and institutional factors enabled or inhibited the application of an institutional capacity-building/performance improvement approach? Describe the enabling and inhibiting factors in relation to scaling up key initiatives and support under Advance, continuing with Advance’s activities after it has ended as well as advancing institutionalization. How does this compare across HEI types (i.e., community and technical training colleges, universities)? To what extent are the factors consistent across institutions? Indicators of change RQ4. What are the indicators of change across each of the five key areas that can lead to anticipated long-term outcomes around institutional strengthening? Factoring in that some outcomes (such as enhanced labor market bridging) would not be fully realized in the short to medium term, are there reliable intermediate indicators across each of the five key areas that can lead to anticipated long-term outcomes around institutional strengthening? 7 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV IV. METHODOLOGY 4.1. EVALUATION SYSTEMS APPROACH The design of this evaluation is based on a systems approach to identifying and analyzing existing processes that affect performance, generate and enable best practices, and/or aim to improve the experience of program beneficiaries. Data collection questionnaires, the selection of the sample of actors to be interviewed, and the analysis of the evaluation questions were conducted with this approach in mind, and are reflected in the evaluation findings in Section V. The ET generated a systems map for each participating HEI that assisted in understanding relationships and interactions, as well as trends in which Advance was able to introduce leverage points, why and how impact was accomplished, and revealed the systems supporting, impeding, or promoting, and generating further institutional sustainability of the impacts and improvements created by Advance. Outcome Mapping. The Evaluation Team used outcome mapping as the main methodology to assess the changes in institutional performance across HEIs. As described by Earl, Carden, and Smutylo, 9 outcome mapping is a method for planning, monitoring, and evaluating development activities that are designed to result in social change. It seeks to clarify how programs or projects are contributing to achieving outcomes through the monitoring and evaluation of factors and actors within their direct sphere of influence. Outcome mapping focuses on identifying and assessing changes in the behavior of people, groups, and organizations with which a program works directly. It does not seek to prove causality or attribution for changes but attempts to show logical linkages between the changes and a program’s activities, thereby enabling a program’s contribution to be understood. Performance is assessed as the program’s contribution to influencing those changes. The outcome mapping methodology allowed the evaluation team to shift focus from the activities, products, and outputs of the program to a focus on changes in capacity, institutional policies, financing and resource inflow, behaviors, and relationships, especially those related to the five key areas that are the focus of this study: 1) curriculum and pedagogy, 2) professional development of faculty and staff, 3) labor-bridging services, 4) admissions and recruitment, and 5) access to scholarships. Outcome mapping is widely used in evaluations in teaching and learning contexts and is a flexible and complementary approach to evaluations. We also chose outcome mapping as the main methodology because it encourages self-reflection and self-assessment, which would be ideal in the context of the HEIs that were part of the study. Outcome mapping helped the evaluation team to determine Advance’s success at improving the institutional performance of HEIs and to assess the program’s sustainability. The approach adopted the use of graduated progress markers to determine if Advance has achieved its desired outcomes. A summary of the progress markers results is available in Annex 1. By using an outcome mapping approach, the ET was able to take a deeper dive into the HEIs’ sphere of influence within the Advance Program itself, and the sphere of interest beyond Advance or the system-wide impact – within the context of each individual HEI and across HEIs as well as in policy spaces across Jamaica. Data collected through KIIs, FGDs, and the ToC review workshop have allowed for an assessment and 9 Earl, S, Carden, F. & Smutylo, T (2001). Outcome Mapping: Building Learning and Reflection into Development Programs. Ottawa: International Development Research Center. USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 8 analysis of the outcomes or changes in behaviors and processes of the HEIs and impact of Advance either in a specific HEI or the overall higher level education system. 4.2. VARIABLES The five qualitative areas (variables) underpinning the study questions include: • Curriculum and pedagogy. Refers to changes or additions to the course of study within each of the HEIs, based on the needs of the labor market as well as changes, including adoption of new teaching practices by faculty; • Professional development of faculty and staff. Refers to how the program supported capacity building of faculty and staff; • Labor-bridging services. Refers to processes that allowed for the alignment of the needs of the labor market with the HEIs, with the ultimate aim of connecting graduates with required skills and competencies with jobs and industry to fulfill their enterprise objectives; • Admissions and recruitment. Refers to the process of attracting disadvantaged youth to pursue studies at HEIs, reviewing and verifying application information and approving matriculation to the HEI; and • Access to scholarships. Refers to the access to resources that would allow disadvantaged youth in Jamaica to be able to attend an HEI that was part of the Advance Program. 4.3. INSTRUMENTS Online Survey. The team administered an online survey to 74 participants, which included administrators, deans, faculty, and students using the online survey tool EmailMeForm. A total of 58 persons completed the survey. The survey was distributed to all participants of focus groups and KIIs about five days prior to the designated day of the informant interview or focus group discussion. The survey instrument was designed to ensure that respondents could complete the survey within 10 to 15 minutes. This tool enabled the capture of quantitative data and provided an important input into the focus group discussions, as these responses helped to guide the focus group discussions, allowing the team to better examine areas or themes that may have emerged from the online responses. Not taking into account students, 92 percent (47 out of 51) of participants completed the online survey. The survey was structured to capture information on: • The overall satisfaction of the HEIs with Advance; • The extent to which Advance and the tools and techniques it introduced contributed to improvements in curriculum design; • The extent to which the tools and techniques provided by Advance were innovative; • The impact of Advance in improving the capacity of staff; 9 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV • The needs and resource requirements for scaling-up Advance; and • The extent to which participants believed that the Advance Program would enable graduates of Advance degrees to be better able to access opportunities in the labor market. A separate online survey was developed for students. Eleven students—from three of the four HEIs (UTECH, Knox, and VTD)—completed the survey, representing a response rate of 48 percent. The survey was sent out to 23 students from the four HEIs. This survey captured information related to the following: • Students’ satisfaction with being selected for a scholarship under the Advance Program; • Students’ perception of the Advance Program; • Whether students were aware that the courses they took were Advance-supported courses; • The approach to teaching on the Advance courses compared to other courses; • The impact of COVID-19 on students’ participation in school and how instructors worked with them; and • Whether they felt that the Advance Program prepared them for the job market. Key informant interviews open-ended questionnaires. For the KIIs, the ET prepared an open￾ended questionnaire, that sought to engage interviewees in a more strategic way to capture information related to: • Comparison of other programs of a similar nature with Advance; • Contribution of Advance to improvements in curriculum development and professional development at the institution, including tangible changes; • The influence of Advance on internal procedures such as strategic planning, curriculum design, teacher professional development, etc.; • The comparison of Advance scholarship recipients with other students; • Lessons learned and best practices; • The profile of Advance students; • Challenges of the institution with respect to implementation and how these were overcome; • Approaches to institutionalization and sustainability; and • Overall impressions of Advance and the impact of COVID-19 on activities associated with the Advance Program. USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 10 KIIs were held primarily with principals, deans, and heads of departments. Whilst the norm for a KII is for a one-on-one interview, the KIIs of all four institutions included multiple persons. Open-Ended Questionnaires. 10 In addition to KIIs, the team also developed and used three other questionnaires primarily for FGDs. These open-ended questionnaires were developed for FGDs with three groups: one for heads of departments, lecturers, and curriculum development specialists; one for administrators and school counsellors; and one for students. 4.4. SAMPLE The Advance Program in Jamaica engaged with the following HEIs: • University of Technology, Jamaica • Vocational Training Development Institute • The eight affiliated colleges under the CCCJ – five are community colleges and three are multi￾disciplinary colleges. The colleges offer programs from the certificate, diploma, associate, and baccalaureate levels as well as several continuing education courses. Rationale for HEI selection. The approved study design included three HEIs for investigation. UTECH and VTDI were clear choices for the evaluation as both institutions were independent of the CCCJ. The other HEI chosen by the team was Knox, a college under the CCCJ, as the ET wanted to understand if there were any relevant differences in Advance implementation with community colleges. Following the ToC workshop, FHI 360 made a recommendation to investigate one more HEI under the CCCJ, specifically Knockalva Polytechnic, due in part to the fact that it was situated in deep rural Jamaica, and it had in recent years been upgraded to a polytechnic college offering a wider range of programs beyond agricultural programs. KPC entered the Advance Program at the end of 2019, and according to the ET findings, the implementation of the program in KPC already included adjustments derived from the lessons previously learned by other Advance institutions. More information about each HEI is available in Annex 3. Furthermore, given the role of CCCJ as the government body of community colleges, the ET also included them as a key informant during the data collection. Some criteria that were considered in the selection of the colleges included: • Urban versus rural – in Jamaica, students in rural areas have fewer opportunities than students in urban areas and therefore focus was placed on selecting at least one rural school (KPC). • Diversity of institutions – rather than selecting four HEIs that were community colleges, for example, the team chose to investigate the only university that was part of Advance, as well as Jamaica’s and the Caribbean’s premier vocational training institution: VTDI. The selection of participants for FGDs and KIIs was determined by the HEIs based on the scope of the evaluation and the staff that were intimately involved in the Advance Program. In the case of students, 10 Survey questionnaires and interview guides are included in Annex 5, consent form is available in Annex 4. 11 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV HEIs provided the ET with a list of students that received scholarships. The ET then contacted students on the list to arrange for KIIs and completion of the survey. 4.5. DATA COLLECTION Participation of the Colleges in the Evaluation. The data collection phase ran from January 18 to March 15, 2022. Table 1 shows the participation level across the four HEIs as well as the CCCJ (not including participants that completed the surveys). A total of 51 persons participated in the evaluation from across the four HEIs and a KII with the CCCJ representing principals, deans, heads of departments, lecturers, administrators, and school counselors. Of the total number of participants, there were 12 males and 39 females. For Jamaica, this imbalance is not unique as several positions of responsibility from across the public and private sectors are headed by and led by women. Indeed, more women than men also graduate for tertiary level institutions. It is important to note that these 51 persons were the total amount of persons that the ET interacted with either through KIIs or FGDs. The persons that participated in the evaluation all worked closely with the Advance Program. TABLE 1: DATA COLLECTION PARTICIPANTS Institution Location Advance Program Status Participation in Evaluation Status UTECH Kingston UTECH started in October 2018 Total participants: 10 3 KIIs (Head of Department, Dean who participated in two KIIs with one focusing on the academic side related to Advance and the other KII related to questions for guidance counsellors, and senior official); 2 FGDs - one with faculty and the other with administration such as officers from recruitment, corporate planner, and guidance counsellors, student interviews with 2 students Males 5, Females 5 ● Deans, Head of Departments (HOD), Lecturers, etc. VTDI Kingston VTDI started in September 2017 Total participants: 11 2 KIIs (Senior Manager, Administration; Vice Principal, Academics; and Bursar), 2 FGDs, 2 student interviews USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 12 Institution Location Advance Program Status Participation in Evaluation Status ● Males 2, Females 9 ● Vice Principals/Deans, HOD, lecturers, etc. Knox Clarendon Knox started March 2018 Total participants: 19 ● 1 KII (Head of Agriculture and Environmental Studies Department; Quality Assurance Officer, Bursar, Examination Coordinator) 2 FGDs, 7 student interviews ● Males 3, Females 16 ● HOD, lecturers, administration Knockalva Hanover Knockalva started September 2019 Total participants: 8 ● 1 KII (Principal, Vice-Principal, Programs Coordinator), 2 FGDs ● Males 2, Females 6 ● Principal, Dean, lecturers, administration CCCJ Kingston CCCJ started in June 2017 Total participants 3 ● 1 FGD ● Females 3 ● Curriculum Development Specialist, Administrator, Registrar Total 51 participants, 12 males, 39 females11 11 Additionally, the ET consulted with USAID and FHI 360 staff during the Theory of Change workshop. 13 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV 4.6. LIMITATIONS The following limitations were anticipated and occurred during the data collection and analysis phase: 1. Online data collection could have compromised data quality due to limited participation of some informants. Due to the COVID-19 pandemic, the entire education system in Jamaica pivoted online from March 2020 to April 2022, with many HEIs at the time of the data collection (January to March 2022) still not operating face-to-face. All the field work for this evaluation was conducted through online surveys and virtual Zoom-based FGDs and KIIs. The targeted research participants were all presumed to have access to—and be comfortable with—this approach, as it paralleled the educational process at the time. The quality of the virtually collected data may have been compromised due to: 1) limited participation of some informants due to the lack of fluidity of the FGD, based on its virtual nature; and 2) an inability to observe non-verbal responses through body language, and in the campus context due to the virtual nature of communication. 2. Insufficient sample size of the student population. It proved to be quite challenging to obtain an adequate student sample size for the FGDs, even though names and contacts were provided by the various HEIs, and several options provided in terms of the timing of the FGDs. Consequently, the FGDs morphed into one-on-one interviews with interested students. It was therefore difficult to make generalizations about the results during the data analysis phase. The ET cannot fully determine the factors that led to the unresponsiveness of students to participate in the FGDs. V. EVALUATION FINDINGS The Advance Program, as mentioned above, was designed to strengthen the capacity of HEIs in Jamaica to provide market-relevant, quality training to disadvantaged youth, so as to secure their options for future meaningful employment and a better quality of life. To achieve this and bridge the gap between technical education offerings and labor market needs, Advance has focused on building the capacity of several HEIs in Jamaica in five key areas: 1) curriculum and pedagogy, 2) faculty/staff professional development, 3) labor-bridging services, 4) admissions and recruitment, and 5) access to scholarships. Accordingly, the evaluation findings are aligned to the five areas of capacity building of HEIs and the research questions were structured around these areas as follows: • RQ1. Change in systems within HEIs and improvements in HEIs’ institutional performance • RQ2. Sustainability and institutionalization of Advance in HEIs • RQ3. Enabling and inhibiting factors • RQ4. Indicators of change Overall, the evaluation showed a strong level of agreement among HEIs about the quality of the services, products, processes, and interactions produced by Advance. Unless otherwise stated, the evaluation findings did not include the views or perception of students or the analysis of their survey instrument. It USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 14 also is important to note that unless otherwise stated, the findings contained in this report are either from the FGDs or from the KIIs. Figure 2 shows some of the quantitative assessment findings. FIGURE 2: PARTICIPANTS/FACULTY AND ADMINISTRATORS OF HEIS PERCEPTION OF QUALITY OF ADVANCE SERVICES From interviews and FGDs, the following comments were expressed about Advance, which supports the data gleaned from the quantitative survey: “Excellent in vocational based training, competency-based methodologies and targeting marginalized persons and facilitating them to become productive, better citizens.” (VTDI Vice Principal and Administrators) “Advance Program was multi-faceted – not just scholarships, but focused on elements such as pedagogy, professional development, and the admission process.” (VTDI, Vice Principal and Administrators) “Advance was sensitive to human needs and cared about the well-being of students...for other programs, the implementing agencies/persons never got fully involved in the human and social peculiarities, and persons worked primarily with data and documents.” “The human interaction of Advance was evident.” (Knockalva Administrators and Faculty) Additionally, HEIs generally agreed that one of the key objectives of Advance—to support disadvantaged youth in Jamaica to access higher levels of education to improve their future employment opportunities—was somewhat achieved, as indicated by the following: “The program is to be commended for giving access to students who would not have normally been able to benefit from a tertiary education. Students were selected based on their financial and economic needs and would not have been able to attend university were it not for the Advance Program, and hence the objective of the program was successful.” (UTECH Dean and Lecturers) “The idea of full scholarships to vulnerable students makes it [Advance] stand out.” (UTECH Dean) 15 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV “The Advance project is a wonderful initiative with very kind personnel who displayed a passion for the development for [the] students, staff, and communities, which will ultimately develop Jamaica. This project support demonstrated commitment, excellence, integrity, mastery, respect, and most of all confidence for all stakeholders. The project also contributed to the college fulfilling its mission and vision and there is great appreciation for the dedication and contribution to the growth and development of the institution.” (Knox Administrators) “It is a very good program, as it assists student in ways that a normal scholarship does not. Without the program, I would not have been able to pay monthly rent and it would be hard to pay the school fee.” (Knox Student) As part of the qualitative assessment undertaken through the FGDs, participants were asked to rank the improvements at the HEI that the Advance Program contributed to with respect to labor-bridging services, admissions and recruitment, and access to scholarships. They ranked these on a scale of 1 to 10, with 1 being low or the worst outcome and 10 signaling the best outcome (better). Table 2 below provides the results of this ranking. TABLE 2: HEI SELF-REPORTED AVERAGE SCORES FOR LABOR-BRIDGING SERVICES, ADMISSIONS, AND RECRUITMENT HEIs Labor-Bridging Services Ranking Admissions and Recruitment Ranking Access to Scholarships Ranking Knox 9 8 10 Knockalva 9 8 10 VTDI 8 8 8 UTECH 5 N/A 6 Average Ranking 7.75 8 8.5 Overall, all HEIs scored Advance above average for the support it provided to the HEIs. For example, as this report will illustrate later on, the HEIs were not aware prior to Advance of the role that HEIs could play in labor bridging services and what these explicitly entailed. Access to scholarships for vulnerable and disadvantaged youth were welcomed by all schools. The approaches introduced by Advance for admissions and recruitment such as marketing were considered new and innovative. Throughout this report, there are varying assessments and responses from HEIs around similar topics and themes. It should be noted that while not explicitly stated in project documents related to Advance, the program seemed to have been built on a system of continuous improvement. For example, issues or challenges experienced by the institutions that began implementation first, such as UTECH and VTDI, were either not mentioned or not experienced by the community colleges in their later start-up periods. Many of the challenges, such as those related to the administration of scholarships and the elements of the scholarships or reporting requirements that were experienced by the HEIs that signed USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 16 MOUs first, such as UTECH and VTDS, had been turned into opportunities or best practices by the time the community colleges began implementation. It is important to note that the interaction between FHI 360 and the specific community colleges began later on as the development of the curricula for the community colleges was led by the CCCJ and the various community colleges only came on board once the delivery of these associate degrees was possible. 5.1. RQ 1: CHANGE IN SYSTEMS WITHIN HEIs AND IMPROVEMENTS IN THEIR INSTITUTIONAL PERFORMANCE Across each of the HEIs assessed, the Advance Program’s inputs led to varying degrees of success at improving their institutional performance. The ET’s outcome mapping approach allowed for the identification and capture of the progress made by HEIs toward institutionalization of positive outcomes of the program, as related to institutional performance. This evaluation revealed several changes to the systems within HEIs that positively affected their overall performance, as presented below in the outcome maps. 5.1.1. CURRICULUM AND PEDAGOGY This area of research corresponds to the Advance Program Result 1: increased number of students in target academic programs demonstrate market relevant skills and the following intermediate results: • Curriculum and pedagogy improved • Teacher, faculty, and staff development refined Under the Advance Program, six associate degree programs, aligned to the needs of the labor market, were developed and/or revised as follows: the information and details on the degree programs that Advance supported were provided through the KIIs and FGDs with the HEIs and with the CCCJ. TABLE 3: ASSOCIATE DEGREE PROGRAMS DEVELOPED OR REVISED HEIs Associate Degree Programs Developed or Revised University of Technology At UTECH, the Advance Program enabled the launch of two new programs: I. Associate in Science (ASc.) Degree in Image Consulting and Fashion Styling (ICFS) II. ASc. Degree in Entertainment Design, Production, and Technology (EDPT) Vocational Training and Development Institute At VTDI, the Advance Program enabled the strengthening of two programs: I. ASc. Animation II. ASc. Events Planning and Management (Advance supported the upgrading of this program and writing of some of the courses) Both of these programs existed prior to Advance and were enhanced following the intervention of Advance to make them more relevant to the industries in the 17 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV HEIs Associate Degree Programs Developed or Revised orange economy, 12 (related to the culture, cultural, and creative industries) and within the context of the overall thrust of Jamaica, to enhance the competitive advantage of the country’s culture, cultural, and creative industries. Council of Community Colleges of Jamaica With the CCCJ, the Advance Program enabled the development of two programs that could be delivered across all of the eight CCCJ colleges: • ASc. Degree in Agro-Processing and Business • ASc. Degree in Health and Wellness Tourism Both the community collegesthat were part of the evaluation – Knox and Knockalva – delivered the ASc. Degree in Agro-Processing and Business. In addition to supporting the development and strengthening of new programs, Advance allowed for the building of the capacity of faculty in the areas of curriculum design and pedagogy. Both UTECH and VTDI faculty and staff were exposed to and acquired curriculum development skills, competencies, and expertise in the development and strengthening of the programs mentioned above. This was less evident for the community colleges, as the programs were developed by Advance for the CCCJ rather than presented to each individual college, given that the CCCJ’s mandate also focuses on curriculum development for the community colleges that it oversees. The CCCJ then passed on the Advance￾supported programs to their colleges for delivery. Essentially, Advance paralleled the normal process for curriculum development at the CCCJ. While the staff at the community colleges did not participate with experts in the development of the curricula, they did benefit from training related specifically to the new curricula and approaches for teaching and experiential learning related to the curricula as happened at UTECH and VTDI. The added benefit of this was that at all colleges, some of this new knowledge was passed on in other courses that lecturers of Advance taught, or to other lecturers within the same departments, etc. This is an example of knowledge transfer within the institutions. Notably, Advance supported the process for UTECH to commence offering associate degrees for the first time in the history of the institution. Prior to Advance, the university had begun to explore the possibility of offering associate degree programs in response to the demand of young persons, the needs of labor market, and the need to improve the educational attainment levels of youth—as well as to contribute to the achievement of national targets for enrollment in tertiary-level institutions. Associate degrees are seen within the education sector as a key factor in improving educational outcomes and attracting young persons who may not have had an interest to move onto 6th form, and later on to university. It increases their educational attainment and has the added advantage of encouraging some students to matriculate into bachelor’s degree programs. The introduction of these six new programs reflects institutional change that aligns with the intended results of the Advance Program. Moreover, these degree programs align with the national priorities of the country as it relates to improving value￾12 The orange economy is also known as the creative economy and includes all sectors whose goods and services are based on the creation of intellectual property, such as design and visual arts, tourism and cultural heritage, new media and software, performing arts, music, and literary arts and publications. USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 18 added in agriculture, taking advantage of opportunities within the orange economy, and diversifying the country’s tourism product to benefit from niche markets, such as those related to health and wellness tourism. These areas continue to remain priorities in Jamaica and are included in the country’s medium￾term framework for 2021 – 2024. The HEIs indicated that the development and/or revision of the curricula by Advance was underpinned by participatory processes and stakeholder consultation involving lecturers, and in many cases, experts in the field. This allowed for the curricula to benefit from the inputs of a wide range of stakeholders, including industry players, staff, and lecturers, and in some cases, students. Administrators and lecturers alike indicated that this approach was well-received and appreciated by both UTECH and VTDI. This approach also stands out as one of the areas that would be institutionalized after the completion of the Advance Program, given the value that lecturers saw in engaging experts and professionals in the field in curriculum development. The two HEIs mentioned the uniqueness of the approach and how it strengthened the curricula and their own competencies. For example, UTECH mentioned that lecturers in the EDPT Program were taught how to apply electrical skills to the entertainment industry: “This was the first-time lecturers were involved in the practical aspects of stage lighting and sound.” (UTECH Academic Official). Both VTDI and UTECH, in the KIIs as well as the FGDs, stressed that the development and review of curricula by Advance also led to the introduction of new and modern approaches to experiential learning—linking industry and academics—with pedagogy focusing on hands-on learning, which had the added advantage of enabling students to interface directly with industry as part of their learning. The faculty at UTECH, for example, was involved in the development of the curricula of both programs, and Advance also provided consultants to support curricula development. The curricula also benefited from the competencies and expertise of practitioners in the field, such as fashion designers, etc., who would act as advisors to support curriculum development. Faculty were able to better understand how to twin theory with practical application and approaches. The focus on practical applications helped to improve the capacity of the faculty in the area of curriculum development. In the case of VTDI, reference was also made by administrators and faculty to the influence of Advance on the curriculum. Whilst the programs supported by Advance existed previously, Advance was key to enhancing both programs by making the courses current and relevant. For example, as part of the ASc. Entertainment and Events Management (EEM), Advance supported the review of the overall program. Advance reviewed the courses for overlaps, consistency in structure, relevance to industry, and new and emerging trends and technologies. Faculty and administrators indicated that the revision of the programs also led to changes in teaching and learning strategies, assessment strategies, and types of assessments. The courses were reviewed to reflect the higher order objectives of Bloom’s taxonomy. 13 Also, a new course in technical theater was introduced under the EEM program. Advance also influenced and contributed to enhancements to the ASc. in Animation program, with the development and offering of new areas such as programming and digital media, including audio and video. Along with these new offerings, two courses were introduced to cover portfolio development. Administrators indicated that 13 Bloom’s taxonomy is a set of three hierarchical models used for classification of educational learning objectives into levels of complexity and specificity. The three lists cover the learning objectives in cognitive, affective, and psychomotor domains. The cognitive domain list has been the primary focus of most traditional education and is frequently used to structure curriculum learning objectives, assessments, and activities. 19 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV these changes have enabled students to complete the program with an organized portfolio of work, which supports their entry into the labor market and/or entrepreneurial efforts. In the case of the two community colleges that the evaluation covered, the opportunity to engage in the development of courses with Advance and be exposed to curriculum development was not possible, as the courses were developed in collaboration with the CCCJ and presented to the colleges to deliver. Notwithstanding, the structure of the curricula of the courses under the programs enabled lecturers to be exposed to practical examples, enabling the twinning of practical application with theory. The qualitative assessment above can be further supported by the quantitative survey undertaken as part of the evaluation by survey respondents (administration and faculty of the HEIs). As shown in Figure 3, 88 percent of participants across all the HEIs either strongly agreed or agreed that the tools and techniques introduced by Advance contributed to improvements in curriculum design. According to findings from FGDs, all HEIs indicated that the involvement of Advance with them also established the importance of engaging with the labor market survey information produced by the Government of Jamaica. This also enabled faculty to engage more with employers and industry players in curriculum development. “There is a shift within faculty in terms of the sources to draw on in making key decisions regarding the design and development of new programs.” (UTECH Official) FIGURE 3: PERCENTAGE OF HEI PARTICIPANTS THAT CONSIDER THAT TOOLS AND TECHNIQUES CONTRIBUTED THE CURRICULUM DESIGN 18% 70% 9% 3% Strongly Agree Agree Neutral Disagree Strongly Disagree 5.1.2. PROFESSIONAL DEVELOPMENT OF FACULTY AND STAFF This area of research corresponds to the Advance Program Result 1: increased number of students in target academic programs demonstrate market relevant skills and the following intermediate result: • Teacher, faculty, and staff development refined HEIs interviewed for this study agreed that that the professional development of faculty and staff was a key component of Advance, and a substantial amount of time and resources were invested in this area of the program, which they deemed as highly successful. The approach to professional development of faculty and staff can be summed up as follows: • Access to cutting-edge technologies that could be taken back to the classroom; • Opportunities for peer-to-peer learning; • New knowledge and information from experts and technocrats in the field; USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 20 • Areas of focus of training received by faculty and staff were sometimes outside of training for the courses developed by Advance and focused on softer skills as well as areas such as entrepreneurship; • The professional development of faculty and staff often extended to staff of the entire HEI, and not just faculty that were engaged with Advance; and • The approaches and new knowledge Advance introduced were often accompanied by train￾the-trainer components, which augured well for institutionalization and sustainability. As shown in Figure 4, 94 percent of survey respondents indicated that Advance positively affected the development of the staff of the HEI, and there currently exists increased capacity in the HEIs. Several opportunities were provided to faculty and staff for capacity-building in a range of areas, some directly related to the programs that Advance developed, as well as courses and workshops that were designed for faculty and staff to help students develop socially, emotionally, and professionally. The content areas of some of these workshops included entrepreneurship, proposal￾writing, and budgeting. These workshops were open to all faculty and staff of the four HEIs and were not limited to the faculty that taught Advance-supported courses. The persons interviewed though KIIs and FGDs considered all training to be relevant to take back into the classroom and transfer to students. This was one of the areas that the four HEIs spoke about in relation to the advantage and importance of these courses to the overall development of students. One training workshop that all HEIs mentioned was in SEL, which was offered to faculty and staff. This training was new to all HEIs, and the train-the-trainer component was key to institutionalizing this training across various departments within the HEIs to be better able to support students that were not necessarily considered vulnerable or even part of Advance. UTECH boasts that three of its lecturers being trained in SEL can be of benefit to the entire university. With respect to the academic programs developed by Advance, both UTECH and VTDI made mention of the interaction between faculty and experts in the field and peer-to-peer learning as part of efforts that contributed to capacity-building. For example, at Knox, “Support was there to ensure that lecturers were comfortable to lecture/teach new courses – engaged in professional development” (Head of Department). There also was emphasis across the HEIs on building the capacity of faculty to deliver different hands-on and practical exercises related to the programs developed. At the community colleges, training was provided to faculty in agri-entrepreneurship and food products development, with a focus on illustrating how to develop new products from agricultural produce—such as using coffee to produce soaps, face scrubs, etc. Faculty indicated that this made the overall program more attractive to youth, with teaching and learning focusing more on an entrepreneurial approach to agricultural production. In the case of UTECH, for example, mention was made of the capacity-building of lecturers in entertainment, in areas FIGURE 4: PERCENTAGE OF HEI PARTICIPANTS THAT CONSIDER THAT ADVANCE POSITIVELY INFLUENCED DEVELOPMENT OF THE STAFF 39% 55% 6% Strongly Agree Agree Neutral Disagree Strongly Disagree 21 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV such as how to apply electrical skills to the entertainment industry. “This was the first time lecturers were involved in the practical aspects of stage lighting and sound” (UTECH Official). Advance also facilitated peer￾to-peer learning by bringing different community colleges (including those that are not part of this study) together to share approaches, best practices, etc., and this also was key in supporting capacity-building efforts and was welcomed by HEIs. To support capacity development of faculty and staff, Advance was able to create opportunities for collaboration with other institutions (not only academic institutions but also government and private sector institutions), both locally and internationally, and this was considered a best practice by faculty and administration of the HEIs. “Advance has a wonderful collaborative technique between public and private sector; lecturers and students were able to engage with a range of private sector entities enabling partnerships to be strengthened” (Knockalva Official). Some of these partnerships, as Knockalva mentioned, were key to facilitating internships for students. At the two community colleges—Knox and Knockalva—lecturers and students were able to engage with a range of local private sector firms in food processing, which contributed to strengthening partnerships between the HEIs and the private sector. They also engaged with the Scientific Research Council – a government agency that focuses on testing foods and other products and applying scientific processes to agricultural products, including agro-processing. At the international level, faculty at VTDI benefited from a study tour to Mexico, which focused on vocational approaches to education and training and student support mechanisms to improve learning outcomes. Following this study tour and in keeping with the requirements of the study tour, participants implemented a project based on their learning experience, specifically a student advisement center. The Student Advisement Center at VTDI has been designed as a safe space for students and allows them to engage in discussions with lecturers. The center also has tutor groups, and students are able to meet with lecturers for tutorials in subjects such as mathematics and English communications, receive support on assignments, and obtain general guidance on other courses or projects. VTDI administrators applauded the study tour to Mexico, which they claimed further enabled faculty to better understand the importance of creating student-centered institutions. The study tour helped them to experience this first hand. The faculty explained that the experience helped them to better embrace a “student-centric” approach to learning, which they considered to be over and above the role of counselors. On the other hand, while VTDI praised the knowledge and benefits gleaned from the study tour afforded to that HEI, UTECH bemoaned the fact that faculty and staff were promised overseas study tours to support capacity-building, but these did not materialize, 14 resulting in staff feeling that the overall opportunities for capacity-building were limited. 5.1.3. LABOR-BRIDGING SERVICES This area of research corresponds to the Advance Program Result 1: increased number of students in target academic programs demonstrate market relevant skills and the following intermediate result: 14 The HEI reported that early discussions had a focus on overseas study tours but as the project progressed these were not discussed further. USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 22 • Labor market bridging enhanced “Approaches to labor-bridging services were unique and UTECH will take them forward – approaches that linked students with industry players, the approaches to networking.” (Dean) Labor-bridging services were viewed by all HEIs as a new and novel approach to supporting students beyond the classroom and helping them transition to the world of work. Prior to Advance, none of the HEIs focused on engaging industry or other players to support and identify employment opportunities for students due to their limited understanding and knowledge of the industries. This was in part because HEIs focus their mandate in large measure on providing higher learning and issuing certification and not aligning their role with supporting students beyond graduation. Advance enabled HEIs to become more aware of the concept and importance of labor-bridging, and how it could be aligned to an HEI’s mandate and consequently, its courses and degree programs. This component of the program, as introduced by Advance, allowed HEIs to consider a more expanded and complementary role for the university—which they welcomed. However, three out of the four HEIs referred to this aspect of the program as the weakest, given that there were several aspects that were not completed, but the aspect of the program that they felt had significant potential. “Engaging industry more was a focus of the program but was not as successful—the benefits were not felt. We hoped that VTDI would have been more engaged with industries—the initial partnership spoke to linkages with industry – but as the project unfolded, we did not really see that impact.” (Vice Principal) On the contrary, officials at Knockalva indicated that not only did they learn quite a lot about labor bridging, but felt that it was an excellent aspect of the program, with all 12 graduates in the first cohort becoming gainfully employed after graduation. The fact that Knockalva seemed to have experienced higher-level benefits related to labor￾bridging services could be attributed to the fact that they started their interaction with Advance later in the overall implementation of the program when several issues and challenges related to many aspects of the program were already addressed. “[The] first cohort, when they went on work experience and based on [the] wealth of knowledge of the trainees – they were gainfully employed with the organizations after completing internships. (Knockalva Principal) As a result, HEIs have recommended that Advance consider rolling out elements or activities of labor￾bridging services to HEIs, allowing the private sector to play a bigger role in HEIs—for example, through support with identifying and aligning private sector entities and industry players with degree programs —as well as supporting HEIs with the development of memoranda of understanding (MOUs) that could be used as a template to foster partnerships between HEIs and the private sector This process of developing MOUs was started at UTECH, for example, but was not completed. The HEIs were introduced to the concept of labor-bridging and were even provided with lists of proposed entities with which they could engage, but the process was not completed due primarily to timing and the end of technical and financial support by Advance to the HEI. Notwithstanding the limited focus on labor-bridging services, HEIs highlighted that they were exposed to the importance of engaging with the private sector, which resulted in several benefits, including an increase in internships and work experiences for their students. For example, UTECH and VTDI were exposed to the private sector and their expertise in curriculum development. Other ways of engaging the private sector would be through accessing internships, work experience, and local study tours for students. The design of courses under the Advance Program also encourages this since courses developed have a strong focus on experiential learning. In the case of Knockalva, the application of 23 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV labor-bridging led to several graduates in the first cohort of the agri-business degree successfully gaining employment upon graduation. Faculty of Knox and Knockalva indicated that internships were a vehicle that increased the knowledge and skills of students and were complementary to the curricula. This was because both lecturers and students were able to engage with a range of private sector entities though internships and study tours, for example, allowing partnerships between academia and the private sector to be strengthened. Whilst there are no formal MOUs between private sector firms and the HEIs, HEIs have indicated that the notion of internships, local study tours, etc., introduced by Advance have opened doors in terms of engaging with the private sector to the benefit of students. The approach to labor-bridging also involved capacity building of staff and students. For example, at VTDI and the other HEIs, labor-bridging workshops were conducted, covering areas such as resume￾writing, developing business plans, and interview skills. It also included supporting students in the animation program with the development of their portfolios. In the case of VTDI, however, they indicated that while there was insufficient data to determine the success of labor-bridging initiatives, some students, following interventions like these training workshops, pursued self-employment. This partly reflects the impact of the entrepreneurship and business plan training and the associated content of those workshops, all part of the labor bridging component of Advance. “One key advantage of these workshops was that they enabled students to become more confident on how they approached the job market on one hand, and also enabled them to get into a business frame of mind and understand how they could become entrepreneurs” (VTDI Official). They were able to engage in visioning exercises that helped them to see themselves in the future and define what they wanted to become. As one student at Knox Community college indicated, “Agro-processing and business management jobs are generally difficult to find. However, the program helped us to pursue entrepreneurial pursuits in terms of opening your own businesses.” 5.1.4. ADMISSIONS AND RECRUITMENT This area of research corresponds to the Advance Program Result 2: increased access for disadvantaged youth in target academic programs and the following intermediate result: • Expanded access through improved recruitment and admissions practices The overarching outcome of Advance is to support disadvantaged youth to access higher education. Therefore, it is important to understand the profile of the youth that were admitted to the HEIs and benefited from Advance scholarships. The matriculation requirements for students that are desirous of taking Advance courses are categorized as follows: • General: Five (5) subjects in Caribbean Secondary Education Certificate / General Certificate of Education including English Language and Mathematics with grade I, II, or III / A, B, or C. or any equivalency recognized by the University Council of Jamaica. • Pathways: Flexible enrollment that allows interested students to further their education with whatever qualification they have in hand. The HEI would then create a personal development plan for each student admitted via this route to help them achieve their educational goal. • Mature Entry: Work programs / experiences in accordance with the Caribbean Community’s and Common Market (CARICOM) procedures for Accreditation of Prior Learning (APL) with USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 24 special consideration given to mature students (those 25 and over) under CARICOM’s Mature Students Clause. The profile of the students that were accepted into Advance courses provides a greater understanding of how and why general admissions and recruitment systems and processes within HEIs had to change to enable achievement of the program’s outcomes (see Table 4). Notwithstanding the general agreement on the role of Advance in making opportunities available to disadvantaged youth, HEIs presented a range of issues as to why some of the project outcomes were not fully achieved, many of which were related to the profile of Advance students. There was consensus across all HEIs on the profile of Advance students as shown in Table 4. “Definitely most disadvantaged, there was evidence of economic deprivation, they could not function outside of Advance funds. When funds from Advance were late, they were unable to buy food or attend classes.” (UTECH Dean) “Many did not value education because of their circumstances or mindset. The Advance students were not as motivated as regular cohort. Many students had challenges related to food, travelling, and accommodations.” (VTDI Official) “They were really the most disadvantaged, although the entire cohort was disadvantaged. They had socioeconomic vulnerabilities, some had completed secondary education level training and had become disengaged from the formal education system because they simply could not afford to move on to further training.” (Knockalva Principal) TABLE 4: ADVANCE STUDENTS’ PROFILE Socioeconomic Status Intellect and Readiness for Higher Education • Most disadvantaged • Evidence of economic deprivation and other socioeconomic vulnerabilities (e.g. living in poverty, part of a single-parent household, living in volatile communities, unable to find employment, limited access to resources such as potable water, sanitation services, food, etc.) • Demotivated and disillusioned due to their economic and social circumstances • Many had experienced unfortunate circumstances, such as losing their parents; losing their homes due to fires or civil unrest; prematurely thrusting them into adulthood • Lowest in terms of qualifications compared to non-Advance HE students • Some had cognitive deficits due to lack of food and nutrition • Low levels of value for education because of mindset, lack of interest, and value system • Some had not completed secondary education and entered through the pathways category of matriculation • Many were disengaged from the formal education system because they simply could not afford it • Many were not ready for the university environment in terms of their social and psychosocial skills 25 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV Socioeconomic Status Intellect and Readiness for Higher Education • Limited finances to access higher-level education At least three HEIs indicated that Advance students “were not of the same quality as their other students that matriculated normally through the university, as they were lowest in terms of qualifications and were not ready for the university environment in terms of their social and psychological skills, resulting in a high degree of failures and dropouts. …They were miles apart from students that matriculated through the University.” (UTECH Official) On the other hand, Knockalva held differing views of Advance students with an official stating that “Advance students were self-assured and exhibited a high level of confidence. They moved into a realm of financial independence as their financial burden was eased tremendously. The difference in academic performance was evident: 100 percent success rate for all courses of Advance scholarship students, with GPAs of 3.0 and above compared to non-scholarship students... [whose] GPA varied between 2.5 and above.” Whilst the HEIs have experience with students from lower socioeconomic groups, including students from quintiles one and two (the lowest socioeconomic groups), the areas of greatest concern were their intellect and readiness for higher education. These lower levels of intellect and qualifications for admission required, for example, changes to marketing as well as how HEIs approached prospective students generally. While Advance helped with marketing, including preparing advertisements and hosting events to bring together HEIs and the target group, the HEIs did the actual advertising, engaged prospective students at Advance events, encouraging them to apply, providing them with information about the scholarships, and doing the initial recruitment (short-listing of prospective students). Once the students qualified at the HEI level based on one of the three categories of matriculation above, Advance undertook further interviews with the students to ensure that these prospective students met the Advance requirements related to vulnerability and this resulted in final selection. “The [Advance] interview process would have ensured that the most vulnerable students were targeted and selected” (UTECH Official). The overall changes of HEIs based on the requirements of Advance for recruitment and selection are presented below: • Most HEIs indicated that the admissions process did not change, as they already had good recruitment and selection practices in place. • Advance better enabled HEIs to undertake assessments of vulnerable youth and determine their suitability for higher education. • The profile of students for admissions changed across HEIs, as in many cases some of the students accepted to the Advance Program would not have normally matriculated. • The marketing and targeting strategies of HEIs were enhanced as they were able to gain competencies in marketing strategies and other related measures for reaching out to disadvantaged youth. USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 26 • Higher than normal volumes of applicants were generated because of the availability of scholarships. 5.1.5. ACCESS TO SCHOLARSHIPS This area of research corresponds to the Advance Program Result 2: increased access for disadvantaged youth in target academic programs and the following intermediate result: • Scholarships for disadvantaged youth “I appreciated not having to worry about tuition as a result of [my] scholarship, and this afforded me the freedom to focus on my studies.” (UTECH student) “Without the program, I would not have been able to pay monthly rent and it would be hard to pay the school fee.” (Knox student) An important component of the Advance Program was the provision of scholarships to improve access to higher education for disadvantaged and vulnerable youth. These were full scholarships covering tuition, stipends, transportation, accommodation, and course materials. In the various interviews and focus group discussions, the officials of the HEIs indicated that the provision of scholarships was the impetus for applications to the program, as most of the disadvantaged youth would not have been able to afford/access tertiary-level education otherwise. Staff and faculty highlighted the hardships students experienced when funds for tuition and stipends for transportation and accommodation were delayed— to the extent that students were not able to attend classes or pay their rent, with faculty often having to assist students through their own personal resources. While the provision of scholarships afforded access to disadvantaged and vulnerable youth, there are examples of dropouts and students not completing their respective programs due to failing courses. This was evident in three of the four HEIs, and more pronounced at UTECH and VTDI. The HEIs attributed these trends to several factors, including the delay in the disbursement of the scholarships, the impact of COVID-19 on the program, which resulted in the shifting of classes online, with some students not having access to the Internet and electronic devices, and in some cases, a lack of socio-emotional support. “Students under the Advance Program didn’t perform as expected. Students required additional support, including motivation, to complete the program, with some not completing. To date some students are being encouraged to complete” (VTDI Official). Students at UTECH and VTDI were most affected by scholarship disbursement issues based on the information shared by officials of both institutions. This was on one hand due to the requirement of Advance to pay expenses related to food, transportation, and accommodation as reimbursables and meaning students had to fill out complex forms as indicated by HEI officials. This process required the accounting staff of the HEIs to support the students to get this done. The HEI would then make the request for these reimbursables to Advance. Payment would be made to the HEI and the students would then be paid. This was considered a cumbersome and lengthy process. In the case of VTDI, there was another hurdle as that institution does not operate its own bank accounts. Instead, its bank accounts are managed through its head office, Heart Trust NTA. Payment would then have to be received by Heart and then the monies were forwarded to VTDI and then to the students. Discussions 27 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV by the HEIs with Advance led to a change in the system with FHI 360 dropping the requirement for reimbursables and instead paying monthly stipends directly to student bank accounts. Table 5 outlines the number of scholarships awarded per HEI and illustrates that UTECH and VTDI saw less than 50 percent completion of the program—in spite of the provision of scholarships. It should be noted that FHI 360 addressed some of the reasons for students withdrawing from the program at UTECH and VTDI by the time implementation began in the community colleges. For example, the challenge outlined below at UTECH would have been resolved prior to the implementation of the Advance Program at the community colleges. “For students that needed the support from the first year, some students dropped out due to frustration from not being able to meet costs related to travel, accommodation and food, as first-year students benefited from the allowance in their second semester of second year. This was too late for many students that were in need” (VTDI Official). TABLE 5: SCHOLARSHIPS PROVIDED BY ADVANCE AND RETENTION RATES HEI Scholarships Awarded Retention and Success Rates University of Technology • Total - 40 scholarships • 22 scholarships awarded for the ASc. degree in EDPT • 18 scholarships awarded for the ASc. degree in ICFS • For EDPT, 12 completed, 3 withdrew without permission and 7 have an incomplete status due to issues such as failing one or more courses • For ICFS, 7 students completed, 4 withdrew without permission, and 7 have an incomplete status Percentage graduating – 47.5 Vocational Training and Development Institute For the 2018 – 2021 period ● Total - 35 scholarships (2018) ● ASc. degree in Animation – 23 scholarships ASc. degree in Entertainment and Events Management – 12 scholarships January to March 2022 ● 15 scholarships awarded for the ASc. degree in Animation ● 12 scholarship holders have graduated/completed studies ● 9 scholarship holders dropped out ● 14 scholarship holders’ studies are delayed15 Percentage graduating – 45.7 15 Delays include students who do not enroll in courses during as academic year due to multiple reasons, but who have not dropped out of the program. Delayed students include, among others, those who have deferred studies for a year or for whom the university has temporarily suspended courses needed to continue in their degree. USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 28 HEI Scholarships Awarded Retention and Success Rates ● 6 scholarships awarded for the ASc. degree in Entertainment and Events Management Knockalva Community College • ASc. degree in Agro-Processing and Business - In the first year, the institution was awarded 12 scholarships to complete 2 years of study • 100 percent success rate for all courses taken by Advance scholarship students – GPAs of 3.0 and above Knox Community College ● ASc degree in Agri-business and Management - first cohort received 10 scholarships ● For the second cohort, 12 scholarships were disbursed ● For the first cohort, 4 moved on to the degree program and are in year 4 (senior) currently; 4 of 10 dropped out of the program from year 1; 5 completed; 1 student needs to complete ● For the second cohort, 3 students completed the program with 2 pursuing the BSc. in Environmental Studies at the college. These 3 students have GPAs ranging from 2,46 - 3.23. Nine students have not completed due to needing to resit examinations Percentage graduating or moving on to a degree program in the first cohort – 50 Percentage graduating or moving on to a degree program in the second cohort – 25 As summarized in Table 6, the officials of the HEIs identified several factors that influenced retention rates. These were then divided into those that were specifically related to the students themselves and the other into the structure of the Advance Program itself. TABLE 6: FACTORS INFLUENCING RETENTION RATES Student-Related Structure of the Advance Program ● Advance students were not as motivated as the regular cohort – who had to pay. Money did not really motivate Advance students ● No contract existed between the Advance Program and students; no accountability mechanism was in place to improve 29 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV Student-Related Structure of the Advance Program ● Many students were not equipped to engage effectively in an HEI - they struggled with punctuality and completing assignments on time. UTECH for example recommended that programs such as Advance should consider putting in place a summer program or a 3-month program that would guide these students on how to function effectively in an HEI ● COVID-19 resulted in some students not being able to fully access online platforms, due to lack of electronic devices as well as access to the Internet, bearing in mind that community wi-fi hotspots in Jamaica are limited. This impact was not limited to Advance students but applied to students across Jamaica ● Psychosocial support for students was insufficient. Even where HEIs had a guidance counselor in place, they may not have had the expertise or experience to support the needs of these disadvantaged youth. Given the targeted group, there was a need for a more structured program that included higher levels of psychosocial support commitment levels to the program – as such it was easy for students to abandon the program ● Advance did not pay for resitting of examinations (where students failed on first attempt) and this affected the overall success of the program as the students did not have the resources to pay for the resits and the HEIs did not change their procedures to accommodate free resits for these students. Table 5 shows, for example, that 35 percent of students at UTECH have an incomplete status as they have failed one or more of their courses. In the case of Knox, 75 percent of the second cohort have not completed as they too need to resit examinations ● At the beginning of the program, many students had issues related to the cost of food, accommodation, and travel. In response, the Advance Program introduced allowances for these expenses, which greatly assisted students. However, many students still had challenges related to the timeliness of these scholarship disbursements (applicable more to the first set of scholarship students who were required to pay first and then be reimbursed later) ● Administrators at UTECH and VTDI moreover indicated that Advance did not quite recognize the cognitive deficits of the most disadvantaged youth – due in part to issues such as lack of food, etc., and the potential impacts that this could have on educational outcomes. The fact that the selection of youth were primarily from quintiles 1 and 2 would suggest that it is highly probable that students could have lacked food and/or decent shelter, with these factors having an impact on learning outcomes Notwithstanding the lower-than-favorable completion rates in at least two of the HEIs, information gleaned from the KIIs indicated that many students that completed their program have been able to obtain jobs, engage in entrepreneurial activities, or even continue on to bachelor’s degree programs. As USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 30 the program rolled out to community colleges, FHI 360 became more involved in the human and social peculiarities, and increased their levels of interaction, supervision, and monitoring of the program with both the HEIs and students, resulting in FHI 360 being more aware of needs and emerging difficulties and were able to address potential issues and challenges before they emerged. This was indicated by both Knox and Knockalva, with the principal of Knockalva stating that Advance stood out among other programs she had been involved with and the “human interaction of Advance was evident. FHI 360 was consistent in supervising and monitoring the program and was always aware of needs and emerging difficulties.” 5.2. RQ2. SUSTAINABILITY AND INSTITUTIONALIZATION OF ADVANCE IN HEIs This section includes an assessment of the extent of institutionalization and sustainability across the five key areas: 1) curriculum and pedagogy: 2) faculty/staff professional development; 3) labor-bridging services; 4) admissions and recruitment; and 5) scholarships, when compared to the progress markers that were established to assess the level of institutionalization of each of the five areas. Figure 6 provides a summary description based on the ET analysis of how each institution approached sustainability and institutionalization. 16 Thus, Figure 6 serves to provide an overall overview of each area for each participating HEI. Additionally, Annex 1 provides a detailed breakdown of the level of sustainability and institutionalization for the different subcomponents by institution, and the following sub-sections provide an in-depth analysis of the level of sustainability and institutionalization for each area. The level of institutionalization and sustainability by key area varies both by area and by HEI, but there are some common trends. Overall, the evaluation team found that all areas are at least at an emergent level of institutionalization and sustainability, which means that the majority of changes introduced by Advance in methodologies, approaches, and techniques are either established or advanced, with some in the early stages of planning. • Advance made available six new or revised courses that are fully aligned with the needs of the labor market and for which there were existing gaps. These courses are already fully institutionalized across HEIs in Jamaica, which means that even after the Advance Program has ended, these courses will be offered. Specifically, institutionalization signifies that the programs developed by Advance are now part of the overall courses and programs offered at the HEIs. For example, at Knox the program developed by Advance has been offered four times and the HEI has approached the University Council of Jamaica to seek accreditation. At UTECH, the Dean informed the ET that both programs are fully institutionalized, having gone through all the university processes for approval such as the academic board for example. In some cases, such as at Knox, Knockalva, and UTECH, modules under programs are already being offered in other programs. At Knox, the administration has enabled students undertaking environmental studies to be able to have a specialization in agro-processing and business management. At Knockalva, where program integration is evident, using the courses under the agro-processing program, the HEI was able to make adjustments to its crop production program to feed directly into agro-processing program. Crops produced under the agro￾processing program such as cocoa, scotch bonnet peppers, and pineapple were provided to 16 Rank description. Latent level: No concrete changes, and no plans in place. Emergent level: Early planning phase to include changes brought by Advance. Established: Evidence of application of Advances approaches at the individual HEI level. Advanced: Strong application and institutionalization at the individual level of Advance approaches. 31 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV the agro-processing program to process into products such as corn pork (a pickled type of pork cured using salt and similar to salted cod fish), beverages, and jams and wines. According to the HEI, twinning food production with agro-processing has enabled students of both programs to better understand the value added in agriculture. • At all HEIs, training opportunities provided for staff have been passed on to students via course delivery. The community colleges created communities of practice among themselves (two that were part of this evaluation as well as others offering the ASc. degree in agri-business developed by Advance) to share information and best practices in course delivery. VTDI developed a student advisement center and has institutionalized the role of a student advisor for all its programs to better support students to achieve improved learning outcomes. • All HEIs have already taken steps to couple academics with labor-bridging services in courses of study outside of Advance. Knox, for example, continues to foster their relationships with different entities for study tours, and has established relationships with the Scientific Research Council to support the agro-processing program in terms of product testing, etc. The design of the curricula for Advance courses and the coupling of theory with practical application will ensure that the linkages with industry continue. This approach in curriculum development was shared by all HEIs that were part of the study. One can also extrapolate this to the other five community colleges that share the same curricula with the two under investigation. Additionally, all HEIs understand the importance of supporting their student population to gain access to the labor market, and all HEIs have started delivering workshops on softer skills (resume-writing, interviews etc.) to the wider school population. • All HEIs have started to utilize the marketing and recruitment approaches of Advance to target disadvantaged youth in their programs. UTECH will now admit students with lower academic levels in associate degrees that were developed with the support of Advance and support these students to adjust to university based on Advance approaches such as SEL that they have adopted. Under regular circumstances, UTECH has English and mathematics requirements for matriculation. To ensure that the most vulnerable students are able to access the associate degree programs, UTECH created a special mathematics and English course that students could take to meet the university requirements. • Most HEIs recognized the value and benefit of providing access to scholarships. Consequently, three of the four HEIs began exploring approaches to access resources and continue to provide financial aid to the most disadvantaged students. Each HEI adopted different approaches, including developing their own scholarship programs. UTECH included the Advance courses as part of its scholarship programs, thereby enabling interested students to be able to apply to the HEI’s scholarship program. UTECH also approached the Ministry of Education to access support for scholarships for these two new associate degree programs that would target vulnerable groups. The Ministry agreed to provide ten half scholarships annually for students enrolled in the program. Knockalva has developed a scholarship program for disadvantaged and vulnerable youth, similar to the approach of Advance and approached alumni of the HEI to support and kick-start this program. The first set of scholarships are to be delivered in academic year 2022/23. Some students at Knox for example were also able to USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 32 access the Rio Tinto Scholarship offered by Jamaica 4H (a scholarship open to students that are from one of three parishes—Manchester, St. Ann, or St. Catherine—and which covers tuition), deferring this scholarship as they were also holders of Advance scholarships, and utilizing the Rio Tinto Scholarship to complete the degree program upon finishing the associate degree. Figure 5 illustrates the sustainability and institutionalization of key elements of Advance within all the HEIs by sphere of influence according to the Outcome Mapping Methodology. 17 It is important to note that several elements of Advance have been institutionalized across the HEIs, but at varying degrees across the five fields of study. The figure shows that the Advance Program influenced several internal procedures within HEIs around the five areas of study and illustrates the difference that Advance has made to higher education in Jamaica. Various measures, tools, and ways of working that Advance introduced were key to fostering institutionalization and sustainability. For example, the approaches to curriculum development, including the importance of participatory approaches involving several stakeholders, such as industry experts, and the role of undertaking needs assessments as well as standardizing experiential learning in curricula, were tools that all HEIs agreed were important to institutionalize across departments and faculties within their HEIs. The section below provides information about the extent of institutionalization and sustainability across the five areas of study in more depth. 17 Annex 2 includes Outcome Map tables by HEI. FIGURE 5: SUMMARY OF SUSTAINABILITY AND INSTITUTIONALIZATION OF KEY ELEMENTS OF ADVANCE INPUTS: ACTIVITIES AND OUTPUTS The Advance Program •Curriculum development •Resources for scholarships and for the professional development of faculty and staff •New ways of working and engaging with private sector and disadvantaged youth OUTCOMES: CHANGES IN BEHAVIOR Outcomes: What was gained by the HEIs •New and better curricula, meeting the needs of the labor market •Introduction to experiential learning •Labs, student services, and new methods and approaches to marketing, recruitment and scholarship programs IMPACTS: CHANGES IN STATE The difference Advance has made •Programs aligned to labor market and youth needs across HEIs •Application and adoption of experiential learning across HEIs •Improved services for students, including advisement centers, scholarship programs, and more disadvantaged students admitted 33 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV FIGURE 6: LEVEL OF INSTITUTIONALIZATION AND SUSTAINABILITY BY HEI AND BY KEY AREAS USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 34 5.2.1. CURRICULUM AND PEDAGOGY Some of the highlights of institutionalization and sustainability related to curriculum and pedagogy include: 1. All six courses, new or revised, have been fully institutionalized and the HEIs are able to offer them to future students, outside of the support of the Advance Program. 2. The support Advance provided to UTECH has allowed this university to be able to expand its degree offerings, to include associate degrees for the first-time. 3. Across all HEIs, the exposure to experiential learning was evident, and was included in course curricula, enabling these approaches to be continued even after Advance ends—and also used in courses that Advance did not develop. This illustrates the value HEIs learned to place on experiential learning. Knox Community College has begun to incorporate experiential learning into other degree programs at the college – such as the BSc. degree in Environmental Studies. 4. Knockalva has approached the University Council of Jamaica for accreditation for the ASc. degree in Agri-business. One of the main successes of the Advance Program was its making available new or revised courses that are fully aligned with the needs of the labor market and for which there were existing gaps. These courses are already fully institutionalized across HEIs in Jamaica, which means that even after the Advance Program has ended, these courses will be offered. One unintended but positive outcome of Advance has been the interest from the wider community of Eastern Hanover (a very rural area) to access training in agro-processing. This has come about as a result of the agro-processing lab Advance provided to Knockalva. Knockalva has indicated that it will be developing a six-month certificate program in food processing for adult learners. The lab is also available as a resource center to the wider community as there is no other agro-processing lab on that side of the island. The extent of institutionalization of curriculum and pedagogy at the HEIs is presented in Table 7. 35 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV TABLE 7: PROGRESS MARKERS - CURRICULUM AND PEDAGOGY INSTITUTIONALIZATION AND SUSTAINABILITY Extent of Institutionalization and Sustainability Latent level (No concrete changes, no plans in place) Emergent level (Early planning phase to include changes) Established (Evidence of application at the individual level) Advanced (Strong application and institutionalization at the individual level) UTECH will adopt the participatory approaches to course development introduced by Advance and engage industry experts in curricula development of appropriate courses of study Knox is considering expanding the ASc. in agri￾business to a full degree program The involvement of Advance established the importance of engaging with the labor market survey information and having conversations with employers. This was true for all HEIs There is a shift within faculty in terms of the sources to draw on in making key decisions regarding the design and development of new programs (UTECH) Knockalva will be developing a certificate in food processing – a six￾month program for adult learners All HEIs have new ASc. degree programs as part of their institutional profile which can be offered in the future Changes in teaching and learning strategies, assessment strategies and types (VTDI) All HEIs are employing social￾emotional learning across their HEIs All HEIs will employ and expose students to hands-on learning and experiential learning approaches The agro-processing lab Advance provided to Knockalva has resulted in interest from the wider community for training in agro-processing The processing lab is the only of its kind on that side of the island (in eastern Hanover) and also is available as a resource center to the wider community Knockalva has applied for accreditation of ASc. degrees from the University Council of Jamaica Knockalva has started undertaking training needs assessments with respect to the review of existing curricula and development of new curricula to reflect the needs of the labor market to enable its course offerings to be more relevant. Courses for the community colleges are now uploaded online – allowing for online instruction in the event of any disruption to education, as occurred during the COVID-19 pandemic, but this also allows for online delivery mechanisms to be adopted to attract students for whom online learning could be a more suitable and cheaper option Knox has begun reviews of other degree programs and associated curricula based on the approaches shared by Advance, such as experiential learning. They have incorporated and applied practical elements and experiential learning to the BSc. degree in Environmental Studies Knox has linked one of its degree programs in environmental science to the Advance-developed program in agri-business, creating the opportunity to provide specializations in agri-business within the environmental science degree (BSc.) At Knox, faculty have started to adapt experiential learning across courses outside of Advance by allocating hours for USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 36 Extent of Institutionalization and Sustainability Latent level (No concrete changes, no plans in place) Emergent level (Early planning phase to include changes) Established (Evidence of application at the individual level) Advanced (Strong application and institutionalization at the individual level) teaching and hours for practical and experiential learning – already completed for students undertaking the environmental science degree program Knockalva has facilitated program integration of the Advance ASc. in agro-processing with the ASc. in agriculture production, enabling both sets of students to appreciate the linkages between the two programs and to showcase the notion of “produce to product” and value￾added in agriculture, as well as synergies between agriculture production and enterprise development Adjustments were made to the crop production program to feed directly into the agro-processing program Under the crop production program, at Knockalva agro￾processing students are turning crops such as cocoa, scotch bonnet peppers, and pineapple into products such as pickled corn pork, beverages, and jams and wines – effectively twinning food production with agro-processing 37 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV 5.2.2. PROFESSIONAL DEVELOPMENT OF FACULTY AND STAFF Some of the highlights of professional development related to curriculum and pedagogy include: • The capacity and skills of staff, primarily across UTECH and VTDI, have been strengthened in curriculum development and design, as Advance fully engaged faculty in the redesign and the development of the degree programs. • The community colleges created communities of practice among themselves (two that were part of this evaluation as well as others offering the ASc. degree in agri-business Advance developed) to share information and best practices in course delivery. • The exposure to training in SEL was new to HEIs, but they have come to appreciate how this is relevant to all students, not only to Advance students or disadvantaged youth, and have therefore created the mechanisms to employ these approaches across their HEIs. • Exposure to coupling academics with professional development activities and training is another area that was new to HEIs, but one in which they have agreed to take forward. TABLE 8: PROGRESS MARKERS - PROFESSIONAL DEVELOPMENT INSTITUTIONALIZATION AND SUSTAINABILITY Extent of Institutionalization and Sustainability Latent level (no concrete changes, no plans in place) Emergent level (early planning phase to include changes) Established (Evidence of application at the individual level) Advanced (Strong application and institutionalization at the individual level) N/A At UTECH, mentorship is taking place – lecturers who interfaced with Advance are encouraged to provide mentorship to other lecturers as it relates to curriculum design, labor bridging, etc. At all HEIs, training opportunities provided for staff have been passed on to students via course delivery UTECH has indicated that the faculty that Advance worked with have broadened their approaches to teaching and have incorporated experiential learning into all their courses, many of which are outside of the Advance curriculum Knockalva continues to employ new training strategies Advance introduced across other VTDI developed a student advisement center and has institutionalized the role of a student advisor for all its programs to better support students to achieve improved learning outcomes USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 38 Extent of Institutionalization and Sustainability Latent level (no concrete changes, no plans in place) Emergent level (early planning phase to include changes) Established (Evidence of application at the individual level) Advanced (Strong application and institutionalization at the individual level) courses, and share them with other faculty 5.2.3. LABOR-BRIDGING SERVICES Some highlights related to institutionalization and sustainability of labor-bridging services include: • HEIs have a better appreciation of the importance of labor-bridging services, including providing students with knowledge and skills beyond academics to enhance their chances of accessing the labor market. HEIs will therefore provide students with workshops on softer skills (e.g., resume-writing and interview skills, entrepreneurship, and building of their LinkedIn profiles). • HEIs have indicated that they will continue to engage industry for support to students and foster and strengthen those relationships to expand access to internships, study tours, and industry engagement. • HEIs have expressed interest in integrating ideas and concepts of labor-bridging services from Advance into their other programs. TABLE 9: PROGRESS MARKERS - LABOR-BRIDGING SERVICES INSTITUTIONALIZATION AND SUSTAINABILITY Extent of Institutionalization and Sustainability Latent level (No concrete changes, no plans in place) Emergent level (Early planning phase to include changes) Established (Evidence of application at the institutional level) Advanced (Strong application and institutionalization at the institutional level) UTECH has drafted MOUs with industry, but none have been signed HEIs in general are interested in establishing MOUs with industry to All HEIs have indicated that they will continue to engage the private sector and industry leaders for access to internships for students and to extend experiential learning outside the classroom All HEIs have indicated that they will integrate All HEIs have already taken steps to couple academics with labor-bridging services in courses of study outside of Advance. Knox, for example, continues to foster their relationships with different entities for study tours, and has established relationships with the Scientific Research Council to support the agro￾Knockalva will use its agro￾processing lab to support training in agro-processing in the wider Hannover community 39 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV Extent of Institutionalization and Sustainability Latent level (No concrete changes, no plans in place) Emergent level (Early planning phase to include changes) Established (Evidence of application at the institutional level) Advanced (Strong application and institutionalization at the institutional level) support labor￾bridging services ideas and concepts of labor-bridging services from Advance into their other regular programs processing program in terms of product testing, etc. All HEIs understand the importance of supporting their student population to gain access to the labor market All HEIs have started delivering workshops on softer skills (resume-writing, interviews etc.) to the wider school population 5.2.4. ADMISSIONS AND RECRUITMENT Some highlights related to institutionalization and sustainability of admissions and recruitment include: • HEIs are better able to engage with disadvantaged youth and have a better understanding of the marketing and communication tools that would attract these students. • HEIs have indicated that they will continue to use the marketing and communication tools Advance introduced to them, not just for Advance courses but also across other courses of study. USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 40 TABLE 10: PROGRESS MARKERS - ADMISSIONS AND RECRUITMENT INSTITUTIONALIZATION AND SUSTAINABILITY Extent of Institutionalization and Sustainability Latent level (No concrete changes, no plans in place) Emergent level (Early planning phase to include changes) Established (Evidence of application at the individual level) Advanced (Strong application and institutionalization at the individual level) HEIs believe that it is important to understand and better address the needs of all students and adopt a student￾centered approach to higher education. They now reflect more on student needs to support learning outcomes and the overall success of students N/A All HEIs have started to utilize Advance’s marketing and recruitment approaches to target disadvantaged youth UTECH will admit students with lower academic levels and support them to adjust to university based on Advance approaches such as SEL that they have adopted for their university-level students 5.2.5. ACCESS TO SCHOLARSHIPS One of the tenets of the Advance Program rests on the provision of scholarships to realize the outcome of providing access to disadvantaged and vulnerable youth to technical education that meets the demands of the labor market. Several of the HEIs recognized the value and benefit of providing access to scholarships, one of which is increased enrollment in program offerings. Consequently, a few HEIs have considered continued financial aid using or adopting different approaches, and developing their own scholarship programs, outlined below. • UTECH approached the Ministry of Education regarding the provision of scholarships for the two programs developed under Advance, with the Ministry now providing at least ten half￾scholarships per annum to this university. Additionally, students are free to apply for any other scholarships available at the University. • Knockalva Community College has developed a scholarship program for disadvantaged youth, similar to Advance, and has approached alumni to support this program, which is planned for launch in academic year 2022/23. 41 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV • Knox Community College has also been pursuing discussions with alumni to further strengthen their involvement in providing financial aid through scholarships, as well as engaging the business community to provide scholarships. • VTDI has been considering engaging the private sector to support scholarships and internships under the two Advance Programs, given that the programs are designed to directly support the labor needs of the industry and strengthen the capacity of the orange economy. TABLE 11: PROGRESS MARKERS - ACCESS TO SCHOLARSHIPS INSTITUTIONALIZATION AND SUSTAINABILITY Extent of Institutionalization and Sustainability Latent level (No concrete changes, no plans in place) Emergent level (Early planning phase to include changes) Established (Evidence of application at the individual level) Advanced (Strong application and institutionalization at the individual level) N/A VTDI used existing scholarship programs and other modes of financing, such as earn-as-you-learn, to attract disadvantaged youth Both VTDI and Knox have approached the private sector to support the provision of scholarships to students that are part of Advance￾related courses Knockalva has developed a new scholarship program similar to Advance with support from alumni Knockalva will access scholarships from the Hanover Charity and Rio Tinto Charity and the College’s alumni as well as Hi Pro (private sector company) to support vulnerable youth UTECH approached the MOEYI for scholarships for disadvantaged youth for Advance courses and has received 10 half-scholarships per annum to apply to the ASc. degrees Advance developed UTECH has made accommodations for student to access regular UTECH scholarships for ASc. degrees Advance developed Knox has been able to gain access to scholarships funded by the Ministry of Labor and Social Security under the PATH (Programme of Advancement Through Health and Education) Program18 for students who maintain a GPA of 2.5 or above 18 The Programme of Advancement Through Health and Education (PATH) is a conditional cash transfer program funded by the Government of Jamaica and the World Bank. It aims to deliver benefits by way of cash grants to the most needy and vulnerable in the society. PATH was introduced island-wide in 2002. USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 42 5.3. RQ3. ENABLING AND INHIBITING FACTORS This section describes both the contextual and institutional factors that enabled or inhibited implementing, scaling up and the institutionalization of Advance. This section was informed by FGDs as well as KIIs. 5.3.1. ENABLING FACTORS ACROSS HEIs As captured through FGDs and KIIs, the positive impact of the Advance Program was due in part to several enabling factors related to organizational governance, program design, resources and capacity, and levels of engagement. Some of the factors that contributed to the success of Advance included: • The overall design of Advance and its all-encompassing approach was an enabling factor in and of itself, specifically to focus on disadvantaged youth, curriculum development, capacity-building of faculty, access to scholarships, and labor-bridging services. • The overall design of the Advance Program was conceptualized for Jamaica and the actual needs of its labor market, which have remained largely unchanged from the studies undertaken in 2016. For example, with respect to the curricula, the practical activities were based on and aligned with Jamaica’s industries; curriculum design was linked to the Jamaica setting. • There was strong evidence of consistent supervising and monitoring of the program, which made FHI 360 aware of needs and emerging difficulties at the earliest possible time and allowed them to take corrective action along the way. In this way, the Advance Program was iterative and built on continuous improvement. A good example of this was the introduction of SEL to help HEIs better support students that were not deemed to be fully prepared for higher education. • The curriculum design, which focused beyond theory to practical applications, including incorporating experiential learning, allowed courses to open up the imagination of students, which generated their innovative thinking. This kind of statement was expressed by all HEIs in varying forms. “The development of the content for the various modules in the specialized areas was a major contribution and the practical approaches to the particular programs as it relates to pedagogy cause great impact” (UTECH Dean). • Engaging faculty in curriculum design and adopting participatory approaches fostered ownership among HEIs and the CCCJ. The program was designed to allow for the interaction between faculty and experts from various fields, which augured well for ownership of courses but also enabled faculty to appreciate the role of the industry experts in curriculum development and design – another key aspect for fostering experiential learning. • Advanced contracted high quality consultants and experts to support curriculum development, from which faculty learned much. • The existence of teamwork within HEIs and the ability of faculty to work across departments was an enabling factor. For example, recruitment and scholarship officers already accustomed to working closely with faculty augured well for Advance. 43 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV • Providing access to capacity building and professional development sessions for staff, and providing them with new methods of teaching, particularly the incorporation of experiential learning, led them to consistently implement Advance activities. • Access to workshops and other capacity-building sessions in areas such as entrepreneurship helped faculty and HEIs to reassess their roles outside the classroom, and to better appreciate the importance of labor-bridging services. • Exposure to, and provision of, examples of tools such as MOUs for engaging with the private sector and other stakeholders. UTECH for example was provided with a draft MOU template that could be finalized to support future engagement with the private sector. They were also provided with a directory of professionals and experts that they could engage with related to the two courses Advance developed. • The availability of several workshops for students to complement their academic learning to better enable them to access the labor market; 100 percent of respondents, administrative staff of HEIs and faculty [of the quantitative survey], agreed or strongly agreed that Advance graduates are better able to access the labor market than others. • Overall, the commitment of HEIs enabled the Advance Program to be successful. The COVID-19 pandemic can both be considered an enabling and inhibiting factor to the success of Advance. On one hand, the pandemic led to the disruption in classes, labs, and engagement of students, due in some cases to students not having access to technological devices and/or the Internet. On the other hand, the quick response of FHI 360 to the pandemic to reduce the disruption in learning led to all courses being formatted for online delivery and placed on the relevant online portals and websites. In some cases, such as at UTECH and Knox, lecturers engaged in one-on-one sessions with some students who were not always able to attend online sessions, to prevent them from lagging behind. 5.3.2. INHIBITING FACTORS ACROSS HEIs Although the above enabling factors led to several Advance successes, the program also experienced inhibiting factors. These are presented below. Inhibiting Factors Related to Students • Some of the HEIs considered many students selected for Advance not ready for higher education, with many missing soft skills such as self-awareness, communication, and emotional intelligence, among others. This contributed to some students not adjusting well to the course of study; close to one-fifth of all scholarship awardees withdrew from their program without permission. Overall, 53 percent of all scholarship recipients have not completed their course of study: 20 percent dropping out and a little over 30 percent not completing due to having examinations to resit. • Administrators considered the emotional, socioeconomic issues students faced to have contributed in some of them not being able to fully engage in learning, resulting in undesirable levels of failures and dropouts. HEIs did indicate that students’ socio-emotional states were USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 44 not factored into the project design, nor was it considered at the time of HEI recruitment and selection. Therefore, HEIs such as UTECH had, at the initial stages, to develop their own programs to help support students. Later on in implementation, having recognized these difficulties, Advance provided training for all HEIs in socio-emotional learning to support their abilities to better help the disadvantaged youth. • Because no contract was established between students and the program or between students and the HEIs, there were no accountability mechanisms in place to help keep students in school, and there were no sanctions on the students if they dropped out of the program without a good reason. As such, students were able to drop out of the program with no consequences, which showed an insufficient commitment to the program by some Advance￾supported students. The approach Advance adopted to not have contracts between Advance and the students or the HEIs and the students is uncharacteristic of how most other scholarship programs in HEIs are administered. • COVID-19 did present several challenges for students as well as to other areas of the Advance Program. Activities that COVID-19 affected included those related to labor bridging services and the implementation of internships. These activities were disrupted due to the measures implemented by the government (curfews and lockdowns) to curb the spread of the virus. Internships and the experience gained by students as well as the partnerships fostered between industry and HEI were seen as key benefits of the program. While the pandemic affected all students in Jamaica and indeed the world, Advance students were particularly affected as Advance’s focus on experiential learning, incorporating practical exercises and demonstrations as well as internships, required students and teachers to be face-to-face, and this aspect of the programs could not easily pivot to being online. Inhibiting Factors Related to Systems and Processes of the Advance Program and the HEIs • At least two of the HEIs indicated that the Advance Program design did not take into account the readiness of the target group to transition to higher education. They indicated that many students did not understand how to operate or cope with their program of study or the requirements of the HEIs. Because no assessment was undertaken either by the HEIs or Advance prior to enrollment, it was only upon students starting the program of study that some social and emotional issues came to the fore. HEIs recognized later on that there was a need to prepare students for university, perhaps in a pre-program or readiness program, starting in the summer prior to the first semester (for about three months), to help them transition to university. In reflection, HEIs indicated that such a program would have also allowed these students to prepare themselves for university and be better equipped with writing and communication skills, which several lacked. UTECH, for example, implemented an intervention to help these students. Faculty met with students every Wednesday for two months and had two-to-three-hour sessions with them to better prepare them for higher education and introduced them to soft skills. They also brought in various professionals to engage with the students. • Advance’s requirements that the HEIs introduce new procedures and complex reporting requirements affected timelines and led to levels of frustration among the HEIs. “Program very demanding, and VTDI was not fully prepared for some of the demands that FHI 360 placed on the 45 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV organization. The volume of work for Advance posed challenges for the limited staff.” (VTDI Official). UTECH official indicated that “UTECH and FHI 360 had several clashes in terms of timelines. USAID’s projectized approach had set timelines that did not always correspond with UTECH’s timelines. Expectations of both parties were not always aligned. For example, administrative requirements to facilitate disbursements to students, the bureaucratic processes of both institutions were at odds at times, which adversely impacted the scholarship recipients.” • Advance’s not paying for resitting examinations led to undesirable rates of non-completion of programs, where students failed just one or two courses. Given their socioeconomic circumstances, these students are not able to pay out-of-pocket for such resits. • Engaging with industry players was not as successful, especially in the first two HEIs—UTECH and VTDI—and this negatively affected the component of the program related to labor￾bridging services. Many promised activities were not implemented, according to the abovementioned HEIs. The establishment of MOUs with entities/companies for internships and employment of students never fully materialized. However, this was not the case with Knockalva, as all students in their first cohort were able to access internships, with many becoming employed at the establishments they interned at once they graduated. • In the initial stages of the program, there were general delays in scholarship disbursements— especially at the beginning of the program, and with two HEIs (namely UTECH and VTDI) reflecting a conflict of the governance processes and administrative procedures of Advance and the governance and capacity processes within the HEIs themselves. At both UTECH and VTDI, the first cohorts of students were initially required to pay upfront costs related to food, housing, and transportation, and then be reimbursed. This created several issues, especially since these were poor and vulnerable students. The entire process was described as “cumbersome and lacked the evidence and appreciation of the contexts of the students” (UTECH Official). 5.4. RQ4. INDICATORS OF CHANGE The indicators of change include reliable intermediate indicators across each of the five key areas that can lead to anticipated long-term outcomes around institutional strengthening. As shown in Table 12, indicators of change analyzed include the percentage of students completing the degree program, the percentage of students requiring funding to complete their program due to failing one or more courses, the percentage of students dropping out of program, the percentage increase in vulnerable students accessing jobs, and the percentage of students moving on to a degree program following completion of Advance, among others. While the initial assessment of these indicators of change would have centered on a quantitative analysis, the HEIs were unable to provide sufficient data to undertake this assessment. Notwithstanding, all HEIs indicated that there were increases in the number of vulnerable students admitted to the degree programs. This was bolstered by the scholarship component of the Advance Program, which offered students that would not have been able to afford the tuition and related costs for higher education the opportunity to pursue the programs. Among the students that have completed their respective programs, some have sought to attain further qualifications by enrolling in Bachelor of Science degrees. This is evident in the case of Knox, where some students who have completed the Advance Program have transitioned to a Bachelor of Science (BSc.) in Environmental Studies program. USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 46 In the case of VTDI, respondents indicated that some students have been working independently following the completion of the program, and have capitalized not only on the degree program, but also on the other areas of training, such as the entrepreneurship training. This outcome aligns with a key objective of the Advance Program: to equip students with the knowledge and skills to start and operate their own enterprises in areas that are recognized as economically viable. Notwithstanding, except for Knockalva Polytechnic College, the completion rate of students was not 100 percent in the other three HEIs in this study. This was particularly evident for VTDI, which had the first cohort of Advance students. The institution indicated that while the recruitment successfully targeted disadvantaged youths, many were not able to afford costs related to traveling, food, accommodation, and exam resits. The first three areas were eventually addressed with the introduction of stipends to all Advance students, once it was recognized that tuition was not enough to allow these disadvantaged youth to attend school. This is another example that Advance incorporated the notion of continuous improvement in its operations, taking corrective action along the way. The lack of stipends for transport and food contributed to many students in the first cohort not completing the program. The Advance Program made adjustments for later cohorts through the provision of allowances for travel, accommodation, and food. These adjustments were implemented in the other HEIs, which enabled improved performance. There is evidence of changes to curriculum and pedagogy across all the HEIs, particularly those that worked closely with the FHI 360 team and experts. In the case of UTECH, Advance supported the introduction of the university’s first two-year associate degree program. This represents a change in the educational offerings of the university toward attracting a wider cross-section of students, including vulnerable youth or students that may have not excelled at secondary school leaving examinations, but still have an interest in pursuing higher education. Recognizing that the university’s requirements would inhibit some vulnerable students from attending, the institution introduced new entrance examinations to allow prospective students the opportunity to demonstrate their readiness for the programs. For the community colleges, the programs Advance supported were developed by the CCCJ and offered to its colleges. As such, most of the changes took place at the professional development level, where faculty and staff were provided with training related to delivery, including the use of experiential learning techniques, and the opportunity to participate in study tours (local) and other forms of training, such as entrepreneurial training. The impact of the study tours was also highlighted by VTDI, which indicated that the study tour on which the staff participated in Mexico was beneficial to all participants, exposing them to processes and procedures across a range of areas, and in turn benefiting the entire institute as the knowledge gained was translated into actions and institutionalized at the institute level. One major change was the establishment of a student advisement center and a more student-centered approach of the institutions in relation to the needs of their students. In the case of Knockalva, respondents indicated that Advance influenced internal procedures in areas such as curriculum review and standardizing practical sessions and theory for courses outside of the Advance Program—a clear indicator of sustainability—applying an approach from Advance to other areas. To support some of these changes, infrastructure adjustments included the retrofitting of the computer laboratory and the establishment of an agro-processing laboratory. This agro-processing laboratory will serve not only the college, but the wider community since this is the only agro￾processing lab on that part of the island. The agro-processing lab has also provided the college with the 47 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV opportunity to develop a new certificate program in agro-processing for community members and farmers who now have strong interest in creating value-added to their agricultural produce. Across all the HEIs there was evidence of new policies, guidelines, and procedures that were adopted and adapted from the Advance Program. This was particularly evident in the changes to curriculum and professional development and recruitment. Despite this, respondents were unable in many cases to clearly demonstrate where these changes have been formalized in documents/policies, and as part of a wider process of integrating and concretizing the outcomes of the Advance Program. Given the timing of this assessment, with some students yet to complete the program and others having only recently completed it, data and information on the success of the labor-bridging activities are limited. Some HEIs indicated the need for assistance in gathering data and information toward determining the number of students that have successfully transitioned to their aligned industry or have started an entrepreneurial venture. Quantitative data outside of completion rates of students, such as data for indicators of change related to career completion rates, among others, were limited across all HEIs primarily because HEIs did not collect data on the Advance Program in this manner. Whilst it was clear that Advance established reporting requirements, the monitoring and evaluation framework around indicators and targets may have been weak and more activity-based as opposed to outcome￾based. Table 12 shows key intermediate indicators of change across all four HEIs. TABLE 12: KEY INTERMEDIATE INDICATORS OF CHANGE Indicators HEIs UTECH VTDI KCC Knox Percentage of students completing the degree program 47.5% ~34.2% 100% 36.4% Percentage of students requiring funding to complete their program due to failing one or more courses 35% Not available 0% 10% (1 student) from the first cohort; total of 45.4% for both cohorts Percentage of students dropping out of the program 17% 25% 0% 10% Percentage increase in vulnerable students accessing jobs Not available Not available 100% of 1st cohort Not available Percentage of students moving on to a degree program following completion of Advance Not available Not available Not available 27% Percentage increase in career completion rates Not available Not available Not available Not available USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 48 Indicators HEIs UTECH VTDI KCC Knox Inflow of resources for curriculum development, scholarships, capacity￾building etc. Increased Increased (about 50% for scholarships) Increased Increased Number of areas of institutionalization and sustainability related to the five areas of study of the evaluation19 13 11 18 14 Extent to which HEIs changed internal policies related to administration of scholarships Partial Partial No change No change Extent to which marketing for recruitment and selection changed Adopted Advance approaches Adopted Advance approaches Adopted Advance approaches Adopted Advance approaches Regarding students transitioning to industry or starting their own businesses, all the HEIs demonstrated challenges in evaluating this due to the lack of any established monitoring system. Also, the impact of the COVID-19 pandemic was deemed an inhibiting factor to immediate transition into industry, particularly for those entering industries that were significantly affected by the pandemic, such as entertainment and health tourism. Across the five areas Advance targeted, there was clear evidence of the institutionalization and sustainability of various components of the Advance Program and clear indicators of change. Knockalva had the highest number of institutional changes: 18. This reflects changes in one or more of the targeted areas of the Advance Program. The institution with the least number of changes was VTDI, with 11, partly reflecting processes already in place that were deemed suitable and did not require change. Indicators of Change for Students Overall, students held the view that they had good or better chances to be employed when they graduated. Some indicated that their chances of employment would be even better than students from non-Advance programs. They expressed the following opinions in interviews and focus group discussions: “The areas of focus by Advance targeted fields such as cosmetics and agro-processing, that are unique and provided job options upon completion. The areas lack trained persons, which provide opportunities for persons completing the Advance Program.” (Knox student) 19 Areas of institutionalization are described in Annex 1. It includes at least an emergent level of institutionalization. 49 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV “Received hands-on experience, exposed to awesome knowledge about agro-processing.” (Knox student) “Two years ago, I would have definitely said that my chances to be employed having benefitted from Advance would be good or better. COVID-19, however, has greatly impacted the entertainment industry and it is challenging to find jobs whether one graduated from the Advance Program or not. The market is quite tight now in finding jobs in the field, and as such I am exploring the entrepreneurial aspect, having studied on the Advance Program.” (VTDI student) “What makes you employable is the ability to market yourself. However, the Advance Program provided some tools and skills that would provide a good/better chance at being employed.” (UTECH student) The performance of Advance scholarship recipients relative to other students varied across HEIs. In the case of UTECH, education leaders as well as faculty and staff indicated that students demonstrated a lack of soft skills, including communication skills, and were not adequately prepared for the responsibilities associated with higher education learning. These students were deemed to be not of the same quality as other students due mainly to their level of education and socioeconomic backgrounds. The experience of VTDI shows that many students lacked the motivation to fully engage in the programs. “The Advance students were not as motivated as [the] regular cohort – who had to pay. That was part of the problem. They were not as persistent. There was a difference – greater support for that batch of students was needed to help them recognize the importance of completing a program successfully.” (VTDI Administrator) Annexes 1 and 2 illustrate the sustainability and institutionalization of key elements of Advance for each of the HEIs, or alternatively, illustrates the extent of institutionalization and sustainability across the five key areas: 1) curriculum and pedagogy; 2) faculty/staff professional development; 3) labor-bridging services; 4) admissions and recruitment; and 5) scholarships. USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 50 VI. DISCUSSION AND CONCLUSIONS This report shows that the Advance Program provided new and innovative approaches to higher education that benefitted the HEIs with which the program worked, and also for several disadvantaged youth, with some benefits of Advance already extending to the wider community. The program has presented Jamaica with a workable and scalable solution for enabling disadvantaged and unattached youth to access higher education and “…facilitate them to become productive, better citizens” (VTDI Official). As per Jamaica’s Prime Minister, Andrew Holness, “Many unattached youths come from inner-city communities. They are talented young people, but many have not had the benefit of a well-structured environment long enough to develop the character, positive attitudes, and the necessary skills to successfully negotiate the challenges of life. The vast majority of unattached youth dropped out of school. They did not gain an education sufficient for their age and stage in life and many of them, unfortunately, will end up in conflict with the law.” Unattached youth,20 also referred to as NEET (Neither in Education, Employment or Training), represent one of Jamaica’s seven vulnerable child groups in the country. Youth (persons aged 15-24 years) make up about 20 percent of Jamaica’s total population. Poverty has contributed to the increase in unattached youth, those between 15 and 24 years of age who are either not in school, unemployed, or not participating or engaged in any training program. About one-quarter of unattached youth only attain an educational level up to Grade 9. 21 School dropouts, poor educational outcomes, and the lack of adequate skills training have contributed to the increase in unattached youth. In 2020, an estimated 22.4 percent of youth were neither in employment, education, or training, representing an untapped proportion of the labor force, who are not engaged in any productive activity. 22 The thrust of the Advance Program, if scaled up, could provide several opportunities to Jamaica for reducing the number of disadvantaged and unattached youth. This is consistent with the overall development objectives of Jamaica. The Advance Program, its best practices, and lessons learned can emerge as a good demonstration of approaches to engaging unattached and disadvantaged youth in higher education more broadly in Jamaica and the Caribbean, and potentially elsewhere around the world. Despite some issues and challenges indicated by the HEIs, the Advance Program has brought to Jamaica new and novel ways to engage disadvantaged and unattached youth in higher education. The program and its associated outcomes align well with several initiatives of the Government of Jamaica that are seeking to address the complex issues faced by unattached and disadvantaged youth in Jamaica. The Advance Program already responds to one of the key national strategies for education and training contained in the country’s national development plan, “Vision 2030 Jamaica,” 23 which calls for the expansion of mechanisms to provide access to education and training for all, including unattached youth. The courses developed under Advance are well aligned with the Jamaica Labor Market Assessment, led by FHI 360. This report is also aligned to the priorities of the Government of Jamaica as contained in 20 Unattached youth is a term used in Jamaica and by the Government of Jamaica to describe youth that are neither in education, employment, or training. 21 Children First, 2018 22 Jamaica Survey of Living Conditions. (2018). The Planning Institute of Jamaica and Statistical Institute of Jamaica. 23 Vision 2030 Jamaica is the country’s first long-term national development plan. It was promulgated in 2009 and has been designed to put Jamaica in a position to achieve developed country status by 2030. 51 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV Vision 2030 Jamaica – National Development Plan. The labor market assessment was therefore timely, as new areas of focus at the national level often need to be accompanied by training and capacity building to ensure that there is sufficient labor to fulfill the stated goals and objectives. For example, the government has as strategic priorities the development of the “orange economy,” with focus on the culture, cultural, and creative industries. 24 Current focus is also on value added in agriculture—to boost agricultural productivity and export-led agriculture growth as well as increasing youth engagement in agriculture. The government is also positioning itself to establish health and wellness tourism as it seeks to diversify its tourism industry and create niche markets. The associate degree programs Advance developed and the program’s structure that includes theoretical underpinnings as well as practical applications will allow those graduating to be well equipped to enter these new and emerging industries. The rationale for scaling up the Advance Program is clear given its alignment with Jamaica’s labor market needs as well as the alignment with other national priorities, including those related to education, reduction in crime and violence, and strengthening the productivity and international competitiveness of the agricultural sector. During the stakeholder engagements—FGDs and KIIs—participants expressed several positive comments about Advance, along with a few disappointments. But what is clear is that the benefits of Advance significantly outweighed any issues and challenges HEIs experienced. The level of institutionalization and sustainability is testimony to the value that the HEIs have placed on the program. It also was clear to the ET that the program was designed around the principle of continuous improvement, which led to the changes being implemented along the way to reduce issues and challenges that emerged. For example, this was evident with respect to the scholarship component of the program, which was amended to allow students to be able to access their stipends more easily, reducing the wait time to receiving these benefits. The changes to the HEIs’ systems due to Advance and the adoption of best practices at the institutional level will allow for the development of a register of best practices and lessons learned that could be shared at the national level and with other institutions such as The University of the West Indies (UWI). The UWI offers several programs related to the orange economy, health tourism, and within its geography programs, there also is focus on areas such as sustainable agriculture. It would be useful to assess how Advance graduates could matriculate to The UWI—a regional university ranked by the Times Higher Education Rankings25 as among the top 600 universities in the world, the top 40 in Latin America and the Caribbean, and the number one university in the Caribbean. Our study’s analysis showed that some graduates had an interest in moving into bachelor’s degree programs, and in future years The UWI could be an option for them. To provide an indication of the overall impact of Advance based on the data collected in the evaluation, the ET adopted a simple approach of assessing under each aspect of the ToC the number of activities implemented by each institution and the results achieved, and then summed these across the four HEIs 24 Abbion Robinson, “The Orange Economy — the next Frontier for the Caribbean,” Jamaica Observer, February 2, 2022, https://www.jamaicaobserver.com/business/the-orange-economy-the-next-frontier-for-the-caribbean/. Accessed on 5/08/2022. 25 Times Higher Education Rankings, accessed online on 5/08/2022 at https://www.timeshighereducation.com/world-university￾rankings/university-west-indies. USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 52 investigated. The ET then ascribed a rank of between one and five (with five being the most favorable outcome) to Advance’s Theory of Change, as follows: • If we reach back to upper secondary-aged youth in Jamaica to help them understand their post￾secondary choices – this was accomplished through the new and innovative marketing approaches that the HEIs were exposed to as a result of Advance. These approaches led to HEIs, some for the first time, interacting and engaging with disadvantaged youth and being able to enroll them in Advance Programs. These approaches are likely to be used by all HEIs in the future. Rank 4 • And reach out to technical training institutions to build their capacity in specific programs and services – this approach was highly successful and has allowed HEIs to expand their curricula to offer new degree programs that are relevant to needs of the labor market, and also to build the capacity of staff in a range of areas—including experiential learning—that have started to be further scaled up in the HEIs themselves, with examples of communities of practice being developed to share learning approaches among HEIs. Within HEIs, there also has been evidence of approaches of Advance being applied within other degree programs that were not part of Advance. Rank 5 • And reach into employers to better understand their workforce needs and how these needs align with technical education programs – whilst this was a core part of Advance, HEIs bemoaned the fact that it was the least elaborated component of the program. Despite this, all HEIs indicated that they learned a significant amount about labor-bridging and the importance of adopting labor￾bridging services as part of their core mandate. Rank 3 • Then more disadvantaged youth will obtain better or new employment. While the information was not always available to determine whether youth that were part of Advance were employed, there were several examples of students benefitting from employment opportunities immediately after completion of their degree. There was also anecdotal evidence that suggested that the students with degrees that received training in entrepreneurship established their own businesses to support the identified industries. Rank 4 VII. RECOMMENDATIONS FOR FUTURE PROGRAMMING The following provides recommendations that could be useful for future programming in higher education, building on the lessons learned and best practices of the Advance Program in Jamaica. Some of these recommendations could be useful for other countries in which USAID Missions may be considering investing in programs that support disadvantaged youth. Recommendations one to four are specific to the GOJ and would allow USAID to engage in strategic discussions with the GOJ in scaling up the results of Advance and adopting and adapting to reduce the proportion of unattached youth, and by extension reducing some of the negative impacts of unattached youth. Recommendations five to seven are not only useful for Jamaica but could become a standard package for all USAID-supported programs of this nature that will set the groundwork for both a community of practice and clearing house mechanism. 1. Develop a policy brief on Advance’s results, successes, and benefits that could be shared with the GOJ to enable further cross-cutting engagement and strategizing at the national level with 53 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV the Ministries of Education, Agriculture, Industry, National Security, Social Protection, and Industry regarding the possibilities of scaling up components of an Advance model to support the achievement of the country’s medium to long-term national priorities, given the close alignment of the program with the country’s national priorities. Supporting disadvantaged youth through targeted higher education programs is emerging as a win-win for countries like Jamaica, as it supports the provision of skilled labor to new and emerging industries while at the same time reducing the probability of disadvantaged youth engaging in activities such as crime and violence. 2. Within the policy brief mentioned above, there should be information related to aligning the outcomes of Advance and its approaches with the other national priorities of government that may be outside of agriculture and the orange economy, but perhaps have a focus on areas in the blue and green economies26 or in the business outsource processing sector for which there is need for skilled labor. 3. The Advance Program as constructed provides an excellent conduit for supporting the recently launched 6th Form Pathways, which is part of the Ministry of Education’s implementation of a seven-year high school-level program. It allows students who complete grade 11 (the equivalent of grade 10 in the United States) to enroll in the program and pursue a two-year course of study with alternative opportunities alongside the traditional sixth-form curriculum. Some students in the Pathways Program could directly feed into HEIs that focus on Advance courses. 4. Given Advance’s positive results engaging vulnerable youth, and the methods and approaches implemented by the program to ensure that they continue their education, it is recommended to support the GOJ, and particularly the Ministry of Education, in reviewing and assessing different modalities for financing education programs for disadvantaged and vulnerable youth, including the alignment of the PATH program with Advance to enable HEIs to continue to offer the program to disadvantaged youth. PATH could serve as a resource to continue providing access to scholarships for disadvantaged youth. 5. Prepare an Advance lessons-learned and best practices register with information related to applicability and adopting and adapting to other countries. 6. Prepare a set of human-interest stories, blogs, and short videos for social media platforms focusing on a few Advance students that excelled. These could be used to motivate the many other disadvantaged youth on the island to consider higher education as an option. Such media content can be placed on government websites that are designed for youth and can also be shared through the secondary school system. Consider establishing Advance youth ambassadors who would serve to motivate other disadvantaged youth, enabling them to understand the benefits of higher education. 26 In a green economy, growth in employment and income are driven by investments that reduce carbon emissions and pollution, enhance energy, and resource efficiency, and prevent the loss of biodiversity and ecosystem services. The blue economy focuses on the sustainable use of ocean resources for economic growth, improved livelihoods, and jobs while preserving the health of ocean ecosystems. USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 54 7. Package the workshop content related to entrepreneurship, undertaking interviews, resume￾writing, and other soft skills, and share them with the Ministry of Education for inclusion in the personal development curriculum at the high school level (particularly at 4th and 5th forms/U.S. grades 10 and 11). Approaches to Scaling up and Replicability, Building on Best Practices, and Lessons Learned Consider scaling up and replicating the Advance model, building on best practices and lessons learned, scaling up in Jamaica, but consider replicating the model elsewhere, placing focus on: • Creating stronger linkages between high schools and tertiary-level institutions to better enable high school students to be motivated to attend tertiary level institutions and to be better prepared. This could also include working with HEIs to develop pre-university programs or summer programs to help youth transition into universities. • Understanding fully the disadvantaged youth in each country—be their challenges poverty, domestic or urban violence, racism, linguistic or ethnic minorities, etc.—to be able to anticipate issues and challenges early and include these in project design. • Ensuring that courses developed benefit from labor market assessments. This can be seen as a best practice since the courses Advance developed were selected based on a labor market needs assessment, which are further aligned to the industries being prioritized at the national level. • Strengthening the capacity of HEIs to better understand their roles in SEL and to better understand SEL approaches and how to apply them. • Packaging of HEI study programs in various formats to meet the diverse learning needs of disadvantaged groups. The experience of Advance Jamaica shows that some students withdrew from their studies because of their inability to adjust to the requirements of higher education. • Considering the repackaging of degree programs into certificate modules to allow students to better engage in self-pacing their studies. Also, where a student may pass several modules, that student can receive certificates and have credits to be able to pick up the course of study at a later date. All HEIs except Knockalva indicated that in many cases students who have been unable to graduate need to resit an exam and cannot afford to pay to do so. Focus also could be placed on designing some courses for online delivery and developing blending learning approaches. Also, as part of the HEIs contribution, consider waiving the fees for resitting exams for these students. • Monitoring and evaluation. A very early establishment of indicators and targets that HEIs monitor and report on are critical to assessing impact. • Establishing accountability mechanisms that extend between the program and the HEIs, the program and the students, and the students and the HEIs. Establishment of scholarship 55 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV agreements with students would be essential, and through the reporting requirements, could help the HEIs track these students’ progress upon graduating. • Labor-bridging services, the various aspects and modalities that could be employed, and the roles and responsibilities of HEIs, including the benefits to be derived. Focus also could be placed on how labor-bridging services can be aligned to the overall mandates of universities. While HEIs were exposed to several aspects of labor bridging services, it is a component that they have indicated that more emphasis should have been placed on due to its importance. • Addressing issues and challenges with programming before they emerge—such as the issue of resits—especially where students fail only one to two courses. What would be the modality to support these students? Could the HEI consider providing the resit at no cost; would Advance consider paying for the resit; would the HEI allow the resit but repayment late through a sort of payment plan? The benefits of the student graduating far outweigh the costs. • The importance of professional development of faculty and staff in areas such as experiential learning—never underestimating the fact that this teaching approach is not widely known or utilized by some HEIs. • Using experts and industry players to support curriculum development in HEIs. This was not a common approach in Jamaica but was one that was very much appreciated since it also provided faculty several areas of new information and knowledge that enhanced their own teaching in the classroom. • Establishing communities of practice among lecturers and HEIs to share approaches to learning and teaching of disadvantaged youth as well as new curricula areas. USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 56 ANNEXES ANNEX 1. DETAILED PROGRESS MARKERS BY HEI AND KEY AREAS OF SUSTAINABILITY AND INSTITUTIONALIZATION The table below assesses the extent of sustainability and institutionalization of five fields of study within each institution using the following criteria: Advanced – Strong application and institutionalization within the HEI Established – Evidence of application at the individual level Emergent Level - Early planning phase to include changes Latent Level - No concrete changes, no plans in place Curriculum and Pedagogy Professional Development of Faculty and Staff Labor Bridging Services Admissions and Recruitment Access to Scholarships UTECH ● Expansion of degree offerings to include associate degrees ● Exposure and application of experiential learning across courses within Advance and outside of Advance ● Advance courses have been developed for online delivery ● Will admit students with lower academic levels and support them to adjust to university based on the approaches such as SEL that they have adopted for the university level students ● Approached the Ministry of Education, Youth, and Information for scholarships for disadvantaged youth for Advance courses and have received 10 half scholarships per annum to apply to the ASc. degrees Advance developed ● Made accommodations for student to access regular UTECH scholarships for ASc. degrees Advance developed ● Two new degree programs with associated curricula/courses ● Employment of social-emotional learning across campus ● Capacity and skills of staff, have been strengthened in curriculum development and their approaches to teaching now incorporate experiential ● Begun to utilize the marketing and recruitment approaches of Advance to target disadvantaged youth 57 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV Curriculum and Pedagogy Professional Development of Faculty and Staff Labor Bridging Services Admissions and Recruitment Access to Scholarships learning in all of their courses, many of which are outside of the Advance ● Training opportunities provided for staff have been passed on to students via course delivery ● Consideration by faculty on drawing on a myriad sources in the design and development of new programs ● Exposure to coupling academic learning with professional development has excited faculty ● Lecturers who interfaced with Advance are providing mentorship to other lecturers as it relates to curriculum design, labor bridging, etc. ● Wil continue to engage with the private sector to access internships for students and to extend experiential learning ● Will integrate ideas and concepts of labor-bridging services from Advance in their other regular programs ● Interest in adopting those participatory approaches to course development Advance introduced has been indicated as well as engaging industry experts in curricula development. Not included yet in strategic plans or other policy documents ● Started the process of drafting MOUs with industry but has not continued the process, although it is considered useful VTDI ● Review of two ASc. programs with associated curricula/courses, resulting in improvement in curricula ● Curriculum more consistent with the needs of the labor market ● Advance courses have been developed for online delivery ● Establishment of student advisement center and institutionalized the role of student advisors to better support learning and learning outcomes USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 58 Curriculum and Pedagogy Professional Development of Faculty and Staff Labor Bridging Services Admissions and Recruitment Access to Scholarships ● Have adopted new teaching and learning strategies, assessment strategies, including experiential ● Employment of social-emotional learning across campus ● Capacity and skills of staff strengthened in curriculum development and design ● Training opportunities provided for staff have been passed on to students via course delivery ● Delivering workshops on soft skills based on handbook Advance developed ● Has begun to utilize the marketing and recruitment approaches of Advance to target disadvantaged youth ● Exposure to coupling academic learning with professional development has excited faculty ● Will continue to engage with the private sector to access internships for students and to extend experiential learning ● Will integrate ideas and concepts of labor-bridging services from Advance in their other regular programs ● Use of existing scholarship programs and other modes of financing such as earn-as￾you-learn to attract disadvantaged youth ● Approached the private sector to support the provision of scholarships to students that are part of Advance-related courses • Interest in establishing MOU with industry players Knox ● Advance courses have been developed for online delivery ● Incorporated and applied practical elements and experiential learning to the BSc. degree in Environmental Studies (program not part of Advance) ● Linked one of its degree programs in environmental science to the Advance￾developed program in agri￾business, creating the opportunity to provide specializations in agri-business ● Created communities of practice to share information and best practices in course delivery ● Knox has been able to gain access to scholarships funded by the Ministry of Labor and Social Security under the PATH (Program for students who maintain a GPA of 2.5 or above) 59 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV Curriculum and Pedagogy Professional Development of Faculty and Staff Labor Bridging Services Admissions and Recruitment Access to Scholarships within the environmental science degree (BSc.) ● Faculty have started to adapt experiential learning across courses outside of Advance by allocating hours for teaching and hours for practicals and experiential learning ● One new ASc. degree programs with associated curricula/courses ● Employment of social-emotional learning across campus ● Training opportunities provided for staff have been passed on to students via course delivery ● Continues to foster the relationships with different entities for study tours and have established relationships with the Scientific Research Council to support the agro￾processing program in terms of product testing, etc. ● Delivering workshops on soft skills based on handbook Advance developed ● Has begun to utilize the marketing and recruitment approaches of Advance to target disadvantaged youth ● Exposure to coupling academic learning with professional development has excited faculty ● Will continue to engage with the private sector to access internships for students and to extend experiential learning ● Will integrate ideas and concepts of labor-bridging services from Advance in their other regular programs ● Approached the private sector to support the provision of scholarships to students that are part of Advance-related courses ● Knox is considering expanding the ASc. in agri-business to a full degree program USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 60 Curriculum and Pedagogy Professional Development of Faculty and Staff Labor Bridging Services Admissions and Recruitment Access to Scholarships Knockalva ● Approached the UCJ for accreditation of the ASc. degree ● Undertakes needs assessments in the review of existing curricula to ensure that curriculum also is aligned to the needs of the labor market ● Advance courses have been developed for online delivery ● Facilitated program integration of the Advance ASc. in agro￾processing with the ASc. in agriculture production, enabling both sets of students to appreciate the linkages between the two programs related to agriculture production and enterprise development ● Knocklava will use its agro￾processing lab to support training in the wider community of Hannover in agro-processing ● One new ASc. degree program with associated curricula/courses ● Employment of social-emotional learning across campus ● Agro-processing lab Advance provided to Knockalva has resulted in interest from the wider community for training in agro-processing ● Agro-procesing lab only one of its kind on the eastern side of the island, and is made available as a resource to the wider community ● Training opportunities provided for staff have been passed on to students via course delivery ● Knockalva continues to employ new training strategies Advance introduced across other courses and share them with other faculty ● Delivering workshops on softer skills based on handbook Advance developed ● Has begun to utilize the Advance marketing and recruitment approaches to target disadvantaged youth ● Development of a new scholarship program similar to Advance with support from alumni ● Will access scholarships from the Hanover Charity and Rio Tinto Charity and the College’s alumni as well as Hi Pro (private sector company) to support vulnerable youth 61 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV Curriculum and Pedagogy Professional Development of Faculty and Staff Labor Bridging Services Admissions and Recruitment Access to Scholarships ● Knockalva will develop a certificate in food processing – a six-month program for adult learners ● Exposure to coupling academic learning with professional development has excited faculty ● Will continue to engage with the private sector to access internships for students and to extend experiential learning ● Will integrate ideas and concepts of labor-bridging services from Advance in their other regular programs USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 62 ANNEX 2. INDIVIDUAL OUTCOME MAP TABLES BY HEI UTECH Inputs, Activities and Outputs Outcomes: Changes in Behavior Impacts: Changes in State The Advance Program What was gained by UTECH The difference Advance has made to UTECH Curriculum development ● Two new degree programs developed • Being able to offer associate degrees in addition to other degree programs offered • Introduced new ways of approaching course development, such as engaging experts Professional development • Introduction to experiential learning • Exposure to training in SEL • Promotion of mentorship among lecturers • Application of experiential learning and social-emotional learning to non-Advance courses • Lecturers who interfaced with Advance are encouraged to provide mentorship to other lecturers as it relates to the areas covered by Advance Resources ● 40 scholarships ● Actively engaging external sources to support scholarship program. ● The Ministry of Education providing at least ten half￾scholarships per annum to the University New ways of working and engaging (private sector, disadvantaged youth) ● Will begin admitting students with lower academic levels and provide support in adjusting to university based on the approaches Advance introduced such as SEL that they have adopted for the university￾level students 63 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV VTDI Inputs Activities and Outputs Outcomes: Changes in Behavior Impacts: Changes in State The Advance Program What was gained by VTDI The difference Advance has made to VTDI Curriculum development • Review and revision of two associate degree programs • The associate degree programs are now more consistent with the needs of the labor market Professional development • Introduction to experiential learning • Exposure to training in SEL • Application of experiential learning and SEL to non-Advance courses • The capacity of faculty and staff has been strengthened to deliver the associate degree courses supported by Advance • Changes in teaching and learning strategies, assessment strategies, and types Resources • 35 scholarships (2018- 2021) • 15 (2022) • Increased access to targeted program of study for disadvantaged individuals New ways of working and engaging (private sector, disadvantaged youth) • Use of student advisors to support students pursuing degree programs • Introduction of student advisors USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 64 KNOX COMMUNITY COLLEGE Inputs Activities and Outputs Outcomes: Changes in Behavior Impacts: Changes in State The Advance Program What was gained by Knox Community College The difference Advance has made to Knox Community College Curriculum development • New associate degree program • Associate degree programs that are consistent with the needs of the labor market Professional development • Introduction to experiential learning • Exposure to training in SEL • Incorporation of experiential learning into other degree programs, such as the BSc. in Environmental Studies • The capacity of faculty and staff has been strengthened to deliver the associate degree courses Advance supported Resources • 22 scholarships • Increased access for disadvantaged individuals to targeted program of study • Actively engaging external sources such as the alumni to support financial aid through scholarships New ways of working and engaging (private sector, disadvantaged youth) • Lecturers and students engaged with a range of local private sector firms in food processing • Strengthening of partnerships with the private sector 65 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV KNOCKALVA POLYTECHNIC Inputs Activities and Outputs Outcomes: Changes in Behavior Impacts: Changes in State The Advance Program What was gained by Knockalva Polytechnic The difference Advance has made to Knockalva Polytechnic Curriculum development • New degree program is offered • Advance influenced internal procedures in areas such as curriculum review, standardizing practical sessions, and theory, for courses outside of the Advance Program ● Associate degree programs that are consistent with the needs of the labor market Professional development ● Introduction to experiential learning ● Exposure to training in SEL ● The capacity of faculty and staff has been strengthened to deliver the associate degree courses supported by Advance ● Started undertaking training needs assessments in reviewing existing curricula and development of new curricula to reflect the needs of the labor market, to enable its course offerings to be more relevant Resources ● Agro-processing lab Advance provided to Knockalva ● Infrastructure adjustments: Retrofitting of the computer laboratory and the establishment of an agro-processing laboratory ● 12 scholarships ● Will use its agro-processing lab to support training in the wider community of Hanover in agro￾processing ● Developed a scholarship program for disadvantaged youth, like Advance. Actively engaging external sources such as the alumni to support financial aid through scholarships ● Increased access to targeted program of study for disadvantaged individuals New ways of working and engaging (private sector, disadvantaged youth) ● Lecturers and students engaged with a range of local private sector firms in food processing ● Strengthening of partnerships with the private sector ● Will begin to access scholarships from new sources to support vulnerable youth through the USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 66 Inputs Activities and Outputs Outcomes: Changes in Behavior Impacts: Changes in State The Advance Program What was gained by Knockalva Polytechnic The difference Advance has made to Knockalva Polytechnic Hanover Charity and Rio Tinto Charity and the College’s alumni, as well as Hi Pro (private sector company) 67 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV ANNEX 3. DESCRIPTION OF PARTICIPATING HIGHER EDUCATION INSTITUTIONS Name Description University of Technology, Jamaica (UTECH) UTECH, Jamaica, formerly the College of Arts, Science and Technology, is located within the Greater Kingston Metropolitan Region in the parish of St. Andrew. The University offers more than 100 programs at certificate, diploma, and degree levels. Its academic offerings and overall development were modeled on the English Polytechnic. The overall emphasis on its teaching and learning is on flexibility of approach, work￾based learning, and professional linkages. Several of its programs are franchised through linkages with community colleges that are under the Council of Community Colleges of Jamaica. Vocational Training and Development Institute (VTDI) VTDI was established in 1970 with the assistance of the International Labor Organization, the United Nations Development Program, and the Government of Jamaica. The institute was established to address the following aims: to train instructors to deliver vocational education in the private and public sectors; to retrain and upgrade the skills of workers for employment and production; to prepare, develop, and document industrial vocational training syllabuses, standards, manuals, and literature for the use of the public and private sectors; and to train and certify students and workers at all levels and grades. It was established in response to the growth in the bauxite industry in Jamaica when workers were needed in this industry, and they lacked the necessary training. Since 1990, VTDI has been managed by the Human Employment and Resource Training National Training Agency and is described as the tertiary arm of the agency. The agency was established in 1982 and is funded by 3 percent of the employer’s wage bill in the country. Today, the Institute has expanded its original mandate of solely providing a diploma in technical and vocational education and training, by offering programs up to the postgraduate diploma level. The Competency Based Education and Training approach is employed when designing, delivering, and assessing programs. It is central to the teaching/learning experience provided at the institution. Council of Community Colleges of Jamaica (CCCJ) The CCCJ, through its regulatory function, seeks to determine and implement common standards in the community colleges to ensure the integrity of the courses from one campus to the next, and also address the need for articulation with other educational institutions—both locally and abroad. The CCCJ prescribes the conditions under which individuals may be admitted as students of community colleges; considers, recommends, and approves curricula to be used in the affiliate colleges; determines conditions for the holding of examinations; and examines and assesses the work of students in the colleges. The CCCJs also advise the Minister of Education on policy and education matters relating to community colleges. Knox Community College (KCC) Knox Community College is a co-educational institution located in the rural parish of Clarendon in an area known as Spaulding. The college has been in operation for more than 45 years. The college operates four campuses and has more than 50 programs in a variety of disciplines, which culminate in the award of certificates, diplomas, associate, and bachelor’s degrees. The College started with three academic programs: Pre￾university Arts and Sciences, Secretarial Studies, and Farm Management. Knockalva Polytechnic College (KPC) The Knockalva Agricultural School was upgraded by the Ministry of Education as a polytechnic college in 2018, thereby providing greater access to tertiary education for persons in western Jamaica. This upgrading resulted in the transition from a secondary, vocational setting to a tertiary-level institution. The college offers opportunities for USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 68 Name Description students predominantly from the parishes of Hanover, Westmoreland, and St. James. The college offers ten programs of study, from certificates to associate degrees. 69 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV ANNEX 4. CONSENT FORM TO PARTICIPATE IN KIIS/FOCUS GROUP DISCUSSIONS Jamaica Advance Evaluation My name is Elizabeth Emanuel. I am here to learn about your experience with the Advance project. We are an independent team from USAID and the Advance project, and we are investigating the effects of the Project’s implementation to understand strategies that have been more successful and those that have presented challenges for Higher Education Institutions (HEIs) to achieve sustainability and institutionalization in key areas such as (1) curriculum and pedagogy, (2) professional development of faculty and staff, (3) labor-bridging services, (4) admissions and recruitment, and (5) access to scholarships. For this reason, we are talking to people who have participated in Advance activities. This document will confirm your authorization to record the session and take some photographs. There are no right or wrong answers, so please feel free to share your experiences. Your real name will not be used in the evaluation report and any identifying information will be removed. Instead of trying to write down everything you share, I would like to record the conversation so that I can listen to it later. It is only for us; no one else will listen to it, and no name will be associated with the conversation. This interview should take between 45 minutes and one hour. ****************************************** Write name if participant agrees to participate and to record the interview / focus group Date: ___________________ ______________________________________ _______________ Participant Name Agrees to record the KII/FGD ______________________________________ _______________ Participant Name Agrees to record the KII/FGD ______________________________________ _______________ Participant Name Agrees to record the KII/FGD ______________________________________ _______________ Participant Name Agrees to record the KII/FGD ______________________________________ _______________ Participant Name Agrees to record the KII/FGD USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 70 ANNEX 5. DATA COLLECTION INSTRUMENTS FOCUS GROUP DISCUSSION GUIDES USAID ADVANCE PERFORMANCE EVALUATION We are an independent team from USAID and the Advance project, and we are investigating the effects of the project’s implementation to understand strategies that have been more successful and those that have presented challenges for Higher Education Institutions (HEIs) to achieve sustainability and institutionalization in key areas such as (1) curriculum and pedagogy, (2) professional development of faculty and staff, (3) labor-bridging services, (4) admissions and recruitment, and (5) access to scholarships. For this reason, we are talking to people who have participated in Advance activities. We are requesting you fill out this survey to students that benefited from the Advance Program. Thanks for your participation. Education Leaders and Faculty Research Question Interview Questions RQ1. Change in systems and improvements in institutional performance Did the Advance Program have any impact on the areas of curriculum development? Which curricula did Advance impact? Would you like to share with us some of those changes? Did Advance support new curriculum development? Did the Advance Program have any impact on professional development of faculty staff? If yes, explain. How does the Advance Program compare to other similar programs that you have been part of (ask for the names of these programs? Did the Advance Program influence in any way your admissions process? Please share. Did the Advance Program enhance labor market bridging services to help students find gainful employment after graduating? If yes, share with us how. How many scholarships did students at your institution receive? Do students have any other access to scholarships for the same programs that Advance targeted? Can you provide any information on the performance of students with scholarships? How do students with scholarships compare to those that did not have scholarships? How did access to scholarships under Advance affect admissions and numbers of students graduating? Did the access to the Advance scholarships increase retention rate of students and the percentage of students graduating this past year? Do you think that Advance influenced internal procedures in your HEI? If yes, please provide examples. (May be related to curricula development, infusion of best practices or lessons learned emerging from Advance, among others). 71 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV Research Question Interview Questions What is the main evidence of change at the institutional level due to Advance? How has Advance contributed to strengthening the curriculum development at your organization? In what ways did Advance contribute to enhancement in the knowledge and skills of staff (or your professional development) as it relates to curriculum development, pedagogy? Are there tangible changes/new processes in how your institution assess and provide linkages between students, curriculum, and market needs? How has the project contributed to improved policies, procedures and services geared to your students? Please provide some examples. On a scale of 1 to 10 (10 being best), how well did the Advance Program contribute to improvement in labor-bridging services, admissions and recruitment, and access to scholarships? Please explain your response. Are there tangible changes in roles, results, relationships rules, and/or resources within each one of the areas mentioned above? RQ2. Sustainability and institutionalization of Advance Based on the changes identified in the questions above, are there any that have been institutionalized? (design and curriculum development, professional development of faculty staff, labor-bridging services, admissions and recruitment, and access to scholarships) To what degree have the changes been institutionalized within your institution? If they have not been institutionalized, are there any considerations? (Maybe probe if the concept of institutionalization is fully understood or what it means to them in this context.) If yes to above, what would you say are the driving forces or factors that encouraged or facilitated the institutionalization process? When we speak about institutionalization, what does it look like? Please describe. Are there any other areas that you believe should be institutionalized for the benefit of the institution? Why have they not been institutionalized? Please share what you think will be needed to continue producing the targeted results in the five prioritized areas – resources, additional support, support from the CCCJ, or the University senate – to institutionalize those areas that you identified? Will resources continue to be assessed after Advance concludes? Do you anticipate that the government will provide additional resources? USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 72 Research Question Interview Questions How is the institution approaching continued financial aid to qualified students after the Advance Program ends? If there has been institutionalization – Ask - Do you think the changes that you made within your institution are sustainable? Why? What mechanisms are in place in the system to ensure that changes in design and curriculum development, professional development of faculty staff, labor-bridging services, admissions and recruitment, and access to scholarships will continue after Advance ends? Share with us if these changes are reflected in internal institutional procedures and policy. Do teachers anticipate continuing/expanding on the changes they have made in their practice going forward? Are you willing to share successes, best practices with other HEIs? If yes, how do you propose these communications would be handled? What are your suggestions for sharing these best practices or lessons learned? And with respect to best practices and lessons learned, do you have any that you would like to share with us? RQ3. Enabling and inhibiting factors Let us reflect a bit more on the five prioritized areas: ● curriculum design and pedagogy ● professional development opportunities ● labor market bridging services ● recruitment and admissions processes ● local scholarships What factors enabled the institution to benefit from the capacity building approaches in the five areas and in turn positively affected the performance of the institution? What factors inhibited success of the interventions of Advance around the five prioritized areas? Let us reflect on the approaches used to build capacity or advance the five prioritized areas. Were the approaches Advance used adequately contextualized for Jamaica? Share with us some of these approaches and the ones you liked best. Also share with use the converse. 73 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV Research Question Interview Questions Share with us any ideas that you may have for improving the approaches or specific approaches per prioritized area Advance used that may be adopted or adapted for future projects in Jamaica or elsewhere. COVID-19 and its associated impacts began in March 2020. How did these changes and disruptions affect Advance implementation from the perspective of your institution? How has the need for online teaching, reduced face-to-face interactions, connectivity, and other related factors influenced the application of the capacity￾building approach? What are the enabling factors? RQ4. Indicators of change Let us now put all that you have said in numbers or undertake a sort of quantitative analysis or measurement of early results. What has been: ● Percentage increase in vulnerable students accessing jobs ● Percentage increase in career completion rates ● Percentage reduction students failing to pass a class, related to Advance Programs ● Percentage increase in the inflow of resources for curriculum development, scholarships, etc. ● Number of updated and new policies and internal procedures related to Advance on: design and curriculum development, professional development of faculty staff, labor-bridging services, admissions and recruitment, and scholarships provision Since Advance, have you been able to document or assess the change (positive or negative) in the number of students in target academic programs that demonstrate market relevant skills? Has there been increased access for disadvantaged youth in target academic programs? What are your overall comments on Advance? USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 74 School Counsellors/Strategic Planners, etc. Research Question Interview Questions RQ1. Change in systems and improvements in institutional performance Did the Advance Program influence in any way your admissions process? Please share. What has improved? What is different? What are you most happy about? Did the Advance Program enhance labor market bridging services to help students find gainful employment after graduating? If yes, share with us how, taking into account your previous/current labor market bridging services. What areas of the advance bridging market services did you think were most useful to your institution? How many scholarships did students at your institution receive? Do students have any other access to scholarships for the same programs that Advance targeted? Can you provide any information on the performance of students with scholarships? How do students with scholarships compare to those that did not have scholarships? How were students targeted for scholarships? What was the process? How confident are you that the students that received scholarships were the most disadvantaged? How did access to scholarships under Advance affect admissions and numbers of students graduating? Did the access to the Advance scholarships increase retention rate of students and the percentage of students graduating this past year? What is the main evidence of change in admissions and access to scholarships since Advance? Are there tangible changes/new processes in how your institution assesses and provides linkages between students, curriculum, and market needs? Please provide us with some examples. How has the project contributed to improved policies, procedures, and services geared to your students? Please provide some examples specific to admissions and scholarship selection. On a scale of 1 to 10 (10 being best), how well did the Advance Program contribute to improvement in labor-bridging services and admissions? Please explain your response. 75 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV Research Question Interview Questions RQ2. Sustainability and institutionalization of Advance Based on the changes identified in the questions above, have any of the changes related to labor-bridging services, admissions and recruitment, and access to scholarships been institutionalized? To what degree have the changes been institutionalized within your institution? If they have not been institutionalized, are there any considerations for institutionalization? If yes to above, what would you say are the driving forces or factors that encouraged or facilitated the institutionalization process? How is the institution approaching continued financial aid to qualified students after the Advance Program ends? If there has been institutionalization – Ask - Do you think the changes that you made within your institution are sustainable? Why? What mechanisms are in place in the system to ensure that changes in labor-bridging services, admissions and recruitment, and access to scholarships will continue after Advance ends? Share with us if these changes are reflected in internal institutional procedures and policy. Are you willing to share successes or best practices with other counselors? If yes, how do you propose these communications be handled? What are your suggestions for sharing these best practices or lessons learned? And with respect to best practices and lessons learned, do you have any that you would like to share with us? RQ3. Enabling and inhibiting factors Let us reflect a bit more on the three of the five prioritized areas: 1. labor market bridging services 2. recruitment and admissions processes 3. local scholarships What factors enabled the institution to benefit from the capacity building approaches in the three areas and in turn positively affected the performance of the institution? What factors inhibited success of the interventions of Advance around the three prioritized areas? Let us reflect on the approaches used to build capacity or advance the three prioritized areas. USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 76 Research Question Interview Questions Were the approaches Advance used adequately contextualized for Jamaica? Share with us some of these approaches and the ones you liked best. Also share with us the converse. Share with us any ideas that you may have for improving the approaches or specific approaches per prioritized area Advance used that may be adopted or adapted for future projects in Jamaica or elsewhere. COVID-19 and its associated impacts began in March 2020. How did these changes and disruptions affect Advance implementation for the three prioritized areas as mentioned above from the perspective of your institution and your specific role? What are your overall comments on Advance? 77 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV KEY INFORMANT INTERVIEW GUIDE USAID ADVANCE PERFORMANCE EVALUATION We are an independent team from USAID and the Advance project, and we are investigating the effects of the project’s implementation to understand strategies that have been more successful and those that have presented challenges for Higher Education Institutions (HEIs) to achieve sustainability and institutionalization in key areas such as (1) curriculum and pedagogy, (2) professional development of faculty and staff, (3) labor-bridging services, (4) admissions and recruitment, and (5) access to scholarships. For this reason, we are talking to people who have participated in Advance activities. 1. Tell us a little bit about your role within [institution name] and with Advance. 2. Share with us your overall thoughts on the Advance Program. 3. How does the Advance Program compare with other programs of a similar nature of which you have been part? 4. How did Advance contribute to improvements in curriculum development and professional development at your institution? Share with us any tangible changes due to Advance at this institution – e.g., changes in pedagogy, curriculum design, etc. 5. Did Advance influence internal procedures in any way? Please consider the following areas: strategic planning, teacher professional development, curriculum design, overarching labor bridging services, access of vulnerable youth to higher education, admissions and recruitment, access to scholarships. 6. How did Advance scholarship recipients compare academically to other students undertaking the same courses? 7. Were scholarship recipients the most vulnerable/disadvantaged students? If yes, describe their vulnerabilities and indicate how the Advance Program was able to address or eliminate those. 8. Would you share the approaches used by Advance with other programs of study or faculties? 9. Are there any lessons learned or best practices that you would like to share? 10. Did you face any challenges with respect to the implementation of Advance (outside of COVID￾19)? 11. How did COVID-19 affect program activities or scholarship recipients? 12. On the issue of sustainability, are you adopting (or have you already adopted) any approaches to institutionalize aspects of the Advance Program? Share with us the areas/components (curriculum design and development/professional development/admissions process/labor￾bridging services/access to scholarships) that are likely to be institutionalized, and their current state. USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 78 13. Would you like to see Advance continue beyond the original project timeline? If yes, make reference to those components you would want to see continuing and those components that you might want to change. 14. Is Advance influencing in any way how students are approaching the job market? 15. Did the Advance Program help bridge or reduce the gap between needs of the labor market and training? If yes, how and to what extent? 16. Is there anything that you would like to share with us that we did not ask? 79 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV SURVEY QUESTIONNAIRES Students of Higher Education Institutions We are an independent team from USAID and the Advance project, and we are investigating the effects of the project’s implementation to understand strategies that have been more successful and those that have presented challenges for Higher Education Institutions (HEIs) to achieve sustainability and institutionalization in key areas such as (1) curriculum and pedagogy, (2) professional development of faculty and staff, (3) labor-bridging services, (4) admissions and recruitment, and (5) access to scholarships. For this reason, we are talking to people who have participated in Advance activities. We are requesting you fill out this survey to students that benefited from the Advance Program. Thanks for your participation. Please rate the following statements on a scale from one to five (five being the highest score or most favorable situation). ● I am pleased that I was selected for a scholarship under the Advance Program. Strongly Disagree ( ); Disagree ( ); Neutral ( ); Agree ( ); Strongly Agree ( ) ● I believe that the Advance Program is an excellent program. Strongly Disagree ( ); Disagree ( ); Neutral ( ); Agree ( ); Strongly Agree ( ) ● I knew the courses that I took that were Advance-supported courses. Strongly Disagree ( ); Disagree ( ); Neutral ( ); Agree ( ); Strongly Agree ( ) ● I prefer the approach to teaching on the Advance courses compared to other courses. Strongly Disagree ( ); Disagree ( ); Neutral ( ); Agree ( ); Strongly Agree ( ) ● I received special help and support from the Advance Program. Strongly Disagree ( ); Disagree ( ); Neutral ( ); Agree ( ); Strongly Agree ( ) ● COVID-19 affected my participation in school. Strongly Disagree ( ); Disagree ( ); Neutral ( ); Agree ( ); Strongly Agree ( ) ● I feel that the COVID-19 pandemic affected the way instructors worked with me. Strongly Disagree ( ); Disagree ( ); Neutral ( ); Agree ( ); Strongly Agree ( ) ● I think that the Advance Program prepared me for the job market. Strongly Disagree ( ); Disagree ( ); Neutral ( ); Agree ( ); Strongly Agree ( ) ● I think that there is need to expand the activities of Advance. Strongly Disagree ( ); Disagree ( ); Neutral ( ); Agree ( ); Strongly Agree ( ) USAID.GOV USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION | 80 Higher Education Institutions We are an independent team from USAID and the Advance Program, and we are investigating the effects of the program’s implementation to understand strategies that have been more successful and those that have presented challenges for Higher Education Institutions (HEIs) to achieve sustainability and institutionalization in key areas such as (1) curriculum and pedagogy, (2) professional development of faculty and staff, (3) labor-bridging services, (4) admissions and recruitment, and (5) access to scholarships. For this reason, we are talking to people who have participated in Advance activities. We are requesting you fill out this survey to students that benefited from the Advance Program. Thanks for your participation. Please rate the following statements on a scale from one to five (five being the higher score or more favorable situation). ● I am pleased that my HEI was selected to participate in the Advance Program. Strongly Disagree ( ); Disagree ( ); Neutral ( ); Agree ( ); Strongly Agree ( ) ● The tools and techniques that my HEI was exposed to contributed to several improvements in curriculum design. Strongly Disagree ( ); Disagree ( ); Neutral ( ); Agree ( ); Strongly Agree ( ) ● I believe that students performed better because of the new and increased capacity of the lecturers and staff. Strongly Disagree ( ); Disagree ( ); Neutral ( ); Agree ( ); Strongly Agree ( ) ● Teachers enjoyed interacting with the Advance Program. Strongly Disagree ( ); Disagree ( ); Neutral ( ); Agree ( ); Strongly Agree ( ) ● Most of the tools and techniques that Advance exposed the HEI to were new and innovative. Strongly Disagree ( ); Disagree ( ); Neutral ( ); Agree ( ); Strongly Agree ( ) ● The HEI were active participants in defining the activities of Advance. Strongly Disagree ( ); Disagree ( ); Neutral ( ); Agree ( ); Strongly Agree ( ) ● Advance has positively affected the development of the staff of the HEI and there is increased capacity. Strongly Disagree ( ); Disagree ( ); Neutral ( ); Agree ( ); Strongly Agree ( ) ● The HEI will need additional resources to sustain or scale up the: o outputs of Advance Strongly Disagree ( ); Disagree ( ); Neutral ( ); Agree ( ); Strongly Agree ( ) o funding for students Strongly Disagree ( ); Disagree ( ); Neutral ( ); Agree ( ); Strongly Agree ( ) o funding for curriculum and instructional updating 81 | USAID/JAMAICA ADVANCE FINAL PERFORMANCE EVALUATION USAID.GOV Strongly Disagree ( ); Disagree ( ); Neutral ( ); Agree ( ); Strongly Agree ( ) o support/resources for student outplacement on internships and job placement Strongly Disagree ( ); Disagree ( ); Neutral ( ); Agree ( ); Strongly Agree ( ) o socio-emotional support of participating students Strongly Disagree ( ); Disagree ( ); Neutral ( ); Agree ( ); Strongly Agree ( ) ● Graduates of the Advance Program will be better able to access job opportunities in the labor market than students that were not part of the program. Strongly Disagree ( ); Disagree ( ); Neutral ( ); Agree ( ); Strongly Agree ( ) ● I will participate in another similar program and encourage colleagues and staff to do the same. Strongly Disagree ( ); Disagree ( ); Neutral ( ); Agree ( ); Strongly Agree ( ) ● I think that there is need to expand Advance’s activities. Strongly Disagree ( ); Disagree ( ); Neutral ( ); Agree ( ); Strongly Agree ( ) ● I think we need to prepare a lessons learned register and best practices register. Strongly Disagree ( ); Disagree ( ); Neutral ( ); Agree ( ); Strongly Agree ( )