Final Performance Evaluation: Bridges to Employment in El Salvador June 2020 (revised July and September 2020) This publication was produced for review by the United States Agency for International Development under task order AID-OAA-M-12-00020. It was prepared by Ivonne Padilla, Patricia Costa, Nancy Murray and Larissa Campuzano of Mathematica. Final Performance Evaluation: Bridges to Employment in El Salvador Cover photo caption: High school students from the flexible education program. Photo credit: USAID Bridges to Employment. DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA iii ABSTRACT Background. This report presents the results of an independent performance evaluation of the Bridges to Employment project in El Salvador funded by the U.S. Agency for International Development (USAID). Questions. The evaluation covered topics related to the extent to which Bridges reached outcome targets; key barriers and facilitators the project faced; stakeholders’ and beneficiaries’ perceptions of achievements; positive and negative results; the integration of gender equality, female empowerment, and social inclusion in project activities; and sustainability. Methods. The mixed-methods performance evaluation uses Bridges monitoring data and three types of qualitative data: (1) semi-structured interviews with key informants (KIIs) including staff from Bridges, training centers, employers, and representatives from USAID/El Salvador and the Salvadorian Vocational Training Institute (INSAFORP); (2) focus group discussions (FGDs) with instructors and trainees; and (3) a comprehensive desk review of Bridges reports. Key Findings. Despite some initial delays in the implementation, Bridges met almost all project milestones, in part because of its focus on collaborating, learning, and adapting (CLA). A total of 11,967 youth completed training. The project also met its employment target: a total of 4,708 trainees secured employment. Bridges improved the quality of the training and support offered and has shifted the focus of the professional training sector so that trainings offered respond to the needs of the labor market. Bridges also strengthened the technical and managerial capacity of local organizations in El Salvador, thereby contributing to USAID’s Journey to Self￾Reliance. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA iv ACKNOWLEDGMENTS This report is the result of the combined efforts of many people, including Barbara Knox￾Seith, Karen Towers, and Megan Meinen, current and former team members of the U.S. Agency for International Development (USAID) Latin America and the Caribbean (LAC) education team. They all provided guidance and support throughout. We are also grateful to Yolanda Martínez (Contracting Officer Representative for the Bridges to Employment Task Order), Orlando Hidalgo, Gabriela Vélez and Chris Moore, current and former team members of USAID/El Salvador, for their helpful input throughout the evaluation. The report would not have been possible without the contributions of Bridges to Employment staff, including the Chief of Party, Caterina Valero and Senior Monitoring and Evaluation (M&E) and Implementation Manager Specialist, Mario Rivas, who shared project materials, provided data, responded to emails, and spent hours on the phone helping us to understand their project. We are very appreciative of all the youth, training center staff, and private sector representatives for allowing us to learn from their perspectives. We also thank our local consultant, Isabel Rodríguez for her support in collecting data in El Salvador. At Mathematica, Jorge Ugaz supported the design and early implementation of the project. Matt Spitzer and Joshua Claxton provided programming assistance for the data analysis. Beryl Seiler provided critical support for this final report. We would also like to thank the editorial and administrative support staff at Mathematica. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA v CONTENTS ABSTRACT ...................................................................................................................................................iii ACKNOWLEDGMENTS................................................................................................................................iv ACRONYMS ................................................................................................................................................IX EXECUTIVE SUMMARY ........................................................................................................................ ES-1 A. Introduction.................................................................................................................................... ES-1 B. Evaluation questions and design .................................................................................................. ES-1 C. Findings of the main evaluation questions .................................................................................... ES-2 D. Lessons learned and recommendations ....................................................................................... ES-4 I. BACKGROUND AND INTRODUCTION................................................................................................. 1 A. El Salvador context ............................................................................................................................. 1 B. U.S. Government and USAID strategy................................................................................................ 1 C. Brief literature review relevant to Bridges ........................................................................................... 2 D. Overview of the evaluation and Bridges to Employment .................................................................... 2 E. Purpose of this evaluation................................................................................................................... 6 II. EVALUATION QUESTIONS, DATA COLLECTION, AND METHODOLOGY........................................ 7 A. Evaluation questions ........................................................................................................................... 7 B. Evaluation design ................................................................................................................................ 7 C. Data collection..................................................................................................................................... 8 D. Analysis plan ..................................................................................................................................... 11 E. Evaluation limitations and steps taken to mitigate them ................................................................... 12 III. FINDINGS: IMPLEMENTATION AND RESULTS ................................................................................ 15 A. How was Bridges implemented?....................................................................................................... 15 B. Was Bridges implemented as planned?............................................................................................ 26 C. To what extent has Bridges reached outcome targets?.................................................................... 30 D. What were key barriers and facilitators to the achievement of project results?................................ 46 IV. STAKEHOLDER PERCEPTIONS OF RESULTS ................................................................................ 51 A. What are stakeholders and beneficiaries’ perceptions of the training program and the support services Bridges provided? .................................................................................................. 51 B. What positive and negative results, intended or unintended, has Bridges produced at the end of the project?............................................................................................................................. 71 FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA vi V. FINDINGS: SUSTAINABILITY.............................................................................................................. 73 A. Are Bridges activities sustainable? ................................................................................................... 73 B. What results produced by Bridges seem to be the most sustainable? ............................................. 75 VI. CONCLUSIONS, LESSONS LEARNED, AND RECOMMENDATIONS.............................................. 78 A. Conclusions and major findings ........................................................................................................ 78 B. Lessons learned ................................................................................................................................ 79 C. Recommendations ............................................................................................................................ 80 REFERENCES............................................................................................................................................ 82 APPENDIX A ADDITIONAL TABLES AND FIGURES APPENDIX B DESIGN REPORT APPENDIX C PROTOCOLS AND FGD GUIDES APPENDIX D LIST OF INFORMATION SOURCES APPENDIX E DISCLOSURES OF COI APPENDIX F SUMMARY OF INFORMATION ABOUT THE EVALUATION TEAM FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA vii TABLES II.1. Data sources for this analysis .......................................................................................................... 9 II.2. Evaluation questions ...................................................................................................................... 13 III.1. Changes in Bridges’ implementation as a result of CLA................................................................ 26 III.2. Implementation targets and goals achieved by year ..................................................................... 28 III.3. Implementation targets updated .................................................................................................... 29 III.4. Number of firms, disaggregated by the number of Bridges trainees hired .................................... 30 III.5. Progress in certification and accreditation ..................................................................................... 33 III.6. Bridges implementation facilitators and barriers according to stakeholders and beneficiaries ................................................................................................................................... 46 IV.1. Main strengths and weakness of training courses as reported by youth....................................... 56 IV.2. Goals of training programs and trainees needs............................................................................. 58 IV.3. Barriers perceived by trainees and addressed by the project........................................................ 60 IV.4. Trainees’ perceived before and after effects of Bridges project .................................................... 63 IV.5. Employers’ perceptions on trainees’ skills and performance......................................................... 66 IV.6. Private sector’ perceptions of inclusion of vulnerable youth in their staff ...................................... 68 IV.7. Positive results produced by Bridges............................................................................................. 71 V.1. Sustainability assessment of TCs and the services provided........................................................ 73 FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA viii FIGURES I.1 Theory of change of Bridges project................................................................................................ 4 I.2. Bridges’ approach to improve youth employment............................................................................ 6 III.1. Municipalities included in the Bridges project ................................................................................ 17 III.2. Bridges implementation timeline .................................................................................................... 19 III.3. Professional training courses offered by Bridges .......................................................................... 25 III.4. Number of training centers strengthened....................................................................................... 31 III.5. Number of training center, NGO, CSO, and private and public sector staff enrolled in Bridges training .............................................................................................................................. 32 III.6. Number of vulnerable youth enrolled in training courses............................................................... 35 III.7. Number of vulnerable youth who completed training courses ....................................................... 36 III.8. Number of youth with improved knowledge or skills following completion of training ................... 37 III.9. Number of trainees with new or better employment following completion of training.................... 38 III.10. Enrollment and completion by grantee .......................................................................................... 39 III.11. Completion of training and employment by grantee ...................................................................... 40 III.12. New or better job by type of training for youth trained ................................................................... 41 III.13. Training completion and employment by sector and sex............................................................... 42 III.14. Number of youth trained by age group compared with the age group of youth with employment.................................................................................................................................... 43 III.15. Youth with employment in the same sector they received training................................................ 44 III.16. Training and employment of youth by municipality........................................................................ 45 IV.1. Percentage of youth employed from youth that completed training the corresponding year ........ 54 IV.2. Youth employed by sector ............................................................................................................. 55 IV.3. Dropout by duration of training course........................................................................................... 57 IV.4. Training and employment of youth in the ICT sector..................................................................... 62 IV.5. Completion of training and youth employed, disaggregated by sex .............................................. 69 IV.6. Completion of training in energy and plastics sectors, disaggregated by sex ............................... 70 FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA ix ACRONYMS ALTHES Asociación de Líderes del Talento Humano (Association of Human Talent Leaders) ASAPROSAR Asociación Salvadoreña Pro-Salud Rural (Salvadoran Association of Rural Health) ASIPLASTIC Asociación Salvadoreña de la Industria del Plástico (Plastic Industry Association) BPA Blank Purchase Agreement BTV Bachillerato Técnico Vocacional (Technical High School Degree) CAIM Centro de Atención Integral para Migrantes (Center for Integral Attention to Migrants) CAPUCOM Centro de Capacitación en Computación (Computing Training Center) CasaTIC Cámara Salvadoreña de Tecnologías de la Información y Comunicación (Salvadoran Information Technology Chamber) CLA Collaborating, Learning and Adapting COED Complejo Educativo Thomas Jefferson (Thomas Jefferson School Center) CONEXIÓN Asociación Conexión al Desarrollo de El Salvador (Connection to Development Association of El Salvador) CSO Civil Society Organization CVPP Crime and Violence Prevention Program DAI DAI Global, LLC DIGESTYC Dirección General de Estadística y Censos (El Salvador’s National Statistical Directorate) DISAL Distribuidora Salvadoreña DLI Direct Labor Intermediation EMPRE Municipal Enterprise Development Unit ESFE/AGAPE Escuela Superior Franciscana Especializada/ Asociación AGAPE de El Salvador (Specialized Franciscan High School/Agape Association of El Salvador) FEPADE Fundación Empresarial para el Desarrollo Educativo (Entrepreneurial Foundation for Development) FGD Focus Group Discussions FGK Fundación Gloria de Kriete (Gloria de Kriete Foundation) FIECA Fundación Innovaciones Educativas Centroamericanas (Central American Innovative Education Foundation) FUNDEPLAST Fundación para el Desarrollo Integral de los Trabajadores de la Industria del Plástico (Foundation for the Integral Development of Workers of the Plastic Industry) FUNDEMAS Fundación Empresarial para la Acción Social (Business Foundation for Social Action) FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA x FUNPRES Fundación Pro Educación de El Salvador (Pro-Salvadoran Education Foundation) FUSALMO Fundación Salvador del Mundo (Salvador del Mundo Foundation) FY Fiscal Year FyA Fe y Alegría (Faith and Happiness Association) GDP Gross Domestic Product GUC Grant Under Contract ICT Information, Communication, and Technology INCA Instituto Nacional de Ciudad Arce (National Institute of Ciudad Arce) INDEL Instituto Nacional de Lourdes Colón (National Institute of Lourdes Colón) INJUVE Instituto Nacional de la Juventud (National Institute for Youth) INSAFORP Instituto Salvadoreño de Formación Profesional (Salvadoran Vocational Training Institute) INSO Instituto Nacional Sonzacate (National Institute of Sonzacate) INTI Instituto Nacional Técnico Industrial (National Technical Industrial Institute) KII Key Informant Interviews LGBTI Lesbian Gay Bisexual Trans Intersex LOP Life of Project M&E Monitoring and Evaluation MCC Millennium Challenge Corporation MEGATEC Modelo Educativo Gradual de Aprendizaje Técnico y Tecnológico (Gradual Educational Model of Technical and Technological Learning) MINEDUCYT Ministerio de Educación, Ciencia y Tecnología (Ministry of Education, Science and Technology) MyV Metas y Visión (Goals and Vision) NGO Nongovernmental Organization OCA Organizational Capacity Assessment OEF Asociación para la Organización y Educación Empresarial Femenina de El Salvador (Association for the Organization and Education of Women in Business of El Salvador) PFG Partnership for Growth Presidential Initiative PROESA Organismo Promotor de Exportaciones e Inversiones de El Salvador (Organization for Promotion of Exports and Investment in El Salvador) RFA Request for Application SDC Software Development Center SSPAS Servicio Social Pasionista (Passion Corporation – Passionate Social Service) TC Training Center TEC Total Estimated Ceiling UDEL Local Economic Development Unit FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA xi UEES Universidad Evangélica de El Salvador (Evangelical University of El Salvador) UFG Universidad Francisco Gavidia UNIVO Universidad de Oriente (Eastern University) U.S. United States USAID United States Agency for International Development USAM Universidad Salvadoreña Alberto Masferrer (Alberto Masferrer Salvadoran University) USG United States Government WFD Workforce Development WIA Workforce Investment Activity This page has been left blank for double-sided copying. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA ES.1 EXECUTIVE SUMMARY A. Introduction This report presents the findings of the final performance evaluation of Bridges to Employment (referred to throughout this document by the shortened name, Bridges) project funded by the U.S. Agency for International Development (USAID) in El Salvador. Bridges is a five-year, $33.9 million project implemented by DAI Global (DAI) and its subcontractors, Plan International and JBS International, Inc. Bridges was designed to help increase and improve employment for vulnerable youth living in selected high-crime municipalities. USAID’s development hypothesis states that “when at-risk youth have access to a high-quality, holistic, and market-relevant package of services and support that utilize and maximize their strengths, they are more likely to improve their income and employment situation.” (DAI 2016). To help increase and improve employment, Bridges has three main objectives: (1) increase stakeholder engagement to improve the enabling environment for workforce development (WFD) and employment; (2) increase institutional strengthening to improve the quality of WFD services that effectively respond to market demand to insert at-risk youth into target economic sectors; (3) increase youth engagement to improve workforce readiness. B. Evaluation questions and design To address the need for high quality evidence on investments in education and employability, USAID contracted with Mathematica as its independent evaluator to design and conduct evaluations of USAID’s investments in education-access interventions in the LAC region through a project known as the Latin America and the Caribbean Reads (LAC Reads) evaluation. This performance evaluation was supported under the LAC Reads Evaluation project and provides insights into how Bridges was implemented and whether the project achieved its targets and key objectives. USAID/El Salvador staff identified several questions to guide this evaluation. 1. What were key barriers and facilitators to the achievement of project results? 2. To what extent has Bridges reached outcome targets in the following areas: employment, improved skills, and educational outcomes among beneficiaries; strengthening of training centers (TCs); number of new/revised certified programs; private sector inclusion of vulnerable youth in their staff; and the sustainability of the private sector firms’ and training centers’ relationship to update curricula and trainings? 3. What positive and negative results, intended or unintended, has Bridges produced at the end of the project? 4. What are the perceptions from employers, trainees, and grantees regarding the training program and the support services provided by Bridges? 5. How were gender equality, female empowerment, and social inclusion integrated in the implementation of activity interventions? Were those efforts successful? Why or why not? To answer these evaluation questions, we implemented a mixed methods performance evaluation using both qualitative and quantitative data. The analysis draws on three types of FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA ES.2 qualitative data: (1) semi-structured, in-person interviews with key informants (KIIs) including staff from Bridges, select training centers, large employers of project trainees, and representatives from USAID/El Salvador and the Salvadorian Vocational Training Institute (INSAFORP); (2) focus group discussions (FGDs) with instructors from TCs and with Bridges trainees (both current and former); and (3) a comprehensive desk review of Bridges’ monthly and semiannual reports and assessments or special studies. The quantitative data comes from Bridges’ monitoring data which was collected by each of the grantees and verified by Bridges. C. Findings of the main evaluation questions 1. To what extent has Bridges reached outcome targets? Bridges exceeded most of the life of project (LOP) targets. Below is a summary of project achievements by objectives. Objective 1. Bridges addressed the legal barriers to youth employment by supporting improvements in existing regulation. Bridges assisted government agencies in creating or updating policies aiming to improve workforce development. In total, 36 laws or regulations were adopted or proposed. The new (or updated) policies established legal frameworks for the rights of youth and responded to the employment-specific needs of youth. Objective 2. Bridges exceeded its goal related to strengthening training centers. Twenty-nine training centers were strengthened in a variety of ways, whether through official capacity-building interventions or through support to manage USAID-funded grants and the associated operational, administrative, and financial requirements. Objective 3. Bridges was able to exceed most targets; 11,967 youth completed training and 4,708 youth were employed in six priority sectors. Bridges’ ability to achieve the employment target is impressive given the local context. El Salvador produces only 30,000 jobs per year on average, but 40,000 jobs are needed on a yearly basis to absorb new entrants to the labor market. The labor market relies on small and medium businesses to drive growth. More than 1,000 individual firms hired Bridges trainees, and 88% of the firms hired less than 5 trainees. Out of the trainees employed, 40 percent received both technical and soft skills training, and 71 percent were employed in the sector for which they received training. This finding suggests that Bridges succeeded in identifying sectors’ needs, and matching training offering with the identified needs. 2. What were key facilitators and barriers to the achievement of project results? Early assessments carried out by Bridges reinforced the mission’s original design and helped the project focus on activities that yielded intended outcomes. The mission relied on several studies and country level analyses to design Bridges. In the first years of implementation, the project carried out important studies to continue to understand the context in El Salvador and develop project activities based on the identified needs. Focusing on the enabling environment, strengthening institutions, and training youth were critical components to youth employment. The Bridges team relied on data, collaboration and learning to identify needs and incorporated adaptations that were critical for the successful implementation of the project. Bridges and USAID/El Salvador embodied the spirit of USAID’s Collaboration, Learning, and Adapting (CLA) approach and worked with the private sector and training centers to shift and modify activities to ensure project success. Some of the most notable adjustments to their FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA ES.3 original design included the following: (1) Bridges integrated life skills training into technical curricula. (2) Bridges offered a flexible education modality for youth to complete high school. (3) Bridges added job placement services for vulnerable youth. (4) Bridges also produced ad hoc training materials on psychosocial support, entrepreneurship, life skills and job readiness for workforce development service providers. The implementation through a Grants Under Contract (GUC)1 model and a reduction in funding were barriers that Bridges overcame through the facilitators described above: CLA and an evidence-based design. Initially, the implementation through a GUC model was challenging since it took an extraordinary amount of effort and hand holding to build grantee capacity. While these capacity building efforts support USAID’s Journey to Self-Reliance, they did result in initial delays to implementation. The shift in U.S. Government priorities in year four also resulted in a reduction in award ceiling and uncertainty for the project and its stakeholders. 3. What positive and negative results has Bridges produced at the end of the project? Bridges elevated the role of the private sector in the workforce development system in El Salvador, and strengthened this system by raising awareness, building capacity of institutions, and creating a bridge between the private sector and training institutions. Because of Bridges’ efforts to work closely with the private sector and engage them in the activities developed by the TCs, an important shift in mindset and practices occurred with the TCs. Training centers have begun offering trainings based on business’ needs instead of trainees’ demands. One of Bridges’ major accomplishments is building the capacity of training centers, and many of these efforts will continue to yield outcomes after the project closes. The full benefits of the organizational strengthening work carried out by Bridges may not have emerged yet, but given the project’s focus on strengthening the capacity of TCs in both technical and managerial aspects, the future of USAID projects stand to benefit from a cadre of training centers with the capacity to implement market-relevant activities aiming to improve youth employability. These efforts reflect a focus on working through local partners and building their capacity as articulated in USAID’s Journey to Self-Reliance2 . Youth improved their socioemotional skills and almost 40 percent of graduates secured a job. Youth who were trained in life skills improved their interpersonal skills, and many noted those skills helped them in the job search process. For most of the youth with employment, this represented their first employment opportunity. We did not identify any significant negative results produced by Bridges, intended or unintended, at the end of project. Stakeholders agreed that the project did not produce any negative results. Challenges arose during the implementation, but these were overcome. 4. What are the perceptions of employers, trainees, and grantees regarding the training program and the support services provided by Bridges? Employers, trainers, and youth lauded the life skills program. Employers and trainees agreed on the importance of teaching youth both technical and life skills. Representatives from 1 A GUC is a mechanism USAID uses to allow a contractor to issue grants with non-governmental organizations or governments. As part of Bridges, a grant fund was established to fund activities carried out by local organizations. 2 The Journey to Self-Reliance is a USAID strategy aimed at achieving locally-sustained results, by focusing, on among other things on strengthening local capacities, and accelerating enterprise-driven development. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA ES.4 the private sector also value the combination of life and technical skills. Youth also felt that developing life skills was particularly helpful because they have renewed self-confidence, which aided the job search process. Because of the success of the life skills modules that Bridges incorporated into the technical training program, INJUVE, El Salvador’s National Youth Institute, has now adopted these modules and will offer this training to youth in El Salvador. 5. How were gender equality, female empowerment, and social inclusion integrated in the implementation? Were those efforts successful? Why or why not?’ Bridges’ laid the foundation for female youth to find employment in traditionally male￾dominated sectors. All activities of the project incorporated some aspect of gender equality and/or social inclusion. Bridges made a concerted effort with training centers to include more females in traditionally male-dominated industries. That effort paid off: some women made significant breakthroughs by completing training and getting jobs in traditionally male￾dominated industries such as logistics, information communication and technology (ICT), plastics, and energy. D. Lessons learned and recommendations 1. Lessons learned The project’s pivot to focus more on job placement services thanks to its CLA efforts, benefited youth and contributed to achieving the project’s employment target. Because of an identified demand, in the third year of the project, Bridges provided all youth with help navigating the job search and interview process, in addition to training. The additional support contributed to accelerating employment in the last two years of the project. Incorporating life skills modules into technical training provided youth with valuable life skills which were viewed favorably by the private sector. The initial project design did not envision providing all youth with both technical and life skills training. However, all stakeholders and beneficiaries agreed with the benefits of youth developing life skills as part of a WFD training program. Employers perceived a difference in Bridges trainees’ job performance from the gained competencies. Trainees also perceived the value of life skills. The strategy to target specific economic sectors was effective to meet employment goals, but different eligibility criteria may be appropriate depending on the sector. Bridges took two strategic approaches to achieve an ambitious job placement goal. Focusing on the commerce/tourism (otherwise known as service industry) and manufacturing sectors, where firms hire youth with the profile of Bridges trainees, was effective to achieve the expected results in employment. Focusing on less traditional sectors such as information and communication technology (ICT) and energy was a groundbreaking opportunity for youth where they can access better incomes. However, in these sectors, it could be useful to explore more tailored eligibility criteria, such as supporting youth who may have been trained elsewhere but still require support in finding jobs or training youth who may have a college degree but continue to need support based on their vulnerability. Implementing through local organizations fosters development and sustainability but USAID should expect that organizations with limited USAID experience will likely take longer to achieve results. When the implementation of a workforce development project is carried out through existing institutions, it is important to consider that outcomes related to youth FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA ES.5 training and employment may take longer to materialize and targets should reflect this slower start-up. Lack of monitoring data on non-completers and youth that did not secure employment only provides a partial overview of accomplishments. Bridges did not monitor the reasons for non-completion or non-employment. There may have been positive outcomes (such as finding a job despite not completing) or return to education (after training) that the project did not monitor which may have provided more information on positive outcomes. 2. Recommendations For USAID and other donors: Set targets that reflect the local context and determine the size of the grants program according to the targets and the capacity of existing organizations. Gross domestic product (GDP), youth employment rates, and the make-up and dynamism of the economy should be taken into consideration when setting employment targets. Because WFD interventions rarely focus on generating jobs- but rather on preparing youth to compete for job opportunities- projects need to consider the availability of jobs (with both small and large companies) when setting employment targets. Similarly, donors need to assess the capacity of organizations to adequately manage grants when determining the size of the grants program. For USAID and other donors: Working on a systems approach to WFD is effective, but may be more effective if interventions are sequenced correctly. Fostering an enabling environment that values youth employability is important. Equally important is ensuring that the institutions engaged to train youth have the capabilities and facilities to provide quality trainings. These two components of the workforce development system are necessary for strong trainings programs. Donors should focus on strengthening local capacity and then offer services to increase employability of youth, or consider contracting training interventions separately so that they start once other components of the workforce development system are strengthened. For USAID and other donors: Target workforce development training and job placement services to older youth if the goal of the intervention is employment. While workforce development training programs are often offered to youth under 18, the private sector prefers to hire youth who are older than 18 due to their level of maturity, as many of them have completed the minimum academic level required by the formal economy (i.e. high school). Therefore, WFD training programs with a focus on employment should train youth in the age group where they are more likely to be employed. For USAID, other donors and implementing partners: Continue to offer life skills training as part of workforce development training. Stakeholders agreed on the benefits of offering life skills training to youth who gained critical competencies demanded by employers. Employers valued hiring youth who were trained with life skills because they perceived a difference in performance. Because of the value of life skills training in improving self￾confidence and resilience of youth, WFD projects should assess existing programs offering life skills, customize them to their needs and include life skills training in all programs. For USAID, other donors and implementing partners: Offer job placement services as an integral component to any WFD project. If employment is an outcome of interest, job placement services should be part of the training program. In addition, WFD projects should consider offering job placement services to vulnerable youth who may have received training elsewhere and/or may already have the skills required but need support to find a job. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 1 I. BACKGROUND AND INTRODUCTION A. El Salvador context Bridges to Employment (referred to throughout this document by the shortened name, Bridges) was designed in the context of high levels of crime, violence, lack of opportunities for youth and out migration in El Salvador. In the summer of 2014, more than 67,000 unaccompanied children and youth left their homes in El Salvador, Honduras, and Guatemala (also referred to as Northern Triangle Countries) to emigrate to the United States (USAID 2017). The spike in migration was seen as a response to high rates of violence and limited economic opportunities. El Salvador homicide rates have decreased in the last five years, but conditions remain unsafe. According to statistics published by the national police, homicide rates declined from 103 per 100,000 in 2015 to 50 per 100,000 in 2019 (Policía Nacional Civil 2019). Despite this improvement, El Salvador continues to have one of the highest homicide rates in the Americas and the world. Both urban and rural areas of El Salvador have areas of high gang presence, thereby limiting the mobility of their residents and in many cases blocking access to economic opportunities. Often, youth born into these communities carry with them the stigma associated with these high-gang areas and their employment potential is stymied. The combination of unsafe situations and geographic-based stigma creates an additional roadblock to a secure future for youth from these regions. Compounding the situation, El Salvador’s economy has experienced muted gross domestic product (GDP) growth in the past five years. Hovering around 2 percent annual growth, the Salvadorian economy does not produce enough jobs to absorb new entrants to the labor market. In fact, the country produces only 30,000 jobs per year, whereas 40,000 jobs are needed on a yearly basis to absorb new entrants to the labor market (World Bank 2019). Unemployment rates among youth are more than twice that of the national average; the official unemployment rate for youth ages 15 to 24 was 13.6 percent in 2018, compared to a national rate of 6.3 percent (DIGESTYC 2018). B. U.S. Government and USAID strategy Bridges was designed under the auspices of the interagency Partnership for Growth Presidential Initiative (PFG). PFG was a bilateral partnership that leveraged resources from both the United States and the government of El Salvador to support a set of common goals for broad￾based economic growth. As part of El Salvador’s participation in PFG, in 2011, a team of economists carried out a constraints analysis to identify the main barriers to growth for El Salvador. The analysis determined that the key to sustained economic growth in El Salvador was mitigating two primary constraints: (1) high levels of crime and insecurity and (2) a weak tradables sector. This analysis also determined that a factor contributing to a weak tradables sector was a lack of skilled workforce. Bridges was also designed to contribute to the U.S. Strategy for Central America, a strategy that aims to address the security, governance, and economic drivers of illegal immigration and illicit trafficking in Central America. Although the strategy’s focus has changed under each of the administrations since Bridges started, Bridges continues to contribute to both the prosperity and security pillars of the strategy. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 2 Bridges also contributes to the United States Agency for International Development’s (USAID’s) Journey to Self-Reliance.3 Through this approach, USAID works to empower local people and organizations to achieve locally sustained results (USAID 2019). In the case of Bridges, partnerships with local organizations and businesses are at the heart of its strategy. Local organizations are the drivers of training services, and the local private sector is instrumental to employing youth. Working together, these organizations are supporting El Salvador’s journey to self-reliance. C. Brief literature review relevant to Bridges Substantial literature shows that early joblessness has long-lasting consequences at both the individual and society level. Research indicates that prolonged or repeated periods of early joblessness can impose a lifetime earnings penalty of up to 20 percent and lead to more joblessness later in life (Gregg and Tominey 2005). Youth who are unemployed or not in school are also more vulnerable to suffering from poor physical and mental health and to becoming involved in the consumption and trade of drugs, crime, and possibly even terrorism (International Monetary Fund 2012; Ali 2013). In an effort to increase youth employment, a global effort has rolled out many youth workforce development (WFD) programs around the world. Many of these have focused on providing technical skills, but an increasing number include a variety of complementary components such as life skills and on-the-job training. The current literature on WFD programs examines the effects of multicomponent programs. A review of 54 studies published between 2001 and 2012 found positive results from performance evaluations of specific multicomponent vocational training programs, especially for countries from Latin America and the Caribbean. These studies echoed the findings from the World Bank (Almeida et al. 2012) that showed integrated programs combining on-the-job training, classroom components, skills training, and counseling are the most effective (Olenik 2013). However, training alone cannot ensure a strong workforce development system. The system encompasses all resources, activities, people, and public and private institutions focused on improving, expanding, and sustaining desired outcomes, as defined for the education system overall in the U.S. Government Strategy on International Basic Education 2018 (USG 2018). WFD system-strengthening activities can involve increasing the capacity of the enabling environment or of training centers (TCs) within the system to deliver relevant and high quality services for youth to enter the labor market. D. Overview of the evaluation and Bridges to Employment USAID contracted with Mathematica as its independent evaluator to design and conduct evaluations of early grade reading and interventions to improve access to education, in the LAC region in a project known as the Latin America and the Caribbean Reads (LAC Reads) evaluation. Under the LAC Reads evaluation contract, Mathematica worked closely with the Bridges implementers to design a performance evaluation of the project (Ugaz et al. 2019)4 . In 3 The Journey to Self-Reliance is a USAID strategy aimed at achieving locally-sustained results, by focusing, on among other things on strengthening local capacities, and accelerating enterprise-driven development. 4 In October 2018, USAID/El Salvador informed USAID/Washington and Mathematica that it had decided to move forward with a performance evaluation for Puentes—as opposed to the impact evaluation that was originally planned—because the program implementation had changed considerably over the early years of program rollout. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 3 this report, we present the final results of our performance evaluation of the project which provides insights into how the project was implemented and whether Bridges reached its targets and key objectives USAID is funding workforce development projects across Central America which have been profiled in Central America WFD annual reports in 2018 (Bagby et al. 2018), 2019 (Bagby et al. 2019) and 2020 (Bagby et al. 2020). Bridges is one of USAID’s largest workforce development investments in the LAC region: it is a five-year, $33.95 million project implemented by DAI Global (DAI) and its subcontractors, Plan International, and JBS International, Inc. USAID designed Bridges to help increase and improve employment for vulnerable youth living in selected high-crime municipalities. USAID’s development hypothesis states that “when at-risk6 youth have access to a high-quality, holistic, and market-relevant package of services and support that utilize and maximize their strengths, they are more likely to improve their income and employment situation.” (DAI 2016). As depicted in the theory of change in Figure I.1, three distinct objectives were designed to help promote the enabling environment, strengthen training centers, and offer training and other support services to youth to help them secure employment. 5 $33,864,538 is the amount approved in the last contract modification in 2019, the original contract amount for Bridges was $42.2 million. 6 Bridges transitioned from calling youth in the program “at risk” to “vulnerable.” We have used the term vulnerable to match the practices of USAID/El Salvador and Bridges but use at risk only if it is stated in original contract documents. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 4 Figure I.1 Theory of change of Bridges project Source: Authors’ theory of change depiction of Bridges project. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 5 In conjunction with other USAID funded projects, Bridges’ long-term goal is to address the underlying factors driving migration from specific municipalities by offering productive economic opportunities. Bridges focused its interventions on targeted vulnerable youth who met the following criteria: • Be between the ages of 16 to 29 • Reside in selected high-crime municipalities as defined in El Salvador’s national security plan, Plan El Salvador Seguro (Plan for a Safe El Salvador) • Require technical training, education, psychosocial support, and/or related assistance to complete their education or to find, keep, and/or improve employment • Demonstrate a commitment to their own development and their willingness to learn and improve their lives by committing the time and effort required to find and maintain employment To help increase and improve employment, Bridges has three main objectives: Objective 1: Improve the enabling environment promoting workforce development for vulnerable youth in targeted sectors Objective 2: Increase institutional strengthening to improve the quality of workforce development services to effectively insert vulnerable youth into targeted sectors Objective 3: Improve workforce readiness demonstrated by targeted vulnerable youth Although all three objectives were implemented at the same time, Objectives 1 and 2 set the foundation for Objective 3 activities. Namely, in order to be able to train youth, Bridges had to work to build an enabling environment optimal for youth workforce development (Objective 1). At the same time, Bridges had to strengthen the capacity of local training centers or civil society organizations (CSOs; Objective 2) to be able to provide high quality training and services to youth, including both professional training and support to complete a high school education (Objective 3). Underlying all three objectives was strong collaboration with the private sector. Under Objective 1, Bridges helped raise the profile of technical and vocational education among private sector stakeholders while also working with the private sector to make its hiring practices more youth friendly. Under Objective 2, collaboration with the private sector was necessary to ensure the curricula and services provided by the training centers responded to the needs of the private sector. Finally, under Objective 3, the private sector helped build the skills of youth by facilitating on-the-job training, offering internship opportunities, and also employing youth (Figure I.2). FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 6 Figure I.2. Bridges’ approach to improve youth employment Source: Bridges reports to USAID. E. Purpose of this evaluation The purpose of this final performance evaluation is to provide insights into how the project was implemented and whether Bridges reached its targets and key objectives. It also includes stakeholders’ perceptions about Bridges’ implementation and results and what lessons have been learned that might inform the next generation of WFD programs in El Salvador and Central America. The intended audience for this evaluation is USAID (both in Washington and in the field), implementing partners and other WFD practitioners, and other donors thinking of funding and designing similar programs. The evaluation will inform stakeholders about the outcomes that resulted from the project. Stakeholders are expected to use the lessons learned to strengthen the components associated with positive outcomes, adjust components that did not work well, and make adjustments based on identified needs. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 7 II. EVALUATION QUESTIONS, DATA COLLECTION, AND METHODOLOGY In this chapter, we discuss the performance evaluation’s primary questions as well as the data collection and analysis we conducted to answer these questions. A. Evaluation questions The evaluation was designed to describe and assess the performance of Bridges and generate knowledge on the extent to which the project prepared vulnerable youth to increase their employability and help them find employment. The evaluation assesses which aspects of the program worked well, which could have been improved, what barriers or facilitators seem to have helped or hindered its implementation, and what key stakeholders’ perceptions are about the benefits of Bridges services to prepare vulnerable youth for employment. Motivated by the desire to inform USAID’s future projects aiming to increase and improve employment of vulnerable youth, USAID/El Salvador representatives identified several questions for this evaluation. These questions can be divided into five global sets of questions: 1. What were key barriers and facilitators to the achievement of project results? 2. To what extent has Bridges reached outcome targets in the following areas: employment, improved skills, and educational outcomes among beneficiaries; strengthening of TCs; number of new/revised certified programs; private sector inclusion of vulnerable youth in their staff; and the sustainability of the private sector firms’ and training centers’ relationship to update curricula and trainings? 3. What positive and negative results, intended or unintended, has Bridges produced at the end of the project? 4. What are the perceptions from employers, trainees, and grantees regarding the training program and the support services provided by Bridges? 5. How were gender equality, female empowerment, and social inclusion integrated in the implementation of activity interventions? Were those efforts successful? Why or why not?’ Table II.2 provides the list of all the questions, the associated data source used to answer the question, and the chapter where the question is answered. For the purposes of readability, we made some minor changes to the questions compared to those that were included in the evaluation design report. Table A.1 in the appendix provides a comparison of the questions and an explanation for the change. B. Evaluation design Performance evaluations are not designed to detect project impacts or attribute changes in participant outcomes to the project or specific components of it because they lack a suitable comparison group or counterfactual for beneficiaries. However, the evaluation will generate and disseminate valuable learning and insights about the implementation and potential effects of the Bridges project as a whole. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 8 In coordination with USAID/Washington, USAID/El Salvador, and Bridges, Mathematica conducted this final performance evaluation relying on mixed methods and using both qualitative and quantitative data. Where possible, we analyzed quantitative data to support findings that were derived from qualitative data. C. Data collection 1. Qualitative data This report draws on three types of qualitative data: (1) semi-structured, in-person interviews with key informants (KIIs) including staff from Bridges, some training centers, some large employers of project trainees, and representatives from USAID and the Salvadorian Vocational Training Institute (INSAFORP); (2) focus group discussions (FGDs) with instructors from TCs and with Bridges trainees (both current and former); and (3) a comprehensive desk review of Bridges monthly and semiannual reports and assessments or special studies. Key informant interviews. Mathematica conducted semi-structured interviews with key informants in two rounds of data collection. We picked late 2019 and early 2020 to balance the time needed to achieve project goals with the implementation of programmatic activities and be able to interview a mix of beneficiaries (former trainees with employment, former trainees without employment, and current trainees). In October 2019, we interviewed Bridges staff in charge of project activities for objectives 1, 2, and 3. We used purposive samples to ensure that our sample met the following criteria: (1) geographic diversity to ensure representation from both San Salvador and other rural areas; (2) varying economic sectors; and (for training centers only) (3) offering both technical and life skills. Based on these criteria, we ended up with more training centers and trainees than we needed for our sample, so we randomly selected which training centers to visit and which trainees to invite to our focus groups. The sample of TCs we visited offered training for youth in key economic sectors—commerce/tourism7 , energy, manufacturing and ICT, provided both technical and life skills training, and were from six different municipalities (Ahuachapán, Ilopango, San Salvador, Santa Ana, Sonzacate, Soyapango). We also interviewed coordinators from training centers that had completed courses. In January 2020, we conducted the second round of data collection. We conducted interviews with representatives of private firms that hired Bridges trainees. The criteria used to select firms was to include those that had hired the highest number of Bridges trainees, we also chose firms who recruited graduates from different training centers and represented different economic sectors. Once we identified the target firms, we interviewed human resources staff or managers of the firms. These firms represented three different sectors and five municipalities. We also interviewed graduates who were currently employed by the firms we were interviewing. We interviewed graduates from 8 of the 10 companies from which we collected data. In this round, we also interviewed staff from USAID/El Salvador and INSAFORP. In Table II.1, we provide details on the number of interviews conducted in both rounds of data collection. Focus group discussions. In October 2019, we held four focus groups with TC instructors to discuss the training being provided, their perceptions on trainees’ skills and potential, and 7 Both commerce and tourism are part of the service sector and refer to the trade of goods and services in the economy. Because this is the way Bridges referred to this sector, we opted to use the same nomenclature so it would match their project reporting. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 9 challenges TCs faced while training vulnerable youth. We also led six focus groups with approximately 48 trainees to discuss their experiences with, overall perception of, and satisfaction with the training program (including content and organization of the courses) and the support services they received from TCs. In January 2020, we conducted five focus group discussions with 33 former Bridges trainees or graduates in which they shared their perceptions on how Bridges training helped them in their job search, current employment, and general job performance. Table II.1. Data sources for this analysis Data source Sample size Municipalities visited Training focus Key topics discussed First round of data collection (October 2019) Bridges staff ● 2 phone interviews ● 1 in-person interview ● 1 focus group (approximately 15 staff) ● San Salvador ● n.a. ● Reflections on design and implementation ● Reflections on barriers and facilitators for the implementation of Bridges project Training center representatives ● 8 interviews ● Ahuachapán ● Ilopango ● San Salvador ● Santa Ana ● Sonzacate ● Soyapango ● 2 manufacturing ● 2 ICT ● 1 commerce/ tourism ● 1 life skills ● 2 formal education ● Reflections on implementation ● Perspectives on barriers and facilitators that directly affected their work and their capacity to partner with private businesses and adjust to a changing labor market ● Perspectives on sustainability Instructors ● 3 interviews ● 4 focus groups ● Ahuachapán ● Ilopango ● San Salvador ● Santa Ana ● Sonzacate ● Soyapango ● 2 manufacturing ● 2 ICT ● 1 commerce/ tourism ● 1 life skills ● 1 formal education ● Reflections on the training being provided ● Perceptions on trainees’ skill improvement and potential ● Challenges faced during the provision of training. Trainees ● 6 focus groups (48 trainees— 25 female and 23 male youth) ● Ahuachapán ● Ilopango ● San Salvador ● Santa Ana ● Sonzacate ● Soyapango ● 2 manufacturing ● 2 ICT ● 1 commerce/ tourism ● 1 life skills ● Experiences, overall perception, and satisfaction with the training ● Expectations of the benefits of the project in their job search and job performance FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 10 Data source Sample size Municipalities visited Training focus Key topics discussed Second round of data collection (January, March, May 2020) USAID/El Salvador staff ● 1 interview (three staff) ● San Salvador ● n.a. ● Reflections on barriers and facilitators for the implementation of Bridges project ● Perceived intended and unintended results of the project ● Perspectives on sustainability INSAFORP staff ● 1 interview (two staff) ● San Salvador ● n.a. ● Reflection on their experience working with Bridges updating or creating new curricula for training courses ● Perspectives on sustainability Employers ● 9 in-person interviews ● 1 phone interview ● La Libertad ● San Salvador ● Santa Ana ● Soyapango ● San Miguel* ● 5 Commerce/ tourism ● 4 manufacturing ● 1 ICT ● Reflections on outreach activities, partnerships between their firms and Bridges or training centers ● Perceived benefits from the project ● Assessment of job performance, technical knowledge and skills, and life skills of Bridges trainees, perhaps in comparison with non-Bridges workers Graduates ● 8 interviews—3 female and 5 male youth ● 5 focus groups –33 participants, 14 female and 19 male youth ● San Salvador ● Ahuachapán ● Santa Ana ● Sonsonate ● Commerce/ tourism ● Manufacturing ● ICT ● Perceptions on how Bridges training has helped them in their job search, current employment, and general job performance Bridges staff ● 2 phone interviews ● n.a. ● n.a. ● Implementation updates for the period January to March 2020 ● Perspectives on sustainability n.a. = not applicable. * Due to the distance between San Salvador and San Miguel (150 kms), the interview with the employer of San Miguel was conducted by telephone. Desk review. In addition to collecting primary qualitative data, we relied on a document review including Bridges’ monthly and semiannual reports, work plans, monitoring and evaluation (M&E) plans, contract modifications, project deliverables such as the four early assessments, and other documents critical to understanding Bridges’ implementation. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 11 2. Quantitative data Monitoring data from Bridges. As part of the capacity building activities, Bridges invested in the development of a data system called SisPuentes. This system has been used to track youth during the course of the training (to monitor enrollment, attendance and performance through pre- and post-tests) and after completion of training (to measure employment status), and to monitor results and progress towards targets. Collecting the data was the responsibility of each grantee and it was then verified by Bridges staff. We used this data to assess how activities are being rolled out, the types of services provided, the geographic reach and beneficiaries disaggregated by sex, and whether implementation targets are being met. We began collecting monitoring data in August 2019; the data was used to help develop protocols and to produce a purposive sample to interview implementers, youth beneficiaries, and firms. We received the last round of data in August 2020 to be able to include information from training programs carried out as of July 31, 2020, when all training courses were completed and to include in our analysis the information on project achievements through July 2020. D. Analysis plan The desk review, KIIs, and FGDs provided a wealth of information for qualitative analysis. We analyzed this information to identify patterns of consensus and instances of divergent or contradictory views. We used two primary methods of analyzing these qualitative data to address our questions: (1) conceptual categorization and (2) data triangulation. Conceptual categorization. To uncover patterns, themes, and issues in the qualitative data, we developed a coding framework to capture a hierarchy of conceptual categories and classifications that are linked to the evaluation questions. We developed a coding framework with conceptual categories linked to the three main objectives of the project and divided into the following categories: design, implementation, barriers and facilitators, perception of results and sustainability. Assigning codes to the qualitative data allowed us to efficiently access and organize information obtained through the KIIs and FGDs in order to identify themes and compile supporting evidence. Ivonne Padilla conducted the coding of the transcripts and Patricia Costa performed the quality assurance of the coding and analysis. We updated this coding framework based on new information as we systematically reviewed and assessed the qualitative data. Data triangulation. Our qualitative analysis sought to identify similarities and differences in perspectives across respondents, complemented by descriptive information from the monitoring data. We used data triangulation to test for consistency in the findings from these data sources. This process allowed us to confirm patterns or findings and identify important discrepancies. For example, we used the coded transcripts from our two rounds of data collection to triangulate responses across different respondent types and to develop a comprehensive understanding of the project’s potential effects by identifying instances in which qualitative findings corroborate, contradict, or help contextualize quantitative findings. General approach to quantitative analysis. In analyzing quantitative data provided by Bridges, we merged youth-level data sets with training and employment information to compare outcomes against targets. Merging the data set allowed us to more carefully track trainees’ progression from enrollment to completion, whether they had enrolled in more than one course, FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 12 and whether they had reported employment. We analyzed some of the project outcomes disaggregated by type of support for youth, sex, economic sector, and municipality. We also carry out additional analyses to understand the differences in employment outcomes based on type of training, length of training and economic sector. We used mixed-methods analysis for questions for which we have both quantitative and qualitative data. For example, we used the conceptual categorization of data from interviews and focus groups to identify key themes that will help contextualize the progress shown in administrative data and project reports. E. Evaluation limitations and steps taken to mitigate them When interpreting the findings from this study, there are several limitations. These have been noted throughout the text, but we summarize them here as well as the steps taken to mitigate them. a) The performance evaluation is not designed to detect project impacts, attribute changes in youth outcomes to the project, or determine which parts of the training programs led to which outcomes, because it lacks a suitable comparison group or counterfactual for service recipients. However, the performance evaluation is a powerful way to generate and share valuable learning and insights about the implementation and potential effects of the Bridges project as a whole. b) Qualitative findings are based on a purposive sample of stakeholders, and thus will not necessarily fully reflect the range of perspectives on Bridges. c) Since we collected data between 8 and 11 months before the end of the project, our sustainability analysis is based mainly on the review of monitoring data and a final interview with Bridges staff conducted in early May 2020. However, we included in the analysis perceptions on sustainability collected in October 2019. d) We are not able to assess long-term employment or job retention because of the lack of monitoring information and the timing of the evaluation. We carried out the following activities to mitigate against two of the limitations we raise above. a) We triangulated the data to have confidence in our findings. While triangulation cannot eliminate our concerns around lack of attribution, it allows us to confirm that different stakeholders have the same perceptions of results, providing added confidence in our findings. In addition, when we had quantitative data to be able to triangulate qualitative findings, we carried out additional analyses to support- or refute- the qualitative findings. b) While we could not interview all project stakeholders and beneficiaries, we designed our sample to maximize our assessment of the program. For example, when selecting the employers to interview, we looked at how many people they had hired in total as well as from which training institution they had recruited youth to obtain a broad perspective. Our selection criteria for which training centers to interview also allows us to get a broad perspective that should account for geographic or sector diversity. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 13 Table II.2. Evaluation questions Evaluation questions Quantitative Qualitative sources Answers in the report Monitoring data KII w/ firms KII Bridges staff/ USAID KII TC Director staff Desk review FGDs w/ instructors FGDs w/ trainees and graduates 1. What were key barriers/facilitators to the achievement of project results? (a) The three objectives are (1) enabling environment, (2) strengthened institutions, and (3) youth workforce readiness. Objectives 1 and 2 create curricula aligned with employer needs; objectives 2 and 3 work to provide youth with market-relevant skills; and objectives 1 and 3 work to create a demand for youth employees. All three objectives interact to increase youth employment. X X X Chapter III (b) How did the grants under contract (GUC) mechanism advance, or detain, implementation and the strengthening of local solutions? X X X Chapter III (c) How did the GUC address sustainability? X X X Chapter V 2. To what extent has Bridges reached outcome targets in the following areas: (a) Educational outcomes, improved skills, and employment, among youth X X X X Chapter III (b) Strengthening of TCs X X X X Chapter III (c) Number of new/revised certified programs X X X X Chapter III (d) Private sector inclusion of vulnerable youth in their staff X X X Chapter III (e) Sustainability of private sector firms’ and training centers’ relationship in order to have updated curricula and trainings X X Chapter V 3. What positive and negative results, intended or unintended, has Bridges produced at the end of the project? What are stakeholders’ perceptions about those results? What results produced by Bridges should be prioritized in future programs? Which ones seem to be the most sustainable? X X X X X Chapter IV FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 14 Evaluation questions Quantitative Qualitative sources Answers in the report Monitoring data KII w/ firms KII Bridges staff/ USAID KII TC Director staff Desk review FGDs w/ instructors FGDs w/ trainees and graduates 4. Perceptions from employers, trainees, and grantees: (a) What are employers’ perceptions about the following? i. Trainees’ technical skills and knowledge? ii. Trainees’ Life skills? iii. Trainees’ on-the-job performance for current position or potential for upward mobility? iv. Training centers and their improved curricula? v. Their willingness to hire youth from high-risk areas? X X X Chapter IV (b) What are Bridges’ trainees perceptions about the following? i. The training they received, including technical training and life skills training, when applicable? ii. How useful the training/certificate has been in helping them get a job? iii. How useful the support services were in getting a job and succeeding at it? iv. How Bridges has improved their future prospects in the labor market? v. Is there more openness from the private sector to hire youth (particularly vulnerable youth)? vi. What are the perceived before and after effects given the project’s intervention? X X X X Chapter IV (c) What are grantees’ perceptions about the following? i. Has engagement with the private sector made a difference in the education approach? ii. In what areas were your institutions most strengthened? iii. What approaches, technical assistance, and activities have been most critical for vulnerable youth job placement success? X X X Chapter IV 5. Additional questions to consider: How were gender equality, female empowerment, and social inclusion integrated in the implementation of activity interventions? Were those efforts successful? Why or why not? X X X X X Chapter IV FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 15 III. FINDINGS: IMPLEMENTATION AND RESULTS In this chapter, we present the main implementation findings and results of Bridges using information from reports, monitoring data, interviews, and focus groups with stakeholders. First, we summarize how the project was implemented to provide the necessary context to answer the evaluation questions. Second, we discussed if the project has been implemented according to its initial design. Third, we present our analysis of progress toward implementation goals. Fourth, we present an analysis of facilitators and barriers to implementation. For the purposes of readability, we reorganized the evaluation questions and they do not appear in the same order as in Table II.2. A. How was Bridges implemented? 1. Guiding principles of the project The Bridges scope of work outlined several overarching principles that were deemed critical for the success of the project. These included support for vulnerable youth; positive youth development; sustainability, scale, and systems development; private sector engagement; gender and social inclusion; USAID gender equality and female empowerment policy; lesbian, gay, bisexual, transgender, and intersex (LGBTI) vision for action; disability policy; and conflict sensitivity. Although all these principles were implemented, there are four principles that steered all Bridges activities. We refer to these as guiding principles and they include (a) focus project activities on selected municipalities, (b) target vulnerable youth, (c) prioritize selected high￾growth sectors, and (d) implement activities aiming to increase youth readiness through existing training centers, following the spirit of USAID’s Journey to Self-Reliance. Below we summarize each of these guiding principles. Underlying these principles was a focus on using data to assess progress, collaborating with USAID, the private sector and other stakeholders to explore changes, and adapting activities to ensure outcomes would be achieved. Together, these efforts represent the embodiment of USAID’s focus on Collaborating, Learning, and Adapting (CLA). a. Focus project activities on selected municipalities Bridges activities were concentrated in 15 priority municipalities of El Salvador, which were then expanded to 24. The original plan was for Bridges activities to be implemented in the 50 municipalities included in the Government of El Salvador’s Plan for a Safe El Salvador. However, USAID decided to focus interventions in a subset of municipalities where other USAID investments were taking place in order to expand and improve learning opportunities offered to youth and thereby increase Bridges’ rate of success. During the first year of implementation, Bridges and USAID worked closely to select priority municipalities through an initial assessment of the workforce development context. To select recipient municipalities, USAID and the Bridges team came up with the following inclusion criteria: 1. Included in El Salvador’s national security plan, Plan El Salvador Seguro (Plan for a Safe El Salvador) 2. Presence of other, relevant, USAID investments8 8 USAID’s other projects include Superate, SolucionES, Crime and Violence Prevention Program (CVPP), and Education for Children and Youth. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 16 3. A population greater than 50,000 4. Existence of a functioning local government9 5. Prioritized by Millennium Challenge Corporation (MCC)/Fondo del Milenio II (FOMILENIO II); specifically, those municipalities targeted for improvements in school infrastructure Based on this criteria, 15 initial municipalities were selected: Ahuachapán, Ciudad Arce, Ciudad Delgado, Cojutepeque, Colón, Jiquilisco, Mejicanos, San Salvador, San Miguel, Santa Ana, Santa Tecla, Sonsonate, Soyapango, Tonacatepeque, and Zacatecoluca. These are marked in dark blue in Figure III.1. However, as implementation evolved, Bridges in consultation with USAID included adjacent municipalities because some municipalities are very small and the geographical distinction between some municipalities has limited practical meaning. While some trainings did take place outside these municipalities in response to requests from training centers or private companies, Bridges only supported youth who live, study or work in the selected municipalities. Bridges held trainings in the following additional municipalities: Acajutla, Antiguo Cuscatlán, Chalchuapa, El Congo, Ilopango, Sacacoyo, San Juan Opico, San Martín, and Sonzacate (municipalities in light blue in the map below). We note that it is the youth (and not the training centers) who must fulfill the requirement to reside, study, or work in the selected municipalities. 9 Municipal Enterprise Development Unit (EMPRE) or Local Economic Development Unit (UDEL). FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 17 Figure III.1. Municipalities included in the Bridges project b. Target vulnerable youth Bridges targeted vulnerable youth only. Bridges targeted vulnerable youth, defined by the program as youth between the ages of 16 to 29 who live, study, or work in the targeted municipalities. Efforts of the project to constantly re-evaluate and adapt, led the project to offer support to returned migrants and youth who have a higher risk of migrating or being internally displaced in the selected high-crime municipalities. These vulnerable youth could receive technical and life skills training, or assistance in completing their high school education, to find or improve employment. c. Prioritize selected high-growth sectors Bridges targeted high-growth sectors in order to prioritize partnerships between training centers and employers in these sectors. The selection of high-growth sectors was based on an initial assessment conducted by the Organization for Promotion of Exports and Investment in El Salvador (PROESA) and the Ministry of Economy. In this study, based on the potential for growth and employment, the government of El Salvador identified the following sectors: manufacturing; food and beverage; textiles; plastics; information, communication, and technology (ICT); aeronautics; and tourism. Bridges conducted a follow-up study in the first year of implementation and narrowed the focus to agroindustry, commerce/tourism, renewable FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 18 energy,10 manufacturing,11 and ICT, based on growth potential, employment potential, and alignment with the skills, capabilities, and aspirations of the youth in high-risk areas. Bridges and USAID collaboratively decided to include textiles as a target sector in the third year of implementation because of the potential employment opportunities given El Salvador’s strong textile industry. Energy was added to create opportunities for women in this sector. d. Implement activities supporting training centers and youth through existing local institutions A grant under contract (GUC)12 program was designed as the main mechanism through which project activities would be implemented. Bridges’ implementation strategy was envisioned as a “whole system intervention” aiming to improve the coordination and capacity of the Salvadoran workforce environment to better prepare, engage, and employ vulnerable youth. To fulfill this approach, Bridges designed and implemented a grant under contract (GUC) program to support organizations in their efforts to improve workforce development initiatives for vulnerable youth. The objective of this grant fund was to enable participating training centers at both the national and local level to implement and sustain workforce development programming for vulnerable youth. USAID provided $12.5 million13 for GUCs (grants ranged from $100,000 to $2,000,000) to support local capacity building and improve the quality of workforce development services. Grants had the dual purpose of strengthening training centers’ systems, operations, and staff as well as funding technical and life skills training for vulnerable youth. Grantees received financial and institutional support to conduct activities that directly supported project objectives. In total, 26 grants were awarded; in Table A.2 in the appendix, we list the description of the grants, the main activities funded by the grants, the grantees, and the value of the grants approved. Through this mechanism, the grantees became the recipients of Objective 2 services and the providers of Objective 3 services. As summarized in section IV, the delay in the release of the grants manual delayed the rollout of training programs. In mid-2016, per USAID’s request, Bridges started the rollout of training activities through a blanket purchase agreement (BPA) where training centers were hired directly to start training programs. Bridges also directly implemented some activities, including providing training courses in 2018 and 2019. We referred to these activities throughout this document as Bridges activities. 10 The identified sector was renewable energy but most of the training courses focused on electricity; we will refer to this sector as the energy sector in the rest of the report. 11 Manufacturing includes plastics and textile sectors. 12 A GUC is a mechanism USAID uses to allow a contractor to issue grants with non-governmental organizations or governments. As part of Bridges, a grant fund was established to fund activities carried out by local organizations. 13 The amount of $12.5 million corresponds to the amount approved in the last contract modification in 2019; the original amount to fund GUCs was $17.5 million. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 19 2. Implementation timeline On September 30, 2015, USAID awarded the Bridges for Employment five-year contract to DAI. The implementation of core project activities can be divided into two phases. Phase I was largely dedicated to the design and development of four assessments: (1) labor market assessment, (2) policy assessment, (3) service provider assessment, and (4) participatory youth assessment. The main findings of these assessments were used to reinforce the design and ensure that project activities responded to the Salvadoran context. Phase II focused on the rollout of activities aiming to achieve the three objectives of the project. Activities to improve the workforce development environment started in the second quarter of 2016 with events aiming to raise awareness within the private sector about hiring practices, patterns of discrimination, stereotypes, and the legal framework for employing vulnerable youth. Activities focused on strengthening training centers started in late 2016 and were tied to the grants program. Activities related to youth training started in June 2016, but training programs offered through grantees started in late 2017. The implementation of the project was somewhat delayed because the rollout of the grants program—the primary mechanism by which Objective 2 and Objective 3 activities would be carried out—took more time than originally anticipated to launch. In December 2015, Bridges submitted the grants manual, and after several rounds of revisions, it was approved in December 2016. At the beginning of 2017, the grant solicitation and selection process started along with some grant-funded activities to strengthen workforce development providers. Finally, the rollout of training programs seeking to improve youth job readiness started in September 2017 (Figure III.2). Figure III.2. Bridges implementation timeline a The grants manual was submitted in December 2015 and after several rounds of revisions, was approved by USAID December 16, 2016. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 20 3. Implementation of the main activities under the three objectives Bridges used a comprehensive approach to achieve three main objectives. The activities to achieve these three objectives are inextricably linked because achievements in one objective increase the likelihood of success in the other objectives. Below, we present the main activities implemented under each of the three objectives. In some cases, activities can contribute to more than one objective. Activities seeking to improve the enabling environment promoting WFD of vulnerable youth (Objective 1) Bridges conducted a policy assessment to inform its strategy and shape its work to improve the enabling environment for youth in El Salvador. In 2016, Bridges conducted a policy assessment that mapped policies, laws, regulations and private sector practices that affect vulnerable youth employment. The assessment identified barriers in hiring practices that deter private companies from employing vulnerable youth, LGBTI population, and other vulnerable groups. Bridges used the recommendations from the assessment to design behavior change strategies and employment reform activities that would promote youth employment. Bridges addressed the legal barriers to youth employment by supporting legal reforms and raising awareness on existing regulations. On February 9, 2017, the Legislative Assembly reformed the Creating Incentives Within the Private Sector for Youth’s First Employment Law. The reform aims to incentivize the hiring of youth by increasing tax incentives offered to companies that provide youth with their first job. The policy assessment also identified a lack of knowledge about laws by youth and employers. To address this barrier, Bridges conducted workshops to raise awareness around the challenges that vulnerable groups face when searching for employment. In addition, Bridges and INJUVE assisted the municipal governments of Ciudad Delgado, Soyapango, Santa Ana, Mejicanos, Zacatecoluca, San Pedro Nonualco, and San Salvador to create or update municipal youth policies. The new (or updated) policies established legal frameworks for the rights of youth and responded to the employment-specific needs of youth in each community. Bridges worked to create a collaborative network with companies to link the technical training offered to youth with the needs of the labor market. Bridges fostered linkages between the training centers and the private sector to ensure that the technical training offered to vulnerable youth matched labor market needs. This collaborative network also helped Bridges facilitate job placement for graduates. The project focused its efforts to build partnerships with firms in prioritized sectors. Bridges developed the stakeholder advisory group to promote the participation of public and private sector stakeholders in the design and implementation of project activities. The goal of creating the stakeholder advisory group was to foster communication and collaboration among stakeholders, inform about Bridges activities, and promote the exchange of information and lessons learned, aiming to create shared benefits for all its members. Throughout the project, Bridges held quarterly meetings with representatives of training centers, youth organizations, private sector companies, and government institutions to discuss challenges in the workforce development system and together design activities that would address the challenges. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 21 Bridges launched a nationwide public communications campaign aiming to improve employers’ perceptions of youth and their capacity to contribute to the workforce. The goal of the campaign was to tackle commonly held stereotypes of youth from specific vulnerable groups (for example, youth living in municipalities with high rates of violence, youth with disabilities, young parents, returned youth migrants, and youth who identify as LGBTI). The campaign was influenced by information on perceptions of vulnerable youth by target audiences (training centers, government organizations, the private sector, and civil society) across the project’s 15 prioritized municipalities. During the rollout of the campaign, USAID and Bridges decided to refocus the campaign on encouraging employers to hire trained youth regardless of their race, color, religion, sex, sexual orientation, age, disability, or any other characteristic, instead of the original goal of changing public perceptions of vulnerable youth. Activities implemented to increase institutional strengthening to improve the quality of workforce development services (Objective 2) Activities under Objective 2 strengthened three types of local organizations: (1) CSOs that offered support (training, job placement, and programs in health and education, among others) to vulnerable youth and marginalized populations in El Salvador; (2) training providers who had a contractual obligation to train youth under the Bridges GUC contract; and (3) government institutions (INSAFORP, Ministry of Education, Science and Technology [MINEDUCYT], and INJUVE) to help them improve the relevance of training by updating existing curricula. Bridges strengthened organizations’ capacity to improve the quality of support offered to youth. Bridges used two primary mechanisms to strengthen organizations: (1) applying the Organizational Capacity Assessment (OCA) and the Technical and Vocational Education and Training Capacity Assessment Tool (TVET CAT) diagnosis and offering training and workshops to address weaknesses identified (such as monitoring and evaluation, communications and branding, and fundraising and sustainability) and (2) providing technical assistance to improve curriculum, capacity building of instructors, equipment, improved facilities, support in their relationship with the private sector, support in job management. Bridges conducted a workforce development service provider assessment that identified the needs of training centers in order to offer high quality services to youth. In 2018, to identify capacity-building needs, Bridges assessed the capacity of 20 training centers. The assessment found that the technical training courses offered at training centers did not match the needs of the labor market. The assessment also found that TCs needed to build operational and institutional capacity to manage activities and improve their facilities. The findings of the service provider assessment shaped the activities aimed to improve the quality and relevance of training available to vulnerable youth. Bridges provided three types of strengthening interventions to TCs: (1) development of new or improved training courses, (2) training of staff and trainers, and (3) physical infrastructure upgrades. Below, we provide a summary of each of the types of interventions. Among employers there is a fear of unknowingly hiring a gang member, bringing risks, such as extortion or other damage, to the company —Finding from the policy assessment FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 22 (1) Improve the content and relevance of training offered to vulnerable youth ● Bridges facilitated the development of new or improved curricula in the ICT sector. In 2018, Bridges collaborated with the Salvadoran Information Technology Chamber (CasaTIC) to develop new training curricula on six different computer programming languages. ● Bridges developed curricula recognized by INSAFORP in the textile and energy sectors. In 2018 and 2019, Bridges and INSAFORP collaborated with TCs, companies, industry associations, and certifying agencies to update existing training curricula to align with current and future needs of the textile and energy industries. All three updated curricula were recognized by INSAFORP. This recognition is important to employers because it is a validation of the training content. Table A.3 in the appendix presents a summary of the industry-recognized certifications and accreditations developed. ● Bridges added new modules to the existing INJUVE life skills curriculum and incorporated this updated curriculum into the training courses. Bridges also worked to expand the life skills curriculum, adding two new modules on a) self-control and b) critical, creative, and innovative thinking skills, to align its existing curriculum. Developed by INJUVE, this updated curriculum included the themes outlined in USAID’s 2016 Youth Power Action: Key Soft Skills for Cross-Sectoral Youth Outcomes (Gates et al. 2016). (2) Build capacity of training centers’ instructors and management staff ● In the formal education sector, Bridges trained tutors from grantees offering flexible education. Bridges supported the certification of 130 tutors in the Ministry of Education, Science and Technology’s (MINEDUCYT) Educational Strategy of Academic Leveling Tutoring after attending a training conducted by Bridges and MINEDUCYT. Out of the 163 certified tutors, 47 worked in the flexible education program led by the Central American Innovative Education Foundation (FIECA) and Fe y Alegría (FyA). The rest of the certified tutors joined MINEDUCYT staff to increase its capacity to assist more youth across the country complete their high school education. ● Bridges organized workshops to build staff capacity. Bridges conducted a series of workshops to train staff at training centers. The workshops focused on the following topics: (1) Bridges’ methodology for job placement of vulnerable youth, (2) use of pre-post assessments to vet youth knowledge and skills prior to joining the program as well as upon completion, and (3) competency-based approach to education. ● Bridges strengthened the capacity of training centers to offer job placement and readiness support to youth. The project offered training and materials to staff from seven TCs where C-Orienta One-Stop Shops opened, four in San Salvador, one in San Miguel, one in Santa Ana, and one in Sonzacate. In these centers, youth received vocational orientation and information about training, educational, and job opportunities. C-Orienta also provided job readiness coaching including resume preparation and mock interviews. ● Bridges trained job placement managers. Bridges organized workshops with job placement managers within the training centers to build their capacity through collaborative learning. During the workshops, job placement managers developed a standardized curriculum for labor orientation services and a standardized process for sharing job opportunities with youth and reporting these activities. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 23 (3) Improve training center facilities ● Through grants, Bridges funded upgrades to training center facilities and equipment. Bridges provided equipment and materials, such as computer and laboratory equipment, didactic resources, and software to training centers offering professional training courses. To strengthen technical public high schools, Bridges improved the facilities of five education centers and provided equipment, tools, and accessories for laboratories in institutes linked to the Gradual Educational Model of Technical and Technological Learning (MEGATEC) and in the headquarters. ● Bridges built grantees’ data management and reporting capacity. Bridges provided support and technical assistance to adopt a customized data management system, SisPuentes, which Bridges also used to report data to USAID. This platform allows training centers to monitor youth beneficiaries (registration, attendance, test scores); track their employment status after they graduate; and evaluate progress toward project indicators. In addition to the strengthening activities mentioned above, Bridges staff dedicated significant time and resources to help TCs manage their grants under contracts. Although these activities do not fall under specific work-plan designated tasks, Bridges staff helped TC staff vastly improve the administrative and financial management processes to be able to secure USAID grant funding, which is significantly more challenging to manage than other donor funding. Based on conversations with Bridges staff and USAID, this aspect of trainee strengthening was likely the most challenging given the limited capacity of grantees as well as the varying needs (administrative, operational, financial). Activities implemented to improve workforce readiness demonstrated by targeted vulnerable youth (Objective 3) Bridges conducted the participatory youth assessment to identify needs and barriers for youth to improve their skills and prospects for employment. The objective of the participatory youth assessment was to understand the context and environment where vulnerable youth develop; identifying their principal needs and the challenges they face in improving their education and prospects for employment. The main recommendations from the assessment were to assist youth in completing secondary education, offer training relevant to the needs of the private sector, facilitate access of vulnerable youth into the labor market with activities seeking to affect hiring practices of the private sector, and improve the job readiness of youth. Bridges provided youth four different pathways to employability based on youth background, interest and geographic availability of the services. Bridges provided services with the aim of increasing or improving the employability of vulnerable youth. The project identified youth needs and offered them a pathway to improve their education (if needed) or their prospects for employment. Once youth were identified, they went through the selection process where their needs and preferences were assessed. When this process was completed, youth had 4 pathways to improve their employability as explained below: It’s been a challenge to get (training centers) in order (financially and administratively); to get documentation they need to be submitted and processed, it’s a long and hand-holding process. –Bridges staff FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 24 1. Activities in the professional training sector.14 Bridges worked to improve the job readiness and employability of vulnerable youth by offering the following training programs: (a) Technical (including theoretical and hands-on) training developed to satisfy the technical needs of employers in the prioritized sectors. The duration of these trainings ranged between 1 and 3 months, although there were several exceptions. One training course in plastics had a duration of 10 months and ICT courses lasted approximately 9 months, each course with three distinct levels which lasted three months. (b) Life skills training taught with the curriculum developed by INJUVE. The content of this training includes self-esteem, teamwork, and proactivity. The duration of this training is at least 64 hours, and the content is taught by certified instructors. (c) Technical and life skills training offered courses with the curricula from the technical training and the 64-hour life skills training module developed by INJUVE. The duration and content of the technical component in these trainings varied by sector. In the commerce/tourism or textile sector, the duration was one month and focused on how to operate textile industrial machines; in the plastics sector, some courses included modules for mathematics, chemistry, and basic English in addition to the technical training on the use of machinery. These courses also included three months of apprenticeship programs in plastics companies and can last up to one year. (d) Labor orientation and vocational orientation through C-Orienta. Youth could participate in a subset of life skills training curriculum focused on job counseling. This included mock interviews, resume preparation, tips for job search, and registration in labor platforms. This training can be taught by instructors from TCs, and last at least four hours. Bridges started implementing training courses in 2016 with the rollout of programs focused on technical training. For the most part, youth could select one of the options above, though 1 percent of youth participated in more than one program. In the second year of project implementation, two grantees offered training exclusively in life skills. However, a module of 64 hours focused on life skills was added in 2018 to most of the technical training courses. By the end of 2019, labor orientation services were also offered to vulnerable youth, and the duration of this service was at least four hours. A total of 643 training courses were offered to youth as depicted in Figure III.3. 14 Professional training refers to vocational training programs, these programs are regulated by INSAFORP. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 25 Figure III.3. Professional training courses offered by Bridges Youth who participated in professional training courses received different types of labor orientation support and training. Youth trained in the first year of the project received training in technical skills or life skills. In contrast, those in 2018 (and beyond) who received technical training also received life skills. Finally, some youth only participated in a workshop on labor orientation. In 2018, job placement services became more systematized. 2. Activities to enter the labor market through direct labor intermediation (DLI). In September 2018, results for training and employment were not progressing at the anticipated pace. Bridges decided to strengthen labor intermediation to facilitate youth job placement. DLI consisted of offering youth tips to improve their resume and perform well in the interviews as well as enrolling them in job placement platforms, sending them to interviews, and following up with them. 3. Entrepreneur support services. During the initial assessment, if a youth had the interest and skills to be an entrepreneur, he or she received information about institutions that provide support to start a business. The project also connected youth to networks of entrepreneurs based on area of interest. Only a small subset of youth participated in these types of activities because they were not the primary focus of Bridges and a parallel USAID activity focused on entrepreneurship existed. 4. Activities to continue formal education.15 Activities to continue formal education addressed one key barrier to employment for youth, drop out before completion of high school. Bridges provided two grants in response to identified needs. Table III.1 presents a summary of changes in the implementation. A more detailed discussion of each of these changes follows. 15 Formal education refers to formal programs under the regulation of the Ministry of Education, such as secondary and postsecondary education. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 26 B. Was Bridges implemented as planned? The Bridges project has been implemented according to its initial design, but thanks to an effort to incorporate CLA, some activities were included or modified. As described in Section A, implementation largely followed the original design. However, shifts in the implementation were made and activities were added or modified in response to identified needs. Table III.1 presents a summary of changes in the implementation. A more detailed discussion of each of these changes follows. Table III.1. Changes in Bridges’ implementation as a result of CLA Main changes in implementation Rationale Implementation results Bridges assisted youth with completing their high school education through a flexible education program. Offer youth opportunities to complete secondary education and improve the likelihood for them to secure a job. A total of 608 youth completed secondary education through the flexible education program. Bridges included a module of 64 hours of life skills training to technical training courses. Provide, in addition to technical skills, life skills that youth need to secure a job and succeed at it. A total of 7,644 youth completed technical training that included the life skills component. Bridges provided as part of their comprehensive support to vulnerable youth help navigating the job search and interview processes. Offer wrap-around services to facilitate the entry of vulnerable youth into the workforce. A total of 933 youth received wrap-around services through grantees MyV and UEES to facilitate job placement. Bridges offered DLI to youth. Offer support with a different approach: grantees guide youth to find a job based on their experience and skills without passing formal training. A total of 809 youth got a job through DLI. MyV = Goals and Vision; UEES = Evangelical University of El Salvador; DLI = Direct Labor intermediation. Bridges offered vulnerable youth the opportunity to complete their high school education through MINEDUCYT’s flexible education program. Bridges identified that in some regions a barrier to achieve employment goals was that youth dropped out of school before completing high school. They also identified that a high school diploma is a requirement for most employers. To address this barrier, the project worked closely with MINEDUCYT and grantees FIECA and FyA to assist vulnerable youth in completing their high school education through the flexible education programs. As part of this strategy, Bridges worked to address the need identified by the Ministry of Education to certify teachers in flexible modalities. At the end of the program, 608 trainees completed high school education through the flexible education program. Bridges incorporated life skills training into the technical training courses in the third year of the project. Bridges incorporated life skills curricula into technical training to build critical competencies demanded by employers through life skills and on-the-job training. In interviews, employers expressed that they valued hiring youth trained with life skills and they perceived a difference in the performance of these youth (presumably because of this training). According to employers, these youth are more We can always teach youth how to operate machines or learn techniques, but if we know that they developed interpersonal skills, that is an advantage for us. —Representative of the private sector FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 27 confident and relate to others better in the workplace. Bridges integrated life skills instruction into all of its technical courses in the third year of implementation. In the third year of project implementation, Bridges systematized the support for youth to navigate the job search and interview process. The project incorporated job readiness support for youth because staff identified that youth needed as much support to navigate the job search and interview process as they needed training. In 2018, Bridges incorporated “wrap-around services” for vulnerable youth that included a combination of life skills training focused on labor orientation, coaching and mentoring, and job placement support. The project also offered DLI, where grantees assessed youth skills and experience, and helped youth navigate the job search process without having to complete a particular technical training course. DLI support included only key features of the labor orientation training: youth received resume preparation, tips to apply for jobs, and advice on performing well in interviews. Bridges provided ad hoc training materials on psychosocial support, entrepreneurship, life skills and job readiness for WFD service providers. Bridges provided training centers with material to improve the quality of the services offered to youth. The material included toolkits (to teach psychosocial, entrepreneurship, and life skills integrated to technical training) and new modules to be included in the curricula for life skills training: Self-control and Creative. Bridges was implemented according to its timeline, but with some delays. The project had some delays in the rollout of activities. As shown in Table III.3, implementation targets for activities aiming to improve the WFD environment—for example, the number of firms implementing changes or regulation adopted to improve WFD—were not met until the second year of program implementation (fiscal year [FY] 2017). The delays to achieve annual targets in this objective are due to the time it took to identify practices or regulations that needed to be improved and the advocacy efforts needed so firms and local authorities could support the recommended changes to their practices. Most of the activities to strengthen training centers (Objective 2) rolled out as planned—namely, implementation goals related to strengthening training providers were achieved in the first year of project implementation. For activities under Objective 3, the implementation of the project was delayed as a result of the delay in the approval of the grants manual and the time needed to build local capacity among grantees. Delays in the enrollment of youth in training courses in the early years affected Bridges’ ability to meet other related indicators to improve youth workforce readiness, such as youth completing training, youth with improved skills, and youth with new or better employment in the first three years. However, as we will explore further, all targets were met or exceeded in the last year of project implementation (FY2020). Despite the shortfall in meeting targets in the first three years of the project, Bridges exceeded most of the Life of the Project (LOP) targets. The only two targets where the program fell short were the percentage of females in training programs and percentage of youth with employment. In both cases we consider this shortfall to be negligible. The percentage of female participants in training programs fell short (47.8 percent compared to the 50 percent LOP target) because it was challenging to recruit female participants in sectors like energy and plastics. However, the project made an important contribution by training female youth in industries perceived as male dominated (see Chapter IV Section A for additional information on Bridges’ contributions in these efforts). By March 2020, Bridges had met its target of 40% of FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 28 youth with new or better employment following completion of WFD programs. However, during the April to July period, approximately 400 youth completed their training program, but due to COVID 19 and the subsequent mobility restrictions, the job search process was halted. As result, the job target dropped to 39.3% and is now 0.7 percentage points below its target as of July 2020. Table III.2. Implementation targets and goals achieved by year Goal met for each fiscal year FY2016 FY2017 FY2018 FY2019 FY2020 Achieved July 2020 Target Goal met? Objective 1: Enabling environment promoting WFD of vulnerable youth Number of firms implementing changes or new practices No Yes No Yes Yes 56 50 Yes Number of laws, policies, or procedures proposed or adopted to improve WFD n.a. Yes Yes Yes Yes 36 12 Yes Value leveraged from the private sector and other donors to contribute to preparing training vulnerable youth for employment No No No No Yes 8.4 million 5 million Yes Objective 2: Increase institutional strengthening to improve the quality of WFD services Number of direct partnerships between TCs and private sector companies Yes Yes No Yes Yes 107 40 Yes Number of organizations and/or service delivery systems that serve vulnerable populations strengthened Yes Yes No Yes Yes 43 40 Yes Workforce development service providers or TCs strengthened Yes Yes Yes Yes Yes 29 21 Yes Number of industry-recognized certifications and/or accreditations developed No Yes No Yes Yes 24 24 Yes Objective 3: Improve workforce readiness demonstrated by targeted vulnerable youth Number of individuals who completed workforce development programs No No No No Yes 11,967 10,000* Yes Number of individuals with improved knowledge or skills following completion of WFD programs No No No No Yes 11,187 8,500* Yes Number of individuals with new or better employment following completion of WFD program No No No No Yes 4,708 4,000 Yes Number of previously out-of-school participants who report enrolling in formal school No Yes Yes Yes Yes 9,775 7,002* Yes Percentage of individuals with new or better employment following completion of WFD programs No Yes No No No 39.3% 40% Noa Percentage of female participants in Bridges programs designed to increase access to productive economic resources Yes No No No No 47.8% 50% No Source: Selected indicators from Bridges results indicators. Notes: Goals with * are updated targets from contract modification 6 and 9. FY2020* data included in this report cover October 1, 2019, through July 31, 2020. a See above the table the explanation about why the percentage of youth with a new or better job that fell short. n.a. = not applicable. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 29 Political changes in the United States resulting in policy changes and funding levels affected the implementation of the project. In an effort to further align the project with the priorities of the United States government administration, Bridges and USAID decided to put more emphasis on employment to deter illegal migration. In the third year of project implementation (FY2018), USAID adjusted the targets to reflect the focus on employment generated by the project. In this revision, the enrollment target was set at 16,000 youth, and targets for youth who completed training and improved their knowledge or skills were lowered. However, the employment target increased from 5,000 to 6,000. During fiscal year 2019, the U.S. government changed its priorities resulting in funding cuts to the Central American region in response to the prevalence of migration to the United States. The project was advised in late June 2019 that USAID did not expect to obligate any additional funds to the project, and a formal proposal and modified work plan were submitted to USAID thereafter. These changes resulted in an approximately $8 million (which represents 20% of the original contract amount) reduction in the award ceiling. Bridges developed a plan for the project to maintain operations but made some adjustments in Objective 3 activities and targets. In Table III.3, we depict the change in targets due to the focus on employment in FY2018 as well as funding cuts in FY2019. In addition to the funding cuts and the subsequent revisions of indicators, the shift in U.S. government priorities hurt Bridges in that it lost its momentum and injected a lot of uncertainty for staff and training centers. Table III.3. Implementation targets updated Original targets Updated targets FY2018 Updated targets FY2019 Number of vulnerable youth enrolled in Bridges workforce development services n.a. 16,000 11,000 Number of individuals who completed Bridges workforce development programs 20,000 14,400 10,000 Number of youth traineda in social or leadership skills through Bridges project n.a. 13,600 9,500 Number of graduates from Bridges programs with improved knowledge or skills 20,000 12,240 8,500 Number of graduates from Bridges programs with new or better employment 5,000 6,000 4,000 Percentage of graduates from Bridges programs with new or better employment 25% 41.7% 40% Source: Bridges monitoring data. a Youth trained refers to youth who completed training. n.a.= not applicable. Bridges’ original design aimed to address the underlying factors driving migration; however, the increased spotlight on migration resulted in greater focus on supporting returned migrants and youth at risk of migrating. Based on some CLA meetings with the mission, the project refocused some of the tasks aiming to create opportunities for returned migrants in El Salvador. Bridges worked with the Center for Integral Attention to Migrants The implementation was going full steam ahead and the announcement of the funding cut felt like an earthquake; we stopped some activities due to the uncertainty and then we decreased our training and employment goals. —Coordinator from a training center FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 30 (CAIM) to provide returned youth migrants with information about the education and training opportunities that they could access through Bridges and its partner TCs. Bridges also worked with TCs to identify the number of returned migrants receiving support from the project and recruited returned youth migrants to receive assistance from the project. Bridges collaborated with USAID Monitoring, Evaluation, and Learning Initiative (USAID/El Salvador’s M&E contract) to assess how many youth participating in Bridges training courses had dropped out to migrate in FY2019. The main finding of the study was that only 2 percent of the youth who dropped out of training programs emigrated from El Salvador (5 out 228 participants). Most of the youth who dropped out of the program returned to school or found a job, which is a positive result. Bridges also included information about the risks of migration in the module The Choices We Make that is part of the material used for life skills training. C. To what extent has Bridges reached outcome targets? Objective 1: Enabling environment promoting WFD of vulnerable youth 1. Private sector inclusion of vulnerable youth in their staff A total of 1,205 firms had hired youth who completed Bridges training courses. The top 10 employers of Bridges trainees were in the commerce/tourism, manufacturing—which includes plastics and textile—and ICT sectors. Top ten employers (referring to the firms that hired the most Bridges trainees) employed approximately 21 percent of Bridges graduates. Among these firms are Grupo Monge, Intradesa, and Central American Software Services. Table III.4 shows the number of firms disaggregated by the number of Bridges trainees hired. The development of close relationships with firms has been an important activity to improve the inclusion of vulnerable youth in the private sector. However, this private sector engagement comes with its challenges and effort given that more than 1,000 different firms employed Bridges trainees and 88% of the firms who hired trainees, hired less than five youth. Table III.4. Number of firms, disaggregated by the number of Bridges trainees hired Number of trainees hired: Number of private firms More than 100 2 Between 50 and 99 11 Between 20 and 49 35 Between 10 and 19 34 Between 6 and 9 51 Less than 5 975 Total 1,108 Source: Bridges monitoring data. Bridges trainees were also hired by 46 public institutions, and 50 nongovernmental organizations. The project supported 43 youth to develop entrepreneurships. In total, 1,247 public and private institutions hired vulnerable youth. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 31 Objective 2: Increase institutional strengthening to improve the quality of WFD services 2. Strengthening of training centers Bridges surpassed its goal to strengthen 21 training centers in June 2019. The project met strengthening targets throughout fiscal years 2016, 2017, and 2018. The goal of this indicator was adjusted from 20 to 21 to reflect the number of training centers supported through the 26 grants in November 2019. In Figure III.4, we depict progress in each project year, the cumulative progress from previous years, and the achievement against the LOP target. Figure III.4. Number of training centers strengthened Source: Bridges monitoring data. Note: FY2020* data included in this report cover October 1, 2019, through July 31, 2020. Bridges exceeded its goal related to enrollment of training centers, nongovernmental organizations (NGOs), CSOs, and public/private sector staff in training. By June 2019, the project had met its target for staff enrolled in training, with 1,418 employees enrolled. A total of 1,952 employees had enrolled in Bridges training (Figure III.5). FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 32 Figure III.5. Number of training center, NGO, CSO, and private and public sector staff enrolled in Bridges training Source: Bridges monitoring data. Note: FY2020* data included in this report cover October 1, 2019, through July 31, 2020. To what extent has Bridges reached outcome targets in new/revised certified programs? Bridges surpassed its target for industry-recognized certification programs and/or TCs accredited. A certified program provides assurance to the private sector that graduates have acquired skills relevant to the industry. Accreditation signals to the private sector that training centers have institutional capacity and educational quality needed to serve clients. INSAFORP (as coordinator of the National Vocational Training System) accredits TCs after assessing whether they have the facilities, instructors, and curricula to offer training programs. In the first two project years, Bridges worked with the industries and the training centers to certify curricula and accredit TCs. Initially, Bridges expected to have 12 certifications and/or accreditations by the end of the project, but given its achievements in earlier years, in 2018 this target was increased to 24. This increase allowed for Bridges to meet its requirement of certifying programs in three different sectors. • In August 2018, a total of 18 industry-recognized new curricula for computer programming certifications were developed in the ICT sector. These new programs were recognized by CasaTIC to ensure they are aligned with the current and future needs of the ICT industry. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 33 • By late 2019, the curricula for bartender-waiter training program was updated and recognized by Distribuidora Salvadoreña (DISAL16). The Centro de Formación Servicio Social Pasionista was accredited as a training center and recognized by INSAFORP. • In May 2019, the curricula for two training programs— Operation of Industrial Apparel Machines and Category 4 Electricians—were updated, and in August 2019, the curriculum for the Maintenance and Repair of Industrial Apparel Machines training program was updated. These last updated programs are recognized by INSAFORP. • By April 2020, the plastics industry association (ASIPLASTIC) was reviewing the curriculum; once approved, it will be implemented in the plastics sector. In the formal education sector, Bridges worked with ESFE-AGAPE to update the curriculum for the commerce/tourism program, and MINEDUCYT approved it in August 2020. In Table III.5, we summarize the progress made in certification and accreditation achieved with Bridges support. • Bridges completed certification for 24 programs and the accreditation of one training center. Bridges has worked with FUNDEPLAST to update the curriculum for a training course in the plastics sector—post-industrial recycling. Table III.5. Progress in certification and accreditation Annual target Progress made in the period Target met? FY2016 0 0 No FY2017 2 0 No FY2018 4 18 Yes FY2019 2a 2 Yes FY2020 4a 4 Yes Life of project target 24 24 Yes Source: Bridges monitoring data. Note: The original LOP target increased from 12 to 24 to comply with the requirement to develop certified programs in three different sectors. a Original LOP target was achieved in FY2018. However, six more certifications or accreditations were included in fiscal years 2019 and 2020 because USAID requested to develop them in three different sectors. 16 DISAL is a beverage distribution company. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 34 Objective 3: Improve workforce readiness demonstrated by targeted vulnerable 3. Educational outcomes, improved skills, and employment among youth Because the project offered competency-based training to improve youth skills and their possibilities of getting new or better employment, Bridges set outcomes targets to monitor and measure results. In this section, we present results for four main outcomes for youth: (a) Enrollment in technical and/or life skills training. The indicator measures the number of youth enrolled in technical and/or life skill courses. Youth could participate in more than one training but were only counted once. (b) Completion of training programs. Completion of training is a binary indicator based on the completion of a training program. The project grantee or partner is responsible for defining the criteria for completion of a training program. (c) Benefit from training and improved skills (technical and/or life skills). Improvement in skills is measured as a higher score or better results in a pre-post assessment conducted by the TCs. The pre-post assessment evaluates the presence of life skills, vocational skills, and/or technical skills. (d) New or better employment. New employment is based on a change of status from not employed to being employed; better employment is based on perception of hired youth and takes into account not only salary but working conditions. Appendix figures A.1 to A.7 present additional targets for the 1) number of indirect vulnerable people benefiting from the project, 2) number of people benefiting from the project, 3) number of youth trained in social or leadership skills through Bridges training, 4) number of youth with improved knowledge or skills following completion of training, 5) number of previously out-of-school participants who report enrolling in formal school, 6) percent of individuals with new or better employment following completion, and 7) percentage of female participants in Bridges training, and the goals achieved in each fiscal year. The Bridges project met, and exceeded, major project milestones in training and employment among youth despite some delays in the rollout of activities. Bridges achieved two major youth readiness milestones: 11,967 youth completed training and 4,708 secured a job. This is an impressive achievement which was reached months before the end of project despite the delays in program rollout and falling short of targets in the first two years. Bridges met its goal for students enrolled in training programs. We cannot assess the extent of enrollment targets achieved in each year of implementation because Bridges started to report this indicator in the second year of implementation, and annual targets were set only for the last two years of project implementation. However, Bridges surpassed its enrollment goal of 11,000 in the fifth project year, a total of 13,535 youth had enrolled in training programs (Figure III.6). FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 35 Figure III.6. Number of vulnerable youth enrolled in training courses Source: Bridges monitoring data. Note: FY2020* data included in this report cover October 1, 2019, through July 31, 2020. N/A = Not available because Bridges project started to report on this indicator in FY2017. Bridges met, and exceeded, its goal for youth who completed training courses. Although the project met its target for youth who completed training courses in late 2019, the project fell short of meeting targets in the first three fiscal years (2016, 2017, and 2018). The lower number of youth who completed the training programs in the first three project years reflects the low enrollment numbers in the first years of the project. The goal for this indicator was adjusted from 14,400 to 10,000 as result of the funding cut in 2019. A total of 11,967 youth completed grantee￾funded training programs (Figure III.7). FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 36 Figure III.7. Number of vulnerable youth who completed training courses Source: Bridges monitoring data. Note: FY2020* data included in this report cover October 1, 2019, through March 31, 2020. Bridges surpassed its goal for the number of youth with improved knowledge or skills, but in the first three project years, fewer youth than anticipated improved their skills. Bridges met its target for youth with improved knowledge or skills (based on an assessment conducted by the training centers) after the completion of the training programs in the last year of implementation (FY2020). The goal of this indicator was adjusted from 12,240 to 8,500 as result of the funding cut in 2019 (Figure III.8). FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 37 Figure III.8. Number of youth with improved knowledge or skills following completion of training Source: Bridges monitoring data. Note: FY2020* data included in this report cover October 1, 2019, through July 31, 2020. In January 2020, Bridges achieved its employment target for vulnerable youth. Little progress was made in employment targets during the first three years of project implementation (Figure III.9). Fewer youth employed than anticipated in the first project years reflects the delays in the implementation of training and the low enrollment in those years. Some of the low employment numbers could also reflect the lack of support to find jobs (such as wrap-around services and labor intermediation). However, in the fourth year of project implementation (FY2019), 2,117 youth had found new or better employment. Bridges met its employment target with 4,022 youth with new or better employment in January 2020. A total of 4,397 trainees got a new job and 311 had a better job. It is important to note that approximately 17 percent of youth with new employment did not receive technical training but rather received DLI. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 38 Figure III.9. Number of trainees with new or better employment following completion of training Source: Bridges monitoring data. Notes: FY2020* covers the period of October 2019 to July 2020. The indicator defined by Bridges refers to the number of youth with new or better employment following completion of workforce development program. However, we identified 123 youth who secured a job but did not complete the training course. Implications of training and employment findings The percentage of youth who completed their training varied by implementing grantee or sector. In Figure III.10, we show the number of youth enrolled and who completed training17 programs by grantee. By July 2020, all training courses were completed. Completion rates were as high as 96 percent for the training conducted by TCs authorized by INSAFORP to develop courses for at-risk youth and the life skills training, and as low as 76 percent for the flexible education program. 17 In the rest of the report, we will refer to youth who completed training as youth trained. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 39 Figure III.10. Enrollment and completion by grantee Source: Bridges monitoring data. Note: The numbers on the left refer to the grantee number associated with the training. For grantees technical training in 4 sectors (manufacturing, which includes plastics, commerce/tourism, energy, and ICT sector). Bridges has higher completion rates compared to similar programs. Overall, the completion rate for trainees who participate in Bridges training was 88 percent. This rate is higher compared to similar interventions. A study reviewing 12 evaluations of vocational training programs from 8 countries (Turkey, Argentina, Colombia, Dominican Republic, India, Kenya, Malawi, Peru) found completion rates of training courses ranging 70 to 85 percent (McKenzie, 2017). Another USAID-funded vocational training program “A Ganar” implemented in Central America, including in El Salvador, had completion rates of 72 percent. It is important to note that comparison between completion rates is difficult given the differences in the nature of the programs and incentives. Another challenge to compare completion rates is that not all programs have the same requirements for completion. The percentage of youth who completed their training and got a job varied by implementing grantee. In Figure III.11, we present the number of youth with employment after completing the grant-funded training course. Grantees offering trainings in manufacturing (plastics and textile sectors) have the highest job placement rate;18 approximately 73 percent of youth who completed plastics training and 53 percent of youth who completed training in textile and apparel got jobs. However, in the plastics sector most of the youth had a better job (as opposed to a new job) after completing the training, implying that many of these trainees were likely employed before. For most of the grantees, the job placement rate is close to 30 percent. Training implemented by Bridges, life skills training and flexible program education, have the 18 We are measuring job placement rate as the percentage of youth who completed a training program and reported a new or better job. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 40 lowest job placement rates, 17, 11, and 11 percent, respectively. Grantees implementing life skills and flexible education programs were not required to place youth in jobs. The lack of this requirement may contribute to the low placement rates. We do not have data to understand if youth who participated in these trainings continued their education or were unemployed. As stated above, 809 youth with new or better employment received DLI from Bridges. Figure III.11. Completion of training and employment by grantee Source: Bridges monitoring data. Note: For DLI, we do not have the number of youth who received the service, only the number of youth employed who received the service. The numbers on the left correspond to the numbers assigned to each grant agreement. More youth who completed technical and life skills training compared to other types of support found a new or better job. Out of the total youth who completed technical and life skills training, 40 percent found a new or better job (Figure III.12). For youth who completed training in life skills only, 19 percent found a new or better job. Approximately 34 percent of youth trained in technical skills only found a job, and 18 percent of youth who participate in labor orientation through C-Orienta found a job. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 41 Figure III.12. New or better job by type of training for youth trained Source: Bridges monitoring data. Note: The total of youth who received DLI services (809) includes 23 youth who also completed Bridges training. Completion and employment outcomes varied by sector and sex. The graph on the left part of Figure III.13 shows the distribution of youth who completed training, organized by key sector and disaggregated by sex. Out of the 11,967 youth trained, more than half participated in a program related to the commerce/tourism sector (7,278), followed by participation in ICT (2,079) and manufacturing (2,183). The graph on the right part of Figure III.14, presents the distribution of employed youth by key sector, disaggregated by sex. The largest proportion of youth with new or better jobs were employed by the commerce/tourism sector (2,441), followed by the manufacturing (1,507) and ICT (605) sectors. Even though grantees offering ICT training met their employment targets, the total number of youth employed in the ICT sector represents 13 percent of the total jobs only achieved via Bridges. This result is consistent with the views of coordinators from ICT training programs located outside of San Salvador who stated that job placement in the ICT sector was more challenging than anticipated because firms did not have enough job openings to absorb the youth trained. In focus groups with graduates of ICT training programs, they noted that most ICT firms were in San Salvador and there were no opportunities to work remotely. Training and employment in key sectors also varied by sex. In commerce/tourism and agroindustry, approximately 55 percent of participants were female, and in the manufacturing training, 46 percent of the participants were female. However, female participation in ICT training programs represented 34 percent and in energy programs 15 percent. In the commerce/tourism sector, 55 percent of youth employed are female. In the manufacturing and agroindustry, females represent almost half of the youth with employment. However, in the ICT sector and energy sectors, women represent less than a third of youth employed. These results are consistent with our findings from the qualitative data collection. Stakeholders agreed that the FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 42 participation of female youth in these sectors was challenging because there is a general perception that women are not able to perform in these sectors. Despite these low percentages, in the energy sector this represents a pretty significant breakthrough because energy is seen as a male-dominated industry (Figure III.13). Figure III.13. Training completion and employment by sector and sex Source: Bridges monitoring data. Most of the youth trained were between the ages of 16 and 24, but the majority of youth with employment were between the ages of 20 and 24. In Figure III.14, we compare the distribution of youth who completed training courses by age group in key sectors (left graph) with the distribution of youth with employment by age groups (right graph). Most of the youth who participated in training courses were between 16 and 24 years old. In programs offering training in commerce/tourism and energy, most of the trainees enrolled were between 16 and 19 years old. In the ICT training programs the trainees were divided almost equally between age groups 16–19 (881) and 20–24 (837). In the manufacturing sector, almost half of the trainees were between 20 and 24 years old. However, in all key sectors, most of the youth employed were between 20 and 24 years old. Coordinators of training centers noted that job placement of younger trainees was a challenge because firms perceived them as immature and at higher risk of quitting in the short run. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 43 Figure III.14. Number of youth trained by age group compared with the age group of youth with employment Source: Bridges monitoring data. Most employed youth received training in the sector of employment, although exceptions exist. In the commerce/tourism sector, out of the 1,881 trained youth with new or better employment, 87 percent participated in a training focused on that sector. In manufacturing, 93 percent of employed youth participated in a training program in the same sector. In the ICT sector 72 percent of youth hired in that sector participated in an ICT training program (Figure III.15). This suggests that the trainings were targeted to the right economic sectors. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 44 Figure III.15. Youth with employment in the same sector they received training Source: Bridges monitoring data. Note: The total number of youth with employment reflects those who completed the training program and got a job. Overall, training offered was focused in departments19 with employment opportunities, but employment opportunities were somewhat dispersed to multiple municipalities within these departments. The number of youth trained was concentrated in several municipalities among five departments; 6,159 youth were trained in seven municipalities of San Salvador; 1,576 youth completed training in three municipalities of Sonsonate, mainly in Sonzacate; 1,349 trained youth received training in six municipalities of La Libertad; in San Miguel 1,213 completed training; and 1,158 youth completed training in three municipalities of Santa Ana (Figure III.16). Most youth found jobs in the same departments training was offered, but in some cases, jobs were distributed among other municipalities within the same department. For example, most of the youth found jobs in the municipality of San Salvador; however, some of them found a job in 16 other municipalities within San Salvador or in other municipalities close by such as San Vicente. In the east, most of the employed youth found a job in San Miguel, the municipality where they were trained, but others secured a job in other municipalities within the department or in nearby departments (Morazán and La Unión). We cannot establish if the mobility found in the data is convenient or not for the youth; in interviews with trainees, they reported that their job was in a different municipality from their residence and that the commute is approximately two hours long. 19 El Salvador is divided into 14 departments, subdivided into 262 municipalities FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 45 Figure III.16. Training and employment of youth by municipality FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 46 D. What were key barriers and facilitators to the achievement of project results? 1. To what extent do these barriers and facilitators include the focus on design and innovation, actors involved, program rollout, and/or external factors (environment)? We summarize some of the main facilitators and barriers that Bridges experienced under different dimensions in Table III.6. A more detailed discussion of each of these follows the table. Table III.6. Bridges implementation facilitators and barriers according to stakeholders and beneficiaries Area Facilitator of effective implementation Barrier to effective implementation Project design ● Focus on studying the context in the first years (Phase I). ● Focus project activities in selected municipalities and sectors where there were employment opportunities. ● Limited initial focus on job placement. ● Including youth under age 18 as program beneficiaries. ● Relationship between private sector and training centers did not always materialize as expected. ● Eligibility criteria for higher skilled sectors (ICT). Actors involved in the implementation ● Change in Bridges leadership. ● Open communication between Bridges and USAID/El Salvador. ● Reluctance by some in the private sector to change hiring practices identified under Objective 1. Project rollout ● Bridges took the time to listen to and understand USAID’s vision and needs for the project. ● Implementation by local institutions ensured an understanding of the context and the development of context￾appropriate strategies. ● Bridges allowed for flexibility in the project to adjust implementation to fulfill identified needs, such as offering life skills and strengthen the job placement services. ● The learning curve and the amount of resources and time invested in building capacity for the grantees resulted in delays in the implementation. ● High expectations and pressures on what could be achieved in the first years of project implementation led to the rollout of some activities that did not incorporate the knowledge acquired in the first phase of implementation and may not have produced the expected results. External factors during project rollout ● None identified. ● The uncertainty generated by the shift in U.S. government priorities. ● Transportation and mobility of youth. a. Project design The effort made to study the context was reflected in the fact that the activities responded to local circumstances. The design of the Bridges project incorporated research conducted prior to the start of the project and complemented that information with critical studies carried out in the first phase of project implementation. The work conducted in this phase was very useful to better understand the I would not change the design of the project because it was based on evidence and it turned out to be a facilitator that the objectives of the implementation were based on the literature. —USAID/El Salvador staff FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 47 context in El Salvador and develop project activities based on the knowledge acquired in Phase 1 of the project. Focusing project activities in selected municipalities and sectors was successful in identifying places where jobs were available. Having selected municipalities and sectors as guiding principles of the project benefited the transition from training to securing a job. Our analysis using Bridges data showed that overall, most of the youth with employment found it in the municipalities where the project provided training. Job placement was not an important element of the program in the early years of implementation, in spite of the fact that it was in Bridges scope. The main outcome of the project was to increase or improve employment of vulnerable youth; to that end, Bridges provided youth with training and labor orientation. However, in the third year of the project when training was increasing, employment was not increasing at the same pace. Bridges staff noted that youth had no experience navigating the job search process and that activities aiming to improve job readiness were as important as training. The project and USAID incorporated CLA principles and mobilized to include more services focused on helping youth find jobs. Implementers perceived including youth younger than age 18 as project beneficiaries as a barrier to the employment goal. Implementers stated that the employment goal of the project was more challenging to achieve with trainees under age 18 because employers believed that youth under age 18 were not mature enough to respond to the needs of their companies. Our quantitative analysis is consistent with this perception. As shown in Figure III.15, most of the youth who participated in training courses were between 16 and 24 years old. However, in all key sectors, most of the youth with employment were between 20 and 24 years old. In sectors such as ICT, in which job positions required highly specialized skills, the criteria to select trainees could differ from the selection criteria of other sectors with less specialized needs. Coordinators from training centers offering courses in ICT expressed that the criteria to select trainees could differ from those of the rest of the sectors given the specialized skills youth need to gain. Coordinators expressed that given the local context and the inherent lack of opportunities for youth, they could include youth with prior technical training, but no job, in similar programs. In their opinion, these youth should be considered vulnerable despite having higher levels of education (often college degrees) because they are stigmatized, given that their place of residence prevents them from entering the labor market. Including youth with some prior education and training as part of the eligibility criteria would facilitate the recruitment of more qualified youth that have the necessary skills or knowledge to thrive and also provide Bridges with higher employment results. Changing this eligibility criteria would help youth who likely would not be able to obtain a job without Bridges support. The design envisioned a close relationship between the private sector and training centers, and this relationship did not always materialize. Based on the project’s design, training centers were training youth to fill the needs of companies in key sectors. However, in For us, a youth is vulnerable if he or she doesn’t have the tools to access the labor market, even if he/she already has technical or professional training. —Training center coordinator FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 48 municipalities like Ahuachapán, few companies in the ICT sector demanded the skills gained by trainees, which made the employment targets difficult to achieve. Another challenge in the relationship between training centers and private companies was that firms hired trainees as apprentices and, once the apprenticeship period was over, did not want to hire the youth as staff because the company did not want to pay the youth a full staff salary. These incidents affected the relationship between the private sector and training centers that Bridges was hoping to build. Part of the strain in the relationship between the TCs and the private sector is the perception by TCs that businesses continue to have practices and policies that limit youth employment such as requiring polygraph tests and pricy medical exams during the interview process which disproportionally negatively impact vulnerable youth. While Bridges worked to raise awareness of appropriate hiring practices, lack of enforcement of the First Employment Law limits its true potential. Future investments could consider focusing on working with government officials on ways to enforce this law. b. Actors involved The change in Bridges leadership improved the coordination between Bridges and different stakeholders. Bridges counterparts and partners in the implementation of the project agreed that the change in Bridges’ leadership facilitated the rollout of the project. Interviewees stated that once new leadership was in place, they felt included in the decision-making process. According to different sources, with the change of leadership came a change in how Bridges was managed to facilitate support for and communication with training centers. The change in leadership facilitated the coordination between the operational staff and the varied needs of grantees on a daily basis. c. Project rollout In the design and implementation of the project, Bridges took the time to listen and understand USAID’s needs for the project. Throughout the project, Bridges staff was very receptive to the objectives that USAID wanted to achieve through the Bridges project. USAID stated that working as a unified team with the implementer was key to achieving the project's objectives. USAID believed that the project outcomes would not have been the same, had the implementer not listened to the objectives of the mission. In the rollout of activities, Bridges demonstrated flexibility to address identified needs, and embodied the principles of CLA. Some notable adaptations included offering life skills training as part of the technical training and strengthening job placement services. Bridges made changes in the project implementation in response to identified needs or to reach the goals set by the project. Important adjustments in the implementation were the addition of the flexible modality and the inclusion of life skills in technical training; youth also needed support to gain interpersonal skills and employers valued that trainees developed life skills. Bridges also strengthened job placement services based on youths’ need to receive support during the job search process. This is a positive result of the project. Bridges identified that a high school certificate is necessary to enter the job market, and they worked with MINEDUCYT to provide youth opportunities to complete high school education —USAID/El Salvador staff FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 49 The rollout of activities under Objective 3 progressed slower than anticipated. The design of Bridges featured trainings that were carried out by local institutions. However, before these programs could be rolled out, the training centers needed to have the necessary capabilities. Because of the need for organizational capacity building at multiple levels (operational and technical), many of the training programs were rolled out years after the Bridges contract was awarded. Due to the pressure to demonstrate results in the early years, Bridges designed quick interventions to roll out trainings directly (as opposed to through local organizations). Stakeholders reported that despite the need to respond to pressures to yield results in the first years, the trainings designed and rolled out prior to having a full grants program underway, likely did not have the intended outcomes. d. External factors The uncertainty generated by the shift in U.S. government priorities affected the scope and the momentum of Bridges. When the U.S. government announced a freeze on funds to Northern Triangle countries, Bridges experienced a six-month period of uncertainty in which staff did not know which activities to stop and which to continue. During this period of great uncertainty, there was no formal modification or a stop-work order. However, USAID/El Salvador communicated to Bridges, that it could not recruit any more youth and enroll them in future trainings. The announcement came at a time when Bridges had reached full momentum in its activities and targets were beginning to be achieved. In response to the budget uncertainty, USAID decided to reduce Bridges’ total estimated ceiling20 by 20 percent. Most of the funding cuts affected Objective 3 activities related to youth readiness and job placement. Youth faced several constraints in mobility and transportation that affected the rollout of project activities. Bridges attempted to reduce barriers to training access by giving trainees stipends and targeting the rollout of training courses in several municipalities. However, the inherent issues with insecurity and ground transportation in El Salvador meant that implementers had to make a great effort to recruit vulnerable youth from areas close to the training, aiming to prevent them from traveling from far away. These efforts often resulted in delays in the rollout of the programs since they needed to meet a minimum enrollment level to proceed. 20 Refers to the maximum amount that the contractor will be paid. The uncertainty also affected the morale of the team; we had to tell a high￾performance team—it’s over (recruitment and enrollment needed to stop). —USAID/El Salvador staff Although it is a small country, insecurity limits youth mobility from one municipality to another, so trainings offered need to be fulfilled within the area. We made a great effort to disseminate information and recruit, so youth knew what we offered and to be able to fill the rooms. —Bridges staff FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 50 2. How did the grants under contract mechanism advance, or detain, implementation and the strengthening of local solutions? With the GUCs mechanism, local capacity was strengthened, which benefits the sustainability of the project. However, working with this mechanism meant that Bridges faced several challenges. In order to manage such a large grants fund adequately, USAID and Bridges staff had to provide a significant amount of coaching, financial management, and contractual support to grantees. Most of the grantees had never worked with USAID and did not understand the administrative and contractual obligations tied to USAID funding. TCs struggled with finance and operational processes such as procurement, invoicing, and providing youth with stipends. The lack of prior USAID experience meant that training centers required a lot of support from Bridges and USAID staff in order to manage and monitor grantee progress. Because of this need, Bridges restructured its organizational chart and developed the grant management department, with a multidisciplinary team tasked to provide both technical and operational support to grantees to be able move implementation forward. While the project did not keep track of the time and effort dedicated to strengthening grantees, these efforts are directly aligned with USAID’s Journey to Self-Reliance. These grantees will likely be able to continue to seek and manage USAID funds in the future, furthering USAID’s efforts under the New Partnership Initiative21. The GUC model delayed the implementation of the project. The requirement that most activities be implemented through the GUC mechanism meant that trainings were rolled out with delays because activities could not proceed without an approved grants manual and trained grantees. It took two years before the first set of grants were awarded and grant funded activities took off. The GUCs model brought both benefits and costs to the project that should be considered in future projects. The GUCs model implied a trade-off between the benefits of building local capacity (linked to USAID’s Journey to Self-Reliance and New Partnership Initiative) and the cost of learning by the grantees. Although training local institutions is an important development objective, it comes at a cost—namely, that achievements and results take time and things do not operate at the speed USAID expects. For projects expecting “quick wins,” having to implement all activities through GUCs can be a challenge. The size of the GUC fund was very large compared to the capacity of local organizations to manage and account for such large grants. Bridges opted for a design that placed value in strengthening local capacity, but future projects may want to find an approach that uses local organizations while not limiting programmatic activities to those under the GUC mechanism, or could consider two different projects, one focused in strengthening the local capacity and the other to offer training and job readiness support. 21 USAID launched the New Partnership Initiative to support the Journey to Self-Reliance and encourage the participation and knowledge of local organizations. I agree that local institutions were worthy of a large investment, but for Bridges, this has taken extraordinary effort and significant hand holding and training. —Bridges staff FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 51 IV. STAKEHOLDER PERCEPTIONS OF RESULTS A. What are stakeholders and beneficiaries’ perceptions of the training program and the support services Bridges provided? In this section, first we present grantees’ perceptions of changes in the education approach and the strengthening that TCs received. Second, we show trainees perceptions of training and support services they received. Third, we present employers’ perceptions on training and skills gained by youth. Fourth, we present employers perceptions on training centers and their prospects to hire vulnerable youth. Finally, we present our assessment on how gender equality was integrated in the implementation of project activities. 1. What are grantees’ (training centers) perceptions about: 1.1. Has engagement with private sector made a difference in the education approach? Bridges transformed TCs’ approach to respond to the needs of the private sector. In El Salvador, the approach in the professional training sector sought to address the demands of youth for training. In fact, youth—as opposed to employers—were perceived as the client and, therefore, the content of the training was designed to fulfill the demands of youth, even if they were not aligned with job opportunities. The technical assistance and support provided by Bridges to engage with the private sector transformed the approach. Perhaps this transformation was most visible in the ICT sector. Coordinators from ICT training centers agreed to change or update the curricula used in ICT sector training to link it to the needs of the private sector. All training centers delivering ICT courses now work with CasaTIC to develop the curricula of the training programs, which they did not do consistently before. The textile industry also changed their approach in response to a collaboration between INSAFORP and the private sector. A representative from a training center reported that it was now using updated curricula developed in coordination with firms in the textile sector thanks to Bridges’ intervention. Representatives of the TCs are motivated to continue updating the training programs, since they see a link between offering market demanded courses and TC’s future financial sustainability. Likewise TCs are aware that offering programs with certified curricula provides them with an extra level of credibility and market relevance which will allow them offer training funded by INSAFORP. Finally, TC also witnessed the benefits of addressing labor market needs—more youth were interested in enrolling in courses that increase their likelihood of securing a job. Training centers adopted more hands-on training interventions to better prepare trainees for future employment. Training centers used a competency-based approach that combined theory-based learning with project work that honed practical skills, unlike previous courses that were predominately theoretical. Training centers staff used hands-on training to keep trainees engaged in the learning process while they learned how to perform specific tasks demanded by the labor market, which in many cases helped youth learn faster. In the ICT sector we have made modifications to the curricula between cohorts of trainings to incorporate the feedback from firms –Training center coordinator FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 52 1.2. In what areas were institutions most strengthened? Grantees were strengthened in a variety of ways, whether through official capacity-building interventions as summarized in Table III.1 or through support for managing USAID-funded grants and their associated operational, administrative, and financial needs. TC coordinators reported having Bridges support to improve TC facilities and purchase new equipment. Coordinators from all training sectors visited during data collection reported that USAID provided equipment and materials, such as computer and laboratory equipment, didactic resources, and software for training centers offering professional training courses. The director of a technical high school (BTV) corroborated that Brides improved the facilities of five education centers and provided equipment, tools and accessories for laboratories in institutes linked to MEGATEC. The equipment and resources were key to providing hands-on learning for youth. Although the support to create alliances with the private sector was deemed valuable, not all training centers reported receiving the same level of support. All coordinators reported that Bridges provided support and technical assistance to create alliances with the private sector. Coordinators from 6 of the 8 training centers interviewed reported that Bridges support helped them increase the number of firms they worked with and improve the relationship with existing partner firms. However, other coordinators expressed that the potential alliances identified by the project were not a good fit for the TC trainees, the identified firms were not interested, and some firms did not have the vacancies to hire Bridges trainees. The challenges associated with engaging with the private sector appear to also impact these training centers’ ability to help youth secure employment. While it was ultimately the TC’s responsibility to build connections with private sector partners, in reality Bridges staff had to provide a lot of support. All TC coordinators reported receiving training and technical assistance to improve organizational systems. In-person interviews with coordinators of seven training centers highlighted the project’s contribution to strengthening human and institutional capacity to improve their ability to manage the organization. Training centers adopted Bridges’ monitoring and evaluation system, which uses a diagnostic assessment to vet youth knowledge and skills prior to and upon completion of the program. Staff also received training and management support to improve financial and management aspects of the centers. As described in Chapter III, findings from the initial assessment (carried out in FY2017) of 20 training centers shaped the project’s work to improve the capacity of training centers and the quality and relevance of training available to vulnerable youth. Follow-up assessments conducted in FY2018 for three training centers—Professional Training in Computing (CAPUCOM), AGAPE, and Association for the Organization and Education of Women Entrepreneurs of El Salvador (OEF)—revealed that they made progress. As shown in Figure A.8 in the appendix, all three training centers improved their overall scores: CAPUCOM improved noticeably in student services, and equipment and material; AGAPE made important improvements in equipment and materials, and organization and management; and OEF improved its finances and overall For industrial clothing courses, we had obsolete machinery (manual machines). Thanks to Bridges support, we have automated machinery that will help youth to have a better chance of passing the selection processes in companies. –Training center coordinator FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 53 institutional infrastructure. For ESFE-AGAPE, Bridges supported the development of curricula for new programs linked to the MEGATEC system, staff were trained in life skills, and administrative processes were strengthened (see text box below). Strengthening ESFE-AGAPE AGAPE Specialized Franciscan High School (ESFE-AGAPE) was one of the training centers awarded a grant in the first round of requests for applications to develop trainings in four economic sectors: commerce/tourism, energy, manufacturing, and information and communication technologies. This grantee developed grant-funded activities in two modalities: professional training and formal education. Grant-funded activities started in September 2017 and were implemented through four technical high schools: 1. National Institute of Ciudad Arce (INCA) 2. National Institute of Lourdes Colón (INDEL) 3. Thomas Jefferson School Center (COED) 4. National Institute of Sonzacate (INSO) Key improvements for ESFE-AGAPE • Two technical degrees authorized by the Ministry of Education • Four technical high schools in Ciudad Arce, Colón, Sonzacate, and Sonsonate linked to the MEGATEC system • Curricula for new programs developed and implemented • Training staff certified as life skills instructors • New Category 4 Electrician curriculum developed and implemented • Administrative processes related to hiring, procurement, and finances improved • Four high schools strengthened through laboratory equipment and institutional infrastructure providing access to people with disabilities • C-Orienta facility established in Sonzacate to provide job placement support services C-Orienta facility in Sonzacate Institutional strengthening to ESFE-AGAPE translated into important benefits for vulnerable youth. – 370 completed courses in three high school degrees – 243 completed training programs for vulnerable youth – 353 received job placement services in Sonzacate FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 54 1.3. What approaches, technical assistance, and activities have been most critical for vulnerable youth job placement success? The percentage of youth who completed training with new or better jobs increased each year of the project (Figure IV.1). As described in Chapter III, Bridges fell short of its employment rate targets (set at 25 percent) in the first three years of program rollout. However, by the fourth year of the project, 36 percent of youth trained found a job and 40 percent by March 2020. As we mentioned above, in the period April to July 2020, approximately 400 hundred youth completed their training, but most of these youth were unable to participate in interviews of find jobs because most economic activities in the country came to a halt in response to COVID 19. We therefore looked at what factors or changes occurred in the third year that may have been successful in helping youth find jobs. Figure IV.1. Percentage of youth employed from youth that completed training the corresponding year Source: Bridges monitoring data. As shown in Figure IV.1, the increase in youth employed occurred in the fourth year of implementation and it might be related to the following factors: 1. The inclusion of life skills training in technical training and formal education. In 2018, Bridges included life skills training in all courses offered, including those offered in the formal education sector (BTV and flexible program). In our qualitative analysis, we identified that including life skills training as part of the overall training program was seen favorably by the private sector, particularly when working with vulnerable youth. This sentiment was especially true in the commerce/tourism sector where interpersonal skills are essential for job performance. This firm decided to collaborate with Bridges because they provide life skills training, which translates into youth with interpersonal skills. –Representative of the private sector FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 55 2. Systematized job placement services. In mid-2018, Bridges started the rollout of activities focused on helping youth find jobs. Bridges offered job readiness sessions where youth (1) received career guidance, (2) participated in mock interviews and received feedback from human resources professionals on how to improve their performance, and (3) received personalized tips and assistance on how to strengthen their resumes. The job placement activities also included the dissemination of youth resumes to companies participating in job fairs, as well as through online job search platforms. In most cases, these job placement services were offered in addition to the training. In addition, Bridges decided to offer DLI to vulnerable youth with (non-Bridges) training that still needed support in finding jobs. 3. Targeting training to key economic sectors. The targeting of key sectors identified in Phase 1 of the implementation was effective in the success of youth placement. A review of the data showed that 85 percent of youth ended up in jobs directly relevant to the sectors for which they were trained, suggesting that youth employed benefited from the focused training to get a job in those specific sectors. Almost half of the youth employed found a job in the commerce/tourism sector (2,441), followed by the manufacturing sector where 32 percent of the youth (1,507) were hired (Figure IV.2). Youth employed in the ICT sector represent 13 percent (605) of the total youth employed. The number of youth employed in the ICT sector was low compared with the number of youth who completed ICT training (2,079). The inclusion of ICT training was strategic given: (1) the expected growth in the short run22, (2) the profile of workers in the industry is digitally savvy youth, and (3) the potential to perform tasks remotely. However, some of the expectations were not fulfilled. Based on conversations with youth, we learned there were very few opportunities for remote work, which prevented trainees from outside of San Salvador to find jobs in this sector. Another barrier to achieving better employment results in this sector was that Bridges trainees were competing for jobs with youth with college diploma in ICT or related degrees. Despite the comparatively low placement of youth in the ICT sector, the employment results in this sector represent a breakthrough because vulnerable youth are entering a market that mostly hires highly educated youth. 2. What are Bridges trainees perception about: 2.1. The training they received, including technical training and life skills training, when applicable? Bridges trainees generally had positive perceptions of the training received. Trainees who participated in technical and life skills training found it to be well organized and 22 The employment labor market assessment Bridges conducted estimated that 58 percent of IT companies expected to expand their workforce within the next 12 months. Figure IV.2. Youth employed by sector FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 56 comprehensive. Participants praised the instructors’ knowledge of the course content and the tools used to keep them engaged. Bridges’ main strengths and weaknesses identified by the participants are listed in Table IV.1. Although most youth had overwhelmingly positive feedback, some areas of weakness were mentioned. Table IV.1. Main strengths and weakness of training courses as reported by youth Strengths of training courses Weaknesses of training courses Content of the training course Instructors – Knowledge of the course content. – Techniques used to keep them engaged. – Instructors available to respond to questions. – None identified. Content or curricula – The course curricula matched the needs of companies in selected sectors. – In some courses, the curricula covered too many topics in a limited period, leading to a superficial review of the material. Duration of the training courses – None identified. – Some courses were too short. TC infrastructure Training center facilities – Adequate for the needs of the course. – Included facilities for people with disabilities. – None identified. Training center equipment – New and modern equipment. – Software, equipment, and machinery were similar to those used in firms. – In the textile courses, participants reported insufficient access to machinery. Training center atmosphere Training center staff (coordinators or school directors) – Coordinators from training centers were very supportive of trainees’ needs. – None identified. Training center environment – There was no discrimination based on gender, political ideology, place of residence, or sexual preferences. – None identified. Most of the trainees reported being satisfied with the content of the technical training. Trainees and graduates stated that the content and the teaching methods used in the technical training were interesting and useful; they liked that the courses included a combination of theoretical and practical examples and mirrored conditions in the workplace. In courses where it was necessary to operate machinery, practicing and becoming familiar with the equipment helped them to feel more comfortable during job interviews. Trainees viewed life skills training as the most helpful support, even among youth who had not found jobs. Nearly all interviewed trainees stated that life skills training helped them throughout the job search process. In fact, many interviewed youth expressed that the life skills component of the training was the most relevant support received from the project and will help FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 57 them beyond the job search process. Life skills training- and particularly socio-emotional training- helped trainees identify their inner strengths, be more tolerant and patient when learning new things, and relate better to other people. They said it has also given them tools to gain self￾confidence. All these tools have helped them to feel more comfortable in the job search process. Most of the trainees reported that courses were too short, but a risk of offering longer courses is that dropout could be higher. Interviewed trainees expressed that they wanted the training courses to last longer. Even in courses that lasted several months, participants reported that many topics were covered in very short periods, with only a superficial review of topics. However, there might be a trade-off between longer courses and dropout from the courses. The results of our quantitative analysis, while not causal, suggest dropout from training courses was higher in training courses that lasted longer. The longest courses tended to be those with more specialized training (Figure IV.3). Approximately 62 percent of those who did not complete training were enrolled in courses that lasted more than two months. For training courses lasting less than a month, the dropout rate was 20 percent; for courses with a duration ranging one to two months, the dropout rate was 18 percent. While we cannot assess the effectiveness of training courses based on duration, there is a tradeoff between course length and completion and employment outcomes. In particular, we found that there are more non-completers in programs that lasted longer, but employed youth participated in training programs that on average lasted longer (Figure III.15). Training met most basic needs, but youth who only enrolled in life skills training said they were interested in participating in additional technical training. Trainees who participated in courses that only offered life skills training expressed strong interest in enrolling in a technical training course but thought they could not enroll given their prior participation in a Bridges supported training. A further review of this concern revealed that only a small subset of youth received life skills–only courses (prior to the incorporation of life skills into all technical training courses). Although Bridges did allow youth to enroll in more than one program when needed, youth had the perception that this was not permitted. It is possible that TC did not encourage trainees to enroll more than once since for reporting purposes, the youth would only be counted once. TC did report allowing youth to take more than one course, but either would not provide associated stipends or would provide about half of the stipend youth received in the first course. These strategies were in place to avoid having youth reenroll purely because of the stipend. I’m using the skills developed in the training not only in the job searching process but in my daily activities. I have more confidence and I’m more social now. –Bridges trainee Figure IV.3. Dropout by duration of training course Source: Bridges monitoring data. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 58 Few trainees had the opportunity to participate in internships with firms. Trainees expected to participate in internships or apprenticeships with local companies during the training, but few training centers offered them this opportunity. Bridges staff noted that not all grantees had the obligation to facilitate internships to the youth as part of the training provided. Moreover, in interviews with employers, only few companies reported to offer internships to youth. Both situations limited opportunities for youth to gain experience. The training programs largely met trainees’ need to gain the technical and life skills required to join the labor market. Most of the trainees interviewed noted that they gained relevant skills through the training courses and had the opportunity to put in practice the knowledge learned. However, some of the trainees stated there were some needs that remained unmet, and Table IV.2 presents a summary of the goals achieved and remaining needs. Table IV.2. Goals of training programs and trainees needs Goals of training programs Goal achieved? By which activity Remaining needs Trainees gained specialized technical skills Yes, through the technical courses training For the subset of youth that participated in life skills training only, there was a continued interest in participating in technical courses. Trainees gained life skills Yes, through the life skills courses or the life skills module of the technical courses None Trainees practiced the knowledge learned Yes, through the hands-on modules within the trainings None Trainees interacted with potential employers during the training No Participants expected to do an internship or apprenticeship during the training courses, but few had this opportunity. Trainees were prepared to perform well during the job search Yes, through the job placement support Not all trainees reported receiving the same amount of job placement support—. Variation existed depending on the training center. 2.2. How useful the training/certificate has been in helping them get a job? The trainees reported that after participating in the project, they feel better prepared to enter the labor market. Most of the interviewed trainees stated that they acquired skills demanded in the labor market. The training course also helped them become familiar with the equipment and tools used in the companies. Trainees mentioned feeling more confident about their potential and the contributions they can make when entering the job market. The trainees reported that being graduates of a Bridges-supported program has prepared them well for the labor market and reduced their stigma. As profiled in the previous sections, most of the interviewed trainees believed that the trainings have helped them feel more confident in their ability to get a job. In addition to feeling better prepared, many trainees believed that graduating from a Bridges-backed training helped the private sector overcome the usual barrier to considering vulnerable youth as possible employees. Trainees FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 59 explained that companies are reluctant to hire youth living in at-risk (or gang-dominated) areas because they are perceived as not educated, and they normally do not have a network or a reference that can vouch for them. However, having a certificate from Bridges has helped trainees overcome these stereotypes. Trainees stated that a Bridges certificate is equivalent to having a personal reference, and they feel they can now be considered for a position for which they would have otherwise not have been considered. 2.3. How useful the support services were in getting a job and succeeding at it? Trainees who had a job and those looking for one agreed that the job placement activities were very helpful. Interviewed graduates reported that the support received to prepare or strengthen their resumes, the coaching on how to respond to interview questions, and advice about how to dress or even how to greet were very helpful. Having mock interviews was particularly useful in the selection processes. Even youth who were still looking for a job said that the support received has helped them to advance in the process. Prior to participating in Bridges training, youth reported never receiving call backs. Now, even if they do not secure employment, they usually advance to at least the second phase of the hiring process. Because of the absence of a counterfactual, we cannot assess whether the youth found jobs because of the job placement activities. However, a review of the data revealed that for the subset of youth that were employed, those that found jobs once job placement activities were institutionalized found jobs faster on average, than those who found jobs before these services were systematically introduced. To determine when job placement was institutionalized, we picked June 2018 because after this date Bridges issued several grants focused exclusively on supporting job services, ramped up job placement trainings in the centers, and rolled out the C￾Orientas. After the institutionalization of job placement services, youth on average found jobs three months after completing the training. In comparison, when job placement services were limited, it took them on average five months to secure a job after completing training. Given this, we believe the qualitative and quantitative data support that job placement services benefitted youth. The amount of support services varied by training center. Almost all trainees reported receiving labor orientation services during the training. In fact, many training centers provided continued labor orientation support after training completion. However, some graduates expressed that after completing the training course, they did not continue receiving follow-up coaching, mentoring, or job placement support. 2.4. How Bridges has improved their future prospects in the labor market? Bridges trainees agreed that Bridges made progress in addressing barriers that youth face to access the labor market, but some barriers remained. Almost all youth interviewed expressed that Bridges improved the perception and confidence that firms have in hiring vulnerable youth. The project promoted the inclusion of vulnerable youth in the labor market by deconstructing some of the stereotypes that private firms held about youth. Bridges also raised The feedback provided by the job placement manager during the mock interviews was very useful. It helped me to better communicate my strengths throughout the job search process –Bridges-trainee FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 60 the awareness of the existing benefits offered in the regulation for hiring youth (Youth’s First Employment Law). However, some barriers remained a challenge for the improvement of the WFD environment. For example, some of the existing regulations to support vulnerable youth is not very effective because of a lack of enforcement. Bridges has provided vulnerable youth a professional network. In focus groups, trainees expressed that the project helped them build a network, even after completing the training. They also reported receiving information about potential job opportunities, not only from the TC staff but also from their newly established network. This is an important benefit of the program. However, in order for it to be sustainable, the TCs need to maintain the network of trainees, keep the alumni engaged in this network and foster the relationship with the private sector. The first steps toward maintaining the network have been made. Asociación de Líderes del Talento Humano (ALTHES) will manage a WhatsApp group, serving as a bridge between private sector employers, outsourcing companies, and headhunters seeking qualified candidates and training centers with skilled graduates. As of July 30th, representatives from USAID and Bridges met with 13 training centers to encourage them to keep participating in the network. Trainees stated that all companies need to improve hiring practices. Most of the youth interviewed reported that some of the hiring practices create financial burdens for youth because in the final stage of the application process, youth need to take medical exams and polygraph tests (which are costly). Some youth used the Bridges stipend to cover these expenses; however, TCs cannot offer this continued financial support once Bridges ends, which makes the need to change these hiring practices more pressing. We summarize the barriers youth perceived in the labor market and whether the Bridges project addressed them in Table IV.3. Table IV.3. Barriers perceived by trainees and addressed by the project Perceived barriers in the labor market Barrier addressed by the project? How? Remaining barriers Mistrust of youth from vulnerable areas. Yes, partly by making alliances with private companies and supporting them in making concrete efforts to improve the workplace environment for youth, such as changes in policies and practices. Some negative stereotypes of youth persist. Employers interviewed expressed that youth living in vulnerable areas continue to face risk by crossing between different gang territories or traveling when it is dark, which can inhibit their ability to comply with the work schedule. Employers do not have incentives to hire youth given the limited number of job openings in relation to job seekers. Yes, partly by raising awareness of existing laws and regulations that offer incentives to hire youth ranging 18 and 29 yearsa No enforcement of regulations. Mechanisms to enforce regulations promoting youth employment and preventing employment discrimination practices are not enforced so there is a perception that they are ineffective. The job placement manager of the training center is in constant communication with us, but we also have a network. If I hear about job opportunities, I share it with other graduates. —Bridges trainee FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 61 Perceived barriers in the labor market Barrier addressed by the project? How? Remaining barriers Youth lack established professional networks. Yes, partly by helping trainees to build a network; staff from the TCs provided information on job openings and cohorts of trainees keep in touch after graduating. Sustainability of the network for vulnerable youth. Once Bridges support ends, TCs need to maintain the network of trainees and their relationship with the private sector. Where internships exist, firms want to continue employing youth as interns instead of staff to keep wages low. Yes, through Objective 1 which aims to improve companies’ hiring practices. Better hiring practices need to be adopted by all companies. Some trainees reported that the transition from intern to staff was challenging because companies wanted to keep paying low salaries. Jobs are not always available in areas close to where youth live, particularly less developed areas. Yes, partly. Overall, training was offered in areas where youth found jobs, but in areas far from San Salvador, youth found it more challenging to find a job. Limited local opportunities for employment. Most trainees employed reported that the job is not in the same municipality and commuting takes them approximately two hours. Other trainees with no job reported they were considering moving to the area where a company is located. Our quantitative analysis suggests that for youth employed, the job is in the same department where the training was received but some jobs are spread out across municipalities within the department. Burdensome costs for youth to comply with requested background checks as part of job applications. Yes, by promoting better hiring practices. Burdensome hiring practices prevailed in the job market. There is a widespread practice from private companies to request applicants’ medical exams and polygraph tests, creating financial burdens for vulnerable youth. a Ley de incentivo para la creación del primer empleo de las personas jóvenes en el sector privado. There were geographic differences in graduates’ perceptions about prospects for getting a job. According to graduates from San Salvador, employers are more open to hiring vulnerable youth because they know that if they participated in Bridges, they are likely to have strong technical skills and have the self-motivation to move ahead. However, interviewed graduates trained in areas far from San Salvador expressed that despite the support they received from Bridges, their job prospects have not changed because the area they live has no job opportunities. For them, improving their job prospects implies moving to San Salvador now that they have gained skills valued by the job market. A review of the data revealed that in sectors like ICT, youth employed were concentrated in San Salvador, which is consistent with trainee perceptions about job prospects in their municipality. In the ICT sector, youth completed training in six municipalities from five departments: San Salvador (764), San Miguel (452), Santa Ana (139), Sonzacate (273), Ahuachapán (111), and Soyapango (105). However, out of the 605 youth with employment, almost half secured a job in San Salvador (354); approximately 30 percent found a job in San Miguel (113); and fewer than 20 youth were employed in Santa Ana, Sonzacate, Ahuachapán, or Soyapango (Figure IV.4). FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 62 Figure IV.4. Training and employment of youth in the ICT sector FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 63 2.5. Is there more openness from the private sector to hire youth (particularly vulnerable youth)? Trainees believed that some progress has been made in reducing stigmas, but barriers still exist. Trainees from San Salvador and Santa Ana (more urban areas) expressed that Bridges has improved companies’ perceptions of youth, especially the perceived potential of youth to contribute to the needs of the companies. Although youth believe there is more openness in considering them, there is still a discrepancy in their acceptance based on the existence of gangs in the municipality. For example, youth believed that those residing in gang-dominated areas still face more barriers to acceptance. Trainees perceived that graduates of San Salvador have an advantage because they are perceived to have access to better education. Trainees indicated that a training program in San Salvador was seen more favorably than a program in a different region, indicating that regional biases may exist. For example, trainees from the ICT sector in Ahuachapán expressed that despite Bridges strategies to raise the profile of youth, they perceived a lack of interest from companies to hire youth, particularly if they are not based in San Salvador. 2.6. What are the perceived before and after effects given the project’s intervention? Interviewed participants perceived positive changes in the technical skills and life skills they gained, in their job search, and their job performance. Youth interviewed reported gaining technical skills that the private sector values; they also believed that their interpersonal skills improved. Some youth described that their ability to interact with other people improved drastically, and they also gained self-confidence, and felt more resilient. Youth expressed that the acquired knowledge and skills helped them navigate the job search process with more confidence. Trainees’ perceived before and after effects of the project are summarized in Table IV.4. Table IV.4. Trainees’ perceived before and after effects of Bridges project Before After Youth technical skills – Youth gained skills that were not relevant to the job market. – Youth gained skills demanded by the private sector. Youth life skills to thrive in the workplace – Youth with few interpersonal skills developed. – Youth gained interpersonal skills— self-confidence, resilience, and team work. Readiness for job searching – Youth engaged in a job search without information or strategy to look for jobs. – Youth were very unfamiliar with the job search process. – Youth targeted their job search to opportunities where their profile was relevant to the job opening. – Youth had more confidence in the job searching process. On the job performance – n.a. – Youth gained confidence to perform work-related tasks. – Youth feel motivated and engaged in their job. n.a. = not applicable. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 64 3. What are employers’ perceptions about: 3.1. Trainees’ technical skills and knowledge? Most of the employers reported they were satisfied with the technical knowledge acquired by youth. Representatives from the private sector reported that trainees who completed the training courses had basic technical knowledge. For example, representatives from the manufacturing sector stated that trainees were familiar with the equipment and that familiarity facilitated the transition to the job, especially if this was trainees’ first employment. In most cases, the firms provided additional training, more specific to their daily tasks in the job, but it was helpful that the youth were already familiar with the tools, machinery, or activities needed for the positions. Despite the desire to recruit youth with more hands-on training, the firms do not have internship programs to offer them. Some of the representatives from the textile and commerce/tourism sectors reported that it would have been useful for trainees to gain more practical experience during the training course. A representative from the textile sector expressed that textile training courses are short, and trainees do not have enough time to practice with the sewing machines. In the commerce/tourism sector, a firm representative mentioned that it would be useful for trainees to participate in internships to harness the skills, knowledge, and theory they learned in the training. However, internship culture needs to further develop in El Salvador. Interviewed employers report not having apprenticeship programs to offer the youth. 3.2. Trainees’ life skills? Employers value the life skills of youth. Employers reported to be satisfied with trainees’ life skills. They emphasized that Bridges graduates were disciplined, responsible, and punctual. Bridges graduates have shown professional behavior; they were engaged with the company’s tasks and can work in teams. Employers expressed that an important benefit of the life skills training is the confidence gained to thrive in the workplace. An employer representative commented that the company’s staff were so impressed with the positive results of life skills training that they have incorporated this type of training for all of their employees. Employers noticed that some interpersonal skills of trainees could be improved. An employer from the ICT sector reported that one aspect to improve in the life skills training is giving youth the tools to ask for help and to say “no.” In the workplace, sometimes youth failed to ask questions when they did not understand a task. Youth also had trouble saying no or at least explain why it was not feasible to conduct a task that was requested with an infeasible timeline. These are important skills that should be emphasized in the life skills training. Another employer suggested that Bridges trainings could delve into topics like negotiation and problem solving, including how to handle frustrating situations. The trainees already have basic technical skills, so it is easier for them to learn how to do the tasks. –Representative of the private sector We like to hire Bridges-trainees because we know that they received the training to develop interpersonal skills. —Representative of the private sector FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 65 3.3. Trainees’ on-the-job performance for current position or potential for upward mobility? All employers expressed satisfaction with the performance of Bridges graduates. Employers reported that Bridges trainees demonstrate professionalism in the workplace. They have enough technical knowledge to learn how to do tasks at the pace the firm requires, and most of them are highly motivated to learn. Trainees show enthusiasm and a positive attitude, which have helped them stand out from the rest of the employees. In an interview, a private sector representative noted that Bridges trainees are doing well in the workplace because they develop a sense of belonging. The representative also noted that trainees feel grateful to the firm and the training center because these youth are experiencing opportunities that they did not previously believe would have existed for them. Private sector representatives noted that Bridges graduates continue to excel once employed and some have even experienced career advancement since their initial employment. Private sector representatives from various sectors expressed that Bridges trainees have the potential to advance professionally in companies. For example, in the construction sector where the work is by project, the company representative mentioned that a youth hired as an electrical technician for a first project was hired for a second project with a better job and more responsibilities because of his good performance. In the plastics sector, employers have also promoted several graduates of the first cohort based on performance. Representatives from the private sector noted the importance of teaching youth both technical skills and how the industry operates, including work schedules. A representative of the plastics sector mentioned that several youth from the first cohort resigned within a few weeks because the youth did not like the company’s work hours. (Employees in the plastics sector work four days in the day shift, followed by two days off, and then four days in the night shift.) To address this problem, the sector has coordinated with training centers so that youth can visit companies and better understand what work is like at an early stage of the training course, to reduce attrition. Employers and trainees agreed on most of the strengths and needs identified regarding skills gained and performance of youth. Our qualitative assessment sought to identify similarities and discrepancies in perspectives across respondents. In Table IV.5, we present employers’ perceptions of trainees’ skills and their performance in the job search process and on the job. The identified strengths and remaining needs were triangulated with the reports from trainees. Overall, employers and trainees agreed on the usefulness of technical and life skills training. Trainees reported that life skills training was particularly useful. Employers noted that Bridges trainees stand out in part because of the interpersonal skills they gained. Regarding the performance on the job, we found some differences on the reports from employers and trainees. When managers have openings in their areas, they ask for Bridges trainees because they have been responsible and proactive. —Representative of the private sector A Bridges trainee hired as a customer service assistant in a store demonstrated great technical skills, and he is in training for a position as technical assistant in electronics. —Representative of the private sector FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 66 Although the two agree that the performance of trainees on the job was good because they have the technical base to learn the firms’ processes at the pace required, employers noted that trainees needed to better understand how the firms’ schedules work. Table IV.5. Employers’ perceptions on trainees’ skills and performance Strengths Remaining needs Degree of agreement with trainees’ perceptions Perception of employers on trainees’ Technical skills – Trainees became familiar with the tools, machinery, or activities needed for the positions. – Trainees should have more experience through more hands-on training or internship opportunities. – Mostly agree. Life skills – Trainees showed professional behavior in the workplace; they were disciplined, responsible, and punctual. – Trainees gained the necessary confidence to perform in the workplace. – Trainees need to learn how to ask for help and develop self-direction. – Trainees need to strengthen negotiation and problem-solving skills. – Somewhat agree; only employers reported that youth need to improve negotiation and problem￾solving skills. Readiness in job search process – Most of the trainees had a good first interview; according to interviewees, they were self-confident and demonstrated an eagerness to learn. – Not identified. – Mostly agree; youth expressed that with Bridges support, they performed better in the job search process. On-the-job performance – Trainees have technical skills to learn how to do tasks at the pace the firm requires. – Trainees need to understand better how the industry operates. – Somewhat agree; only employers reported that youth need to understand work schedules of the industry. Career advancement – Bridges trainees are motivated and engaged to keep learning and advance in their careers. – Not identified. – Agree; youth and employers expressed that trainees were motivated and engaged in the job. 4. What are employers’ perceptions about: 4.1. Training centers and their improved curricula Most employers reported that the training centers’ curricula respond to their human capital needs. Employers from different sectors corroborated that the curricula of the training courses offered through Bridges are consistent with their needs for technical knowledge and interpersonal skills. The involvement between employers and training centers with respect to curriculum development varied. Most private sector representatives reported having provided feedback on existing curricula; in the case of the ICT and manufacturing sectors, trainings were designed in conjunction with private sector representatives. In the case of the manufacturing sector, some firms identified staff to provide input into the curriculum to ensure it responded to company FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 67 needs. The representative from the ICT sector noted that he was even involved in hiring training instructors. 4.2. Their willingness to hire youth from high-risk areas? Similar to trainees’ perception, most of the employers reported being more open to hire vulnerable youth based on the experience they have had with the Bridges trainees. Representatives of the private sector interviewed mentioned that with the experiences they have had, they trust that Bridges graduates will have the knowledge and motivation to carry out their tasks. In our interviews with employers, most representatives from firms in the manufacturing and commerce/tourism sectors reported that, for the most part, they have hired vulnerable youth. Moreover, some of them reported that for jobs in operational areas, they hire youth with Bridges trainees’ characteristics— in terms of education (high school certificate) and age. However, they expressed that the experience with Bridges has been valuable, and they look forward to hiring vulnerable youth because they have been committed employees, and have performed well in their tasks. A new model in the ICT sector is addressing some of the barriers to hire youth. Three grantees implemented training programs through Software Development Centers— training and practice laboratories where youth and companies work together on IT projects. The idea was that employers can hire for remote internships or project-based “gig economy” assignments. A review of the quantitative data and qualitative information gathered showed that implementing this model was challenging; TCs and trainees reported that there were very few opportunities for remote work, which prevented trainees outside of San Salvador from finding jobs in this sector. Our data show that approximately 30 percent of trainees in this sector found a job, a lower rate than the 35 percent of trainees employed in the service sector and the 74 percent employed in manufacturing. However, a representative from an ICT firm located in San Salvador noted that he perceived a slow change in the traditional recruitment process. Companies may be more open to hiring software development center (SDC) trainees (Bridges trainees in the ICT sector) and not just graduates from the traditional IT schools (college graduates or technicians). FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 68 Table IV.6. Private sector’ perceptions of inclusion of vulnerable youth in their staff Perceptions from stakeholders who perceived a change in the inclusion of vulnerable youth in their company or the sector “Bridges has definitely generated awareness within the sector. Now SDCs have become a common source of employees when companies look to hire staff.” –Representative from the ICT sector Perceptions from stakeholders who perceived that their company or the sector already included at-risk youth in their staff “The company already hired at-risk youth, but the project has opened up more possibilities; we discovered new places that now we see as a source of employment, a place where we would go to offer work.” –Representative from textile sector “We already hired at-risk youth before Bridges because it is challenging to find someone who does not come from a gang-dominated zone. With the project, our will to include them in our staff was reinforced because we noticed that they are very responsible, punctual, and they really want to work.” –Representative from manufacturing sector “Before Bridges, we already hired at-risk youth and we plan to continue doing so, especially if we know that they have received life/soft skills training.” –Representative from the service sector “We already hired at-risk youth before the project, for us what is important is that the applicant demonstrates interpersonal knowledge and skills in the selection process.” –Representative from the manufacturing sector In summary, employers reported they were satisfied with the content of the curricula of the training courses, the practice-based approach, and the trainees’ performance on the job. In the text box below, we present a summary of the main activities developed by the project to address the firms’ needs. How did Bridges address the firms’ needs? ✔Communicate constantly through the stakeholder advisory group. The advisory group engaged Bridges staff, TCs, the public and private sectors and was key in identifying the firms’ needs and finding ways to address these needs. ✔Design, develop, or update the curricula for the training courses in partnership with employers to ensure that youth learned the skills required in the labor market. In sectors such as ICT or textiles, the content of the courses offered is linked to the needs identified by firms in the sector. In the rest of the sectors, some employers noted that they provided feedback on the course curriculum, taking into account their human capital needs. ✔Equip training centers with resources to provide hands-on training. Trainees and employers noted that having machinery or equipment in the TCs was very useful. Trainees became more secure after having the opportunity to learn basic techniques, and the transition to learn more specialized machinery was easier. ✔Train youth with important life skills needed to succeed in the professional environment. The project provided vulnerable youth with life skills that are valued by private sector firms. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 69 4. How were gender equality, female empowerment, and social inclusion integrated into the implementation of activity interventions? Stakeholders interviewed reported that the integration of youth with disabilities occurred mainly through infrastructure improvements in TCs, and efforts to include LGBTI community in intervention activities were challenging because El Salvador still faces high levels of prejudice directed at LGBTI groups. The design and implementation of all the activities under the Bridges project were inclusive of young women (and some specifically focused on increasing female participation) but overall these efforts did not close the existing gap in employment. Each of the objectives under Bridges incorporated some aspect of gender equality and/or social inclusion: Objective 1: Enabling environment promoting WFD of vulnerable youth. Bridges provided training and technical support to firms to strengthen policies and practices on gender inclusion and diversity in their hiring policies and practices. Objective 2: Increase institutional strengthening to improve the quality of WFD services. As part of the capacity-building activities, Bridges provided training to employees of project-funded training centers on how best to foster the inclusion of female youth, including administrative staff, placement managers, and instructors. Objective 3: Improve workforce readiness demonstrated by targeted vulnerable youth. Bridges targeted enrolling equal numbers of female and male participants. The enrollment and completion of training by sector varied, but overall 49 percent of youth who completed training were female participants. Out of the total youth with employment, 48 percent are female youth. In addition, Bridges made some impressive inroads in traditionally male-dominated industries (Figure IV.5). Figure IV.5. Completion of training and youth employed, disaggregated by sex Source: Bridges monitoring data. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 70 Bridges is laying the foundation for female youth to join male-dominated labor markets. Bridges made a concerted effort with training centers to include more female youth in traditionally male-dominated industries, such as IT and energy, and worked with instructors to encourage women to enroll in and complete training. In the energy sector, 33 young women completed training in industrial electrical maintenance, electromechanical maintenance, Category 4 Electrician, and BTV in electrical engineering. In the plastics sector, 80 young women completed training in technical operator of machines, quality inspector in the manufacturing of plastic packaging, and technologies for digital pre-press (Figure IV.6). Figure IV.6. Completion of training in energy and plastics sectors, disaggregated by sex Source: Bridges monitoring data. For me (being in a male dominated career) has been a very nice experience and I am happy for what I’ve learned. I have a 20 year-old sister who wants to follow in my footsteps. She’s motivated with all the things I’ve told her and I feel proud to know that someone wants to follow my steps and I feel very happy (about that). –Dalia, Graduate of a Bridges electronics course FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 71 Breaking the mold: Female youth enter the plastics sector The plastics firm Termoencogibles has incorporated women into its workforce after collaborating with Bridges to improve the curricula of some courses. In the plastics sector, operational work was previously carried out entirely by men. In 2018, the firm hired two women for operational areas who were introduced to the company by Bridges. By January 2020, 80 women have joined the firm (15 from Bridges training courses). This change required adapting the facilities because there were previously no restrooms or locker areas for women. The experience of hiring women has been so positive that the company hopes more women will join the company when opportunities arise. B. What positive and negative results, intended or unintended, has Bridges produced at the end of the project? Overall, the project achieved positive results under the three objectives aimed to improve the workforce development environment. Among other results, the project has increased the awareness of existing laws that could benefit firms if they hire youth and has transformed the dynamics in the professional training system. The project has also improved socioemotional skills of the youth. However, constant pressure for the project to deliver results in the short run led to the perception that activities implemented before the grants manual was approved were less effective. (We have no quantitative data to assess this perception because not all grantees were required to report employment.) In Table IV.7, we summarize the positive results produced by the Bridges project. Stakeholders agreed that the project did not produce any negative results. None of the interviewed stakeholders reported any significant negative results, intended or unintended, Bridges produced at the end of the project. Table IV.7. Positive results produced by Bridges Positive results Objective 1: Enabling environment promoting WFD of vulnerable youth − Raise awareness of existing laws Objective 2: Increase institutional strengthening to improve the quality of WFD services − Shift mindset of training centers to ● offer trainings based on the needs of the companies (not the demands of trainees) ● include job placement in its services offered − Capacity building and institutionalization of job placement services FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 72 Positive results Objective 3: Improve workforce readiness demonstrated by targeted vulnerable youth − Socioemotional improvement of youth − Increase awareness that opportunities do exist in El Salvador What are stakeholders’ perceptions about those results? Which ones should be prioritized in future programs? Overall, stakeholders overwhelmingly agreed on Bridges’ focus and achievements. Based on their perceptions and our data, we believe that future programs should incorporate the following features of Bridges programing: ● Incorporating three objectives. As highlighted in Figure I.1 in the introduction, all three objectives work well together; however, they likely should be sequenced differently. Future WFD should work on laying a strong foundation first (through enabling environment and training center capacity building) before youth can really be trained. ● Phased approach. Implementing based on needs assessment ensures that project interventions are aligned with needs. It appears that the early assessments carried out by Bridges helped staff focus on activities that would yield positive outcomes. ● Training content. Trainings should include modules on both life skills and technical skills. Programs should also include theoretical and practical components that respond to the needs of the private sector. The practical components of the trainings should include hands-on modules; modules on how the industry operates; and, where feasible, internship and apprenticeship programs. ● Job placement services. WFD projects should not be limited to only training. If employment is an outcome of interest, projects should not only include job placement services for training recipients but also offer this service to vulnerable youth who may have received training elsewhere. ● Implementing through local organizations. When WFD projects are implemented through existing structures (which fosters development and sustainability), the outcomes related to youth training and employment may take years to materialize. This delay may not be palatable to some funders, and different modalities may be required in parallel to yield outcomes of interest in the early years. ● CLA. Many of Bridges’ attributes referenced above would not have materialized had Bridges and USAID not been flexible and willing to adapt activities. Including life skills in technical training and providing robust job placement services would not have been possible without this flexibility. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 73 V. FINDINGS: SUSTAINABILITY In this chapter, we assess the degree to which Bridges activities are sustainable in preparation for a planned closeout in November 2020. First, we provide a summary of how the grants under the contract mechanism addressed sustainability throughout the project. Second, we present our assessment of the degree to which the relationship between the private sector and TCs—specifically related to updating curricula—is sustainable. Third, we present an assessment of which results produced by Bridges seem to be the most sustainable. One limitation of this analysis is that the perspectives and insights from stakeholders and beneficiaries at the end of Bridges implementation may not be fully reflected because data collection occurred 10 to 13 months before the end of the project. A. Are Bridges activities sustainable? As stated above, Bridges was designed to effect lasting change by focusing on three main objectives to improve coordination and capacity of the Salvadoran workforce development. Its primary mechanism to ensure sustainability is implementing training activities through local grantees to strengthen local organizations and achieve sustained results. The training and most of the services provided by the strengthened TCs are sustainable. However, they need to keep investing in fostering the relationship with the private sector and ensuring that their offerings respond to the needs of the market so that future donors or beneficiaries will be willing to pay for their services. 1. How did the GUCs address sustainability? Table V.1 summarizes our sustainability assessment on two criteria identified as critical for the sustainability of TCs and the services they offered: (1) institutional capacity and (2) services provided. Overall, we found that the TCs strengthened institutional capacity and the provision of training and other support services have a strong potential to be sustainable in the long run, given the key sustainability elements developed during the implementation of the project. Table V.1. Sustainability assessment of TCs and the services provided Criterion Key elements to sustainability Discussion Sustainability potential Institutional capacity of TCs Institutional infrastructure – Adequate facilities and equipment – Bridges provided facility upgrades to TCs, including internet infrastructure and accessibility improvements. – Bridges also equipped TCs with tools and equipment needed to offer high-quality technical and life skills training. – Bridges and TCs jointly developed improvement plans to strengthen the institutional infrastructure. – TCs were required to sign maintenance plans to ensure new/updated equipment remained operational. – Strong (as long as TCs adhere to maintenance plans) FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 74 Criterion Key elements to sustainability Discussion Sustainability potential Organizational capacity – Standardized use of data management system to track youth beneficiaries and evaluate progress toward indicators of interest – Bridges supported the adoption of a monitoring, evaluation, and learning (MEL) culture with the institutionalization of a data management system called SisPuentes. It is expected that TCs with strengthened data management capacity will continue using the system once the project ends. – Bridges’ plan to ensure sustainability of SisPuentes was the adoption of the system by public institutions. The project has presented the benefits of the system to government institutions but as of March 2020, none of these institutions had adopted the system. – Moderate – Improved financial management capacity – Through the grants process, Bridges staff provided a lot of training, guidance, and best practices in financial management. This improved capacity will allow these organizations to continue to manage USAID grants, as well as other donor funding. – Strong Access to funding – Access to funding to support services provided – TCs reported having plans or being in the process of being authorized and certified by INSAFORP. Once TCs are authorized and certified, they can train more youth with INSAFORP funding. – Moderate Services provided Professional training – Strengthened staff capacity – Grantees were equipped with life skills instructors accredited by INJUVE. – Staff were trained in building long-term alliances with private sector firms. – Staff participated in workshops to improve instructors’ didactic skills. – Strong – Curricula aligned with labor market needs – Bridges left behind new or updated curricula for 23 modules of training courses. – Bridges supported processes to update curricula in the future. – Strong – Adoption of life skills curricula – Bridges developed a toolkit to guide TCs’ efforts to incorporate life skills into technical curricula. – INJUVE’s adoption of the new modules developed by Bridges ensures that youth will keep receiving life skills training. – Employers and TCs recognized the value of life skills training, which is key to sustainably continue offering this type of training. – Strong Education in the formal sector – Strengthened staff capacity to provide flexible education – Curricula aligned with labor market needs – Bridges supported the expansion of MINEDUCYT’s reach by strengthening two TCs authorized to provide flexible education. – Bridges supported the training and certification of staff from grantees and government officials as flexible education instructors. – Strong FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 75 Criterion Key elements to sustainability Discussion Sustainability potential Job placement services – Trained staff to provide job placement services – Each TC had a job placement manager hired by Bridges, but it is up to the TCs to retain job placement managers in their staff after the projects ends. – Moderate – Job placement support facilities – C-Orienta (one-stop shops) were established as a step toward institutionalization; however, TCs need to use their funds or have new agreements to keep offering these services. – Moderate Psychosocial assistance – Trained staff to provide assistance – Bridges strengthened the capacity of civil society organizations and government institutions providing youth with psychosocial support to improve their professional development and employment. – Bridges developed a psychosocial assistance toolkit to guide the assistance for vulnerable youth after the project ends. However, institutions need to keep investing in ensuring they have human resources to continue providing this type of support. – Moderate 2. Is the relationship between the private sector and training centers sustainable in order to continue to update curricula and trainings? Our assessment of the sustainability of the collaboration between TCs and firms to update curricula and training programs is based on (1) project activities implemented to foster a strong relationship between TCs and firms and (2) the results from the collaboration between private sector firms and TCs. Overall, we found that throughout project implementation Bridges fostered a strong relationship between TCs and firms. TCs became more aware of the benefits of engaging with the private sector, and firms learned about the types of services the TCs provided and the types of youth they trained. Being aware of each other is the first step toward ensuring a sustainable relationship. By the end of the project, both groups of stakeholders were engaged in this relationship, often communicating directly without Bridges’ intermediation. We also found that this relationship produced positive results for both parties. For TCs, relevant training programs resulted in more youth becoming interested in the courses and private firms finding applicants with knowledge and skills relevant to their industries. Given these results, it is likely that the relationship between the private sector and TCs will continue as long as the value of continued partnerships is evident to those involved. B. What results produced by Bridges seem to be the most sustainable? Objective 1: Enabling environment promoting WFD of vulnerable youth The adoption of new/improved legal frameworks to support employment opportunities for vulnerable youth is key for the sustainability of the WFD system. Bridges supported the development and increased the awareness of policies focused on reducing the systemic barriers that vulnerable youth faced to access the labor market. The approval of municipal-level policies to support youth was a key milestone to improve the enabling environment of WFD for youth in FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 76 the long run. Bridges also raised the awareness of national laws that aimed to promote youth employment. Although the future of the WFD system relies on these policies and laws being implemented and enforced, Bridges laid the foundation that was needed to take advantage of an enabling environment that promotes and fosters youth development. The collaboration between TCs and the private sector is also seemed one of the most sustainable results. As stated above, during the implementation of the project, both groups of stakeholders were engaged in the partnership to improve the professional training system. Future collaboration should focus on more internship opportunities in firms during training, given that employers and TCs agreed that more practice experience was needed. Since both stakeholders reported they perceived positive results, that collaboration may continue in the long run. However, the nature of the communication might be different because the stakeholder advisory group, which was an important communication channel, will not continue after the project ends. Despite the fact that this platform will most likely cease to exist, Bridges made a concerted effort to strengthen the relationship by allowing TCs and the private sector to interact directly without playing an intermediary role. To continue expanding the network of firms, Bridges also left behind a toolkit to guide TCs on building and fostering the relationship with the private sector (Guía de relacionamiento con empresas del sector privado). Additional support from government institutions will likely be required to replicate this partnership and collaboration in other economic sectors. Objective 2: Increase institutional strengthening to improve the quality of WFD services Grantees supported by the project have changed their approach to offering training courses and this will permeate the professional training system in the future. The focus on training youth based on the needs of the private sector is likely to be one of the most sustainable results from Bridges. TCs’ institutional capacity was strengthened to ensure availability of trained staff, adequate facilities, and tools to provide vulnerable youth with the skills the labor market values. TCs now have incentives to keep offering training programs that will equip trainees with relevant skills, because more youth will be interested in courses that have been proven to help them find jobs. Capacity building of training centers. By definition, local capacity is an important step toward sustainability and Bridges made sure that all TCs were strengthened to improve the quality and relevance of training and other services. TCs felt most strengthened in areas that required the most investment (upgraded facilities and equipment), since it is unlikely that they would have otherwise received this type of physical support. However, they also valued the support to build organizational capacity and improve instructors’ skills. In combination with the TCs’ commitment to provide regular maintenance to the upgraded facilities and equipment, improved management makes it more likely that TCs will have the facilities, learning materials, and staff capacity to continue to address the needs of vulnerable youth. Objective 3: Improve workforce readiness demonstrated by targeted vulnerable youth The project enhanced employers’ perspective of the value and potential of youth to contribute to firms. Employers and trainees agreed that the project promoted the inclusion of FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 77 vulnerable youth in the labor market by deconstructing some of the stereotypes that private firms have held about youth. Overall, representatives from private sector firms noted that they were willing to hire vulnerable youth. In sectors such as manufacturing and commerce/tourism— where employers reported that they had hired vulnerable youth before the implementation of Bridges—employers expressed their continued willingness to hire vulnerable youth because Bridges youth have been committed employees and have performed well in their tasks. Even in the ICT sector, where traditionally applicants tend to have a college or technical degree, there appeared to be more willingness to hire vulnerable youth from Software Development Centers, despite a slower change in the recruitment practices. The positive perceived contribution of youth to the labor market is sustainable as long as (1) TCs and firms continue working together to keep including the needs of the private sector in the training curricula, and (2) more changes are adopted at the policy level and better practices are adopted in companies to decrease the barriers that prevail for vulnerable youth to access the labor market. Youth improved their self-worth. Youth who participated in Bridges training programs gained skills and access to networks that will help them pursue personal and professional growth. For most trainees, life skills training helped them identify their inner strength and relate better to other people. It also gave them tools to gain self-confidence. These are skills that have benefited and will continue to benefit youth while they pursue their professional and personal development. Because life skills training was one of the most valued components of the project, by both youth and employers, TCs are interested in keeping this support as part of the professional training offering. In an interview, INSAFORP staff noted the institution was interested in integrating life skills into technical courses funded by the agency. In January 2020, INSAFORP was in the process of conducting a pilot training session with 27 instructors to assess if the institution would formally adopt the methodology. As of March 2020, INSAFORP’s decision was delayed because training was suspended due to COVID-19. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 78 VI. CONCLUSIONS, LESSONS LEARNED, AND RECOMMENDATIONS This report documents Bridges’ implementation and generates knowledge on the extent to which the project prepared vulnerable youth to increase their employability. Here we list conclusions corresponding to each evaluation question and then discuss lessons learned and recommendations for policymakers and donors. A. Conclusions and major findings Bridges experienced delays in the first years of project implementation but was able to exceed most of the life of project outcome targets, partly because of its focus on CLA. In the first years of implementation Bridges laid the foundation for the project by developing a detailed grants manual; assessing the needs of the enabling environment, training centers, private sector, and youth; and focusing on capacity building of training centers. Because of these efforts, as well as over-ambitious targets, Bridges failed to reach annual targets in the first years of implementation. However, once all activities were in place, Bridges exceeded the majority of the life of project outcome targets well before the project ended, in part because of its willingness to add and modify interventions to ensure youth were trained and found employment. Bridges’ focus on strengthening the enabling environment, training local institutions, and delivering training while ensuring participation of the private sector throughout was effective, but a modified sequence may have made implementation easier. Bridges was designed with three distinct objectives that were implemented in parallel. However Objective 1 (improve the enabling environment promoting workforce development for vulnerable youth in targeted sectors) and Objective 2 (increase institutional strengthening to improve the quality of workforce development services to effectively insert vulnerable youth into targeted sectors) were necessary conditions to move forward with Objective 3 (improve workforce readiness demonstrated by targeted vulnerable youth). It would have been more effective to only roll out Objective 3 activities once more progress was made in the first two objectives. Given the work it took to set a solid foundation under Bridges, Objective 3 activities were primarily implemented over the course of less than three years. So, a limitation of sequencing Objective 1 and 2 activities first, is that Objective 3 activities will be time-constrained given the 5-year cycle of USAID projects. Alternatively, in the future, given the work Bridges accomplished under Objectives 1 and 2, the work on strengthening the WFD system could be the focus of the first year, leaving the remaining four years to roll out market-relevant training. Bridges’ most effective interventions to help youth find employment were likely combining life skills in technical skills training and providing job placement support. Bridges made two significant design changes as a result of their CLA efforts: (1) requiring that all technical training incorporate a module in life skills, and (2) ensuring that all youth have access to job placement services. Training center staff, private sector representatives and Bridges youth confirmed how valuable it was for youth to gain these skills since they resulted in greater self-confidence and resilience. Youth also noted repeatedly how helpful it was to receive additional support to help navigate the job search process. Bridges’ influence will likely be sustainable beyond the contract period and Bridges activities embodied the spirit of USAID’s Journey to Self-Reliance. Because of the focus on FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 79 strengthening existing training centers and bringing a renewed awareness of the integral role the private sector needs to play in workforce development, many of the activities under Bridges will likely continue beyond the life of the project. In addition to introducing professional trainings that respond to the needs of the market, Bridges left behind guides and collaborations that local stakeholders can continue to take advantage of to improve workforce readiness of Salvadorian youth. Bridges also helped thousands of Salvadoran youth find their first job, giving these youth professional experience that will undoubtedly help them in the future. B. Lessons learned The project’s pivot to focus more on job placement services benefited youth and contributed to achieving the project’s employment target. Given the identified needs of youth and the positive results from providing these services, job placement services should be offered along with training as a package. Bridges institutionalized job placement services in the third year of implementation because staff identified that youth needed as much support to navigate the job search and interview process as they needed training. Given the emphasis on employment, job search support should have been integrated in the design from the beginning. These efforts seem to have accelerated employment in the last two years of the project. Incorporating life skills modules into technical training provided more youth motivation, enthusiasm and interpersonal skills which were viewed favorably by the private sector. Bridges initially offered separate tracks for technical and life skills training. However, once the training programs were rolled out, Bridges modified the curricula of the technical courses to include a module on life skills. This decision was viewed favorably by both youth and employers who noted the benefits of the newly acquired life skills. The strategy to target specific economic sectors was effective to meet employment goals, but the variation of results suggests that different eligibility criteria may be appropriate depending on the sector. The majority of Bridges trainees found jobs in the commerce/tourism and manufacturing sectors, compared to the number of youth employed in ICT and energy sectors. Overall, the focus on the commerce/tourism and manufacturing sectors, where firms hire youth with the profile of Bridges trainees, was a good approach to achieve the expected results in employment, especially given the ambitious employment target. Focusing on ICT and energy sectors – although more challenging for youth to gain employment, – was also important because these jobs represent groundbreaking opportunities for vulnerable youth where they can access better incomes. However, in these sectors, it could be useful to explore a more tailored eligibility criteria, such as considering youth with some education or raising the age limit. By altering the criteria to respond to market needs, Bridges may have been able to achieve even higher employment numbers in these sectors while still supporting vulnerable youth who likely would not be able to obtain a job without Bridges’ support. The targets in the first two years were likely overambitious given the focus on assessments and implementing activities through GUC. Given that Bridges had to work to build an enabling environment optimal for youth workforce development and to strengthen the capacity of local training centers or civil society organizations to be able to provide high quality training and services to youth, the targets for enrollment, completion, and employment in the FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 80 first years of implementation were likely overambitious and did not take into account the necessary groundwork to build a foundation necessary for future success. Implementing through local organizations fosters development and sustainability, but USAID should expect that organizations with limited USAID experience will likely take longer to achieve results. Workforce development projects willing to implement through existing organizations should realize that the outcomes related to youth training and employment may take longer to materialize and therefore yearly targets should reflect this. This delay may not be palatable to some funders, and different activities may be required in parallel to yield outcomes of interest in the early years. Lack of project-monitoring data on non-completers and youth that did not secure employment only provides a partial overview of project accomplishments. Bridges tracked youth who completed and who were employed but did not monitor the reasons for non￾completion or non-employment. There may have been positive outcomes (such as finding a job despite not completing) or return to education (instead of finding a job) that the project could not monitor, and therefore account for. Having this data provides a more complete picture of the total positive outcome of the project. C. Recommendations For USAID and other donors: Set targets that reflect the local context and determine the size of the grants program according to the targets and the capacity of existing organizations. GDP, youth employment rates, and the make-up and dynamism of an economy should be taken into consideration when setting employment targets. Because WFD interventions rarely focus on generating jobs- but rather on preparing possible employees- projects need to consider the availability of jobs when setting employment targets. Another important consideration is the types of firms driving growth. For many countries, small and medium enterprises drive employment, thus project efforts such as Bridges cannot only target large businesses when generating employment opportunities for youth. Similarly, donors need to assess the capacity of local organizations to adequately manage grants fund when determining the total size of the grants program. Future projects may consider an approach that allows for the use of local organizations while not limiting programmatic activities to those under the GUC mechanism. For USAID and donors: Working on a systems approach to WFD is effective but may be more effective if interventions are sequenced correctly. Fostering an enabling environment that values youth and their employability is important. Equally important is ensuring that the institutions engaged to train youth have the capabilities and facilities to provide quality trainings. These two components of the workforce development system are necessary to be able to roll out high quality youth trainings. Donors should focus on strengthening local capacity and then offer services to increase employability of youth, or consider separating the training interventions so that they start once other components of the workforce development system are strengthened. For USAID and other donors. Offer job placement services as an integral component to any workforce development project. If employment is an outcome of interest, job placement services should considered essential elements of job readiness training. In addition, WFD FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 81 projects should consider offering job placement services to vulnerable youth who may have received training elsewhere and/or may already have the skills required but just need support to find a job. For USAID, other donors and implementing partners: Continue to offer life skills training as part of workforce development training. Stakeholders agreed on the benefits of offering life skills training to youth who gained critical competencies demanded by employers. Employers valued hiring youth who were trained with life skills because they perceived a difference in performance. Trainees also perceived the life skills training as the most helpful support, not only in the job search process but also in developing their self-confidence. Because of the value of life skills training in improving self-confidence and resilience of youth, WFD projects should include life skills training in all programs. Many different life and soft skills curricula exist which projects can adapt to their circumstance and geographic contexts. For USAID, other donors and implementing partners: Target workforce development training and job placement services to older youth if the goal of the intervention is employment. While WFD training programs are often offered to youth under 18, the private sector in El Salvador prefers to hire youth who are older than 18 (even if by law they are allowed to employ youth younger than 18). This may also be the case in other countries/contexts. To maximize resources, workforce development training programs with a focus on employment should recruit and train youth in the age group where they are more likely to be employed. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 82 REFERENCES Ali, Mohamed. “The Link Between Unemployment and Terrorism.” TEDCity 2.0, 2013. Available at https://www.ted.com/talks/mohamed_ali_the_link_between_unemployment_and_terrorism. Accessed February 8, 2018. Almeida, Rita, Jere Behrman, and David Robalino. “The Right Skills for the Job?: Rethinking Training Policies for Workers.” World Bank Publications, 2012. Bagby Emilie, E, Felix, P. Costa, N, Murray, and B. Seiler. First Annual Central America Workforce Development Report. Submitted to USAID/Washington. Washington, DC: Mathematica, July 2018. Available at https://pdf.usaid.gov/pdf_docs/PA00T8JM.pdf. Bagby Emilie, E, Felix, P. Costa, N, Murray, and B. Seiler. Second Annual Central America Workforce Development Report. Submitted to USAID/Washington. Washington, DC: Mathematica, August 2019. Available at https://pdf.usaid.gov/pdf_docs/PA00W77J.pdf. Bagby Emilie, E, Felix, P. Costa, N, Murray, and B. Seiler. Third Annual Central America Workforce Development Report. Forthcoming. DAI. “El Salvador Employment Training Project Startup Work Plan: October 1–December 31, 2015.” Report prepared for USAID. El Salvador: DAI, November 2015. DAI. “El Salvador Employment Training Project Year 1 Work Plan: October 1, 2015–September 30, 2016.” Report prepared for USAID. El Salvador: DAI, January/February 2016a. DAI. “USAID Bridges to Employment Semi-Annual Report: October 1, 2015-March 31, 2016.” Report prepared for USAID. El Salvador: DAI, April 2016b. DAI. “USAID Bridges to Employment Year Two Work Plan: October 1, 2016–September 30, 2017.” Report prepared for USAID. El Salvador: DAI, September 2016c. DAI. “Draft USAID Bridges to Employment Semi-Annual Report: April 1, 2016-Setember 30, 2016.” Report prepared for USAID. El Salvador: DAI, October 2016d. DAI. “Draft USAID Bridges to Employment Semi-Annual Report: April 1, 2017-Setember 30, 2017.” Report prepared for USAID. El Salvador: DAI, October 2017a. DAI. “USAID Bridges to Employment Semi-Annual Report: October 1, 2016-March 31, 2017.” Report prepared for USAID. El Salvador: DAI, April 2017b. DAI. “USAID Bridges to Employment Semi-Annual Report: October 1, 2017-March 31, 2018.” Report prepared for USAID. El Salvador: DAI, April 2018a. DAI. “USAID Bridges to Employment Year Three Work Plan: October 1, 2017–September 30, 2018.” Report prepared for USAID. El Salvador: DAI, May 2018b. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 83 DAI. “USAID Bridges to Employment Semi-Annual Report: April 1, 2018-September 30, 2018.” Report prepared for USAID. El Salvador: DAI, October 2018c. DAI. “USAID Bridges to Employment Year Four Work Plan: October 1, 2018–September 30, 2019.” Report prepared for USAID. El Salvador: DAI, November 2018d. DAI. “USAID Bridges to Employment Semi-Annual Report: October 1, 2018-March 31, 2019.” Report prepared for USAID. El Salvador: DAI, April 2019a. DAI. “USAID Bridges to Employment Semi-Annual Report: April 1, 2019-September 30, 2019.” Report prepared for USAID. El Salvador: DAI, October 2019b. DAI. “USAID Bridges to Employment Year Five Work Plan: October 1, 2019–September 30, 2020.” Report prepared for USAID. El Salvador: DAI, November 2019c. DAI. “USAID Bridges to Employment Semi-Annual Report: October 1, 2019-March 31, 2020.” Report prepared for USAID. El Salvador: DAI, May 2020. Dirección General de Estadística y Censos (DIGESTYC). “Encuesta de Hogares de Propósitos Múltiples. 2014 Standard of Living Survey.” San Salvador: DIGESTYC, 2018. Gates, Sarah, Laura Lippman, Noel Shadowen, Holly Burke, Obed Diener, Morrisa Malkin, and FHI 360. “YouthPower Action: Key Soft Skills for Cross-Sectoral Youth Outcomes.” Report prepared for USAID. Washington, DC: YouthPower Action, 2016. Gregg, Paul, and Emma Tominey. “The Wage Scar from Male Youth Unemployment.” Labour Economics, vol. 12, no. 4, 2005, pp. 487–509. International Monetary Fund. “Scarred Generation.” Finance and Development, vol. 49, no. 1, March 2012. Olenik, Christina, and Caroline Fawcett. “State of the Field Report: Examining the Evidence in Youth Workforce Development.” Washington, DC: USAID, February 2013. Policia Nacional Civil. “Por estas razones están bajando los homicidios: Duro golpe a las estructuras criminales durantes 2016-2018 y 2019.” San Salvador: Policia Nacional Civil, 2019. The World Bank. “The World Bank in El Salvador: Overview.” January 2019a. Available at http://www.worldbank.org/en/country/elsalvador/overview. Accessed March 19, 2020. Ugaz, Jorge, Larissa Campuzano, and Nancy Murray. “Final Performance evaluation plan for Puentes para el Empleo” Mathematica, 2019. Available at https://pdf.usaid.gov/pdf_docs/PA00WPV3.pdf. USAID. “El Salvador FY2016 Full Performance Plan Report (PPR).” Washington, DC: USAID, March 2017. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA 84 USAID. “The Journey to Self-Reliance.” August 2019. Available at https://www.usaid.gov/selfreliance. Accessed August 24, 2019. United States Government (USG). “U.S. Government Strategy on International Basic Education: Fiscal Years 2019-2013.” Washington, DC: USG, 2018. This page has been left blank for double-sided copying. APPENDIX A ADDITIONAL TABLES AND FIGURES This page has been left blank for double-sided copying. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA A.3 1. Changes in evaluation questions Table A.1. Changes in evaluation questions Original evaluation questions Change in the question (if applicable) Justification 1. What were key barriers/facilitators to achieve project results? (a) Focus on design and innovation, actors involved, program rollout, and external factors (environment). (a) To what extent do these barriers and facilitators include focus on design and innovation, actors involved, program rollout, and/or external factors (environment)? Clarity (b) How have the grants under contract (GUC) mechanism advanced, or detained, implementation and the strengthening of local solutions? (b) N/A Clarity (c) How did the GUCs address sustainability? (c) N/A Clarity 2. To what extent has Bridges reached outcome targets in these areas: (a) Employment, improved skills, and educational outcomes among beneficiaries (a) Educational outcomes, improved skills and employment among youth Clarity (b) Strengthening of TCs (b) N/A (c) Number of new/revised certified programs (c) N/A (d) Private sector inclusion of vulnerable youth in their staff (d) N/A (e) Sustainability of private sector and training centers relationship to have updated curricula and trainings (e) Sustainability of private sector and training centers relationship in order to have updated curricula and trainings Clarity 3. What positive and negative results, intended or unintended, has Bridges produced at the end of the project? What are stakeholders’ perceptions about those results? Which ones should be prioritized in future programs? Which ones seem to be the most sustainable? What results produced by Bridges seem to be the most sustainable? Clarity 4. Perceptions from employers, trainees and grantees: (a) What are employers’ perceptions about: i. Trainees regarding technical skills and knowledge? Life skills? On-the-job performance? i. Trainees’ technical skills and knowledge? ii. Trainees’ Life skills? iii. Trainees’ on-the-job performance for current position or potential for upward mobility? Clarity Clarity Intended purpose ii. Training centers and their improved curricula? iv. N/A iii. Private sector’s own hiring practices? v. Their willingness to hire youth from high-risk areas? Intended purpose (b) What are Bridges’ trainees perceptions about: i. The training they received, including technical training and life skills training, when applicable? i. N/A ii. How useful has the training/certificate been in helping them get a job? ii. How useful the training/certificate has been in helping them get a job? Clarity FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA A.4 Original evaluation questions Change in the question (if applicable) Justification iii. How useful were the support services in getting a job and succeeding at it?; How has Bridges has improved their future prospects in the labor market?; Is there more openness from the private sector to hire youth?; What are the perceived before and after effects given the project’s intervention? iii. How useful the support services were in getting a job and succeeding at it? iv. How Bridges has improved their future prospects in the labor market? v. Is there more openness from the private sector to hire youth (particularly vulnerable youth)? vi. N/A Clarity (c) What are grantees’ perceptions about: i. Has engagement with private sector made a difference in the education approach? i. N/A ii. In what areas was your institution most strengthened? ii. In what areas were your institutions most strengthened? Clarity iii. What approaches, technical assistance and activities have been most critical for vulnerable youth job placement success? iii. N/A 5. Additional questions to consider How were gender equality, female empowerment, and social inclusion integrated in the implementation of activity interventions? Were those efforts successful? Why or why not? N/A FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA A.5 2. Grant description, activities funded and grantees Table A.2. Grant description, grant-funded activities, and grantees RFA Grant description Capacity building activities provided to TCs (Objective 2) Activities offered by TCs to improve youth readiness (Objective 3) Grantees Value of the grants approved 001 Grants for technical training services and the strengthening of TCs - Strengthen the organizational capacity building of TCs’ staff and operations - Provide technical, life skills, and labor orientation training to vulnerable youth - SSPAS - Fe y Alegria - ESFE-AGAPE - INTI $977,933 $1,330,69 $1,000,588 $652,842 002 Grants for the development of software development centers (SDCs) - Establish fully equipped SDCs - Train youth on software development and on six programming languages - Facilitate job placement for trained youth - USAM - UNIVO - FGK $927,930 $888,932 $1,328,926 003 Grants for training centers authorized by INSAFORP to develop 40 training courses for youth - Create or update curricula for training courses - Strengthen the organizational capacity building of TCs - Provide technical and life skill training - Offer internship and job placement opportunities - AGAPE - OEF - CAPUCOM $46,553 $140,904 $30,623 005 Grants for life skills trainings for at-risk youth - Strengthen TCs, human talent management - Train vulnerable youth in life skills - CONEXION - ASAPROSAR - FUNPRES - FUSALMO - Talento Humano - Fe y Alegria - FEPADE $38,420 $35,820 $32,083 $34,930 $28,506 $78,871 $120,851 006 Grants for blended high school education modality for youth - Strengthen the capacity of the TCs through training of instructors - Assist the completion of high school education of vulnerable youth - Provide life skills training - FIECA - Fe y Alegria $338,857 $184,798 007 Grants for workforce development in the plastics industry - Provide machinery and laboratory equipment to TCs - Integrate new teaching methods and methodologies into curricula - Offer specialized training for vulnerable youth in seven technical careers in the plastics industry - FUNDEPLAST (in kind) - FUNDEPLAST (standard) $1,845,256 $542,420 008 Grants for wrap-around services for vulnerable youth and facilitating their incorporation into the workforce - Strengthen the capacity of the TCs through training instructors and administrative staff - Offer counseling, job readiness, and support services - Establish partnerships to implement changes in private sector companies - MyV - UEES $528,793 $285,177 009 Grants for workforce development in the textile and apparel industry - Strengthen TCs through training of instructors and administrative staff - Renovate training facilities and provide equipment, training supplies, and materials - Provide high quality technical and life skills training for youth - Fe y Alegria - AGAPE $423,290 $620,702 Unsolicited proposal - Strengthen the capacity of CSOs and training centers - Train vulnerable youth in technical and life skills - Facilitate the employment of trainees through www.alempleo.org - FUNDEMAS $498,013 Note: In 2017, Bridges published a request for applications (RFA 004) to fund virtual high school education. In El Salvador, Universidad Francisco Gavidia (UFG) is the only educational institution accredited by the Ministry of Education, Science, and Technology (MINEDUCYT) to provide virtual high school education. After several rounds of negotiations with UFG FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA A.6 regarding its proposal for RFA 004, Bridges was not convinced that the proposal and budget aligned with the project’s objectives. RFA = request for application; SSPAS = Passionate Social Service; ESFE-AGAPE = Specialized Franciscan High School; INTI = National Technical Industrial Institute; USAM = Alberto Masferrer Salvadoran University; UNIVO = Eastern University; FGK = Gloria de Kriete Foundation; OEF= Association for the Organization and Education of Women Entrepreneurs of El Salvador; CAPUCOM = Professional Training in Computing; CONEXIÓN = Connection to Development Association of El Salvador; ASAPROSAR = Salvadoran Association of Rural Health; FUNPRES = Salvadoran Education Foundation; FUSALMO = Salvador del Mundo Foundation; FEPADE = Entrepreneurial Foundation for Development; FIECA = Central American Innovative Education Foundation; MyV = Goals and Vision (Metas y Vision); UEES = Evangelical University of El Salvador; FUNDEPLAST = Foundation for the Integral Development of Workers of the Plastic Industry; FUNDEMAS = Business Foundation for Social Action. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA A.7 3. Additional information on the industry-recognized certifications and accreditations developed Bridges developed 18 certified training programs and supported the accreditation of one TC. Bridges collaborated with TCs, firms, industry associations and INSAFORP to update existing training curricula for two courses in the textile sector and one course in the energy sector. These courses were recognized by INSAFORP, which is an important step towards sustainability because all INSAFORP-authorized TCs in El Salvador can offer this training course. Bridges also developed new training curricula on six different computer-programming specializations to align with the current and future needs of the IT industry. With three different levels of training offered for each specialization, the new curricula have resulted in 18 industry￾recognized, computer-programming certifications. Centro de Formación Social Pasionista received the accreditation from INSAFORP that certifies that the training center has the facilities, instructors, and curricula to offer training programs. In August 2020, MINEDUCYT approved the updated curricula for the commerce and tourism program developed by ESFE￾AGAPE technical baccalaureate (See table A.3). Table A.3. Industry-recognized certifications and accreditations developed Curricula Recognized by Sector Date Certified updated programs Maintenance & Repair of Industrial Apparel Machines: INSAFORP Textile 14-08-19 Operation of Industrial Apparel Machines INSAFORP Textile 31-05-19 Category 4 Electricians INSAFORP Electricity 31-05-19 Bartender waiter DISAL Commerce/tourism 14-12-18 Commerce and tourism program in technical baccalaureate MINEDUCYT Commerce/tourism 31-08-20 Certified new programs (18 total) Junior programmer C#MVC – 1 curricula developed for each of the 3 levels CASATIC ICT 18-08-18 Junior programmer C# XAMARIN -- 1 curricula developed for each of the 2 levels CASATIC ICT 18-08-18 Junior programmer XAMARIN -- 1 curricula for Level 3 CASATIC ICT 18-08-18 Junior programmer PHP-- 1 curricula developed for each of the 3 levels CASATIC ICT 18-08-18 Junior programmer in JAVA -- 1 curricula developed for each of the 3 levels CASATIC ICT 18-08-18 Junior programmer in CS HTMLCUT (Web designer) -- 1 curricula developed for each of the 3 levels CASATIC ICT 18-08-18 Junior programmer in CS HTMLCUT (Web designer) Level 3 CASATIC ICT 18-08-18 Junior programmer analyst (Tester de Software) -- 1 curricula developed for each of the 3 levels CASATIC ICT 18-08-18 Accreditation of training centers Accreditation of Centro de Formación Servicio Social Pasionista INSAFORP Commerce/tourism 23-01-19 Source: Bridges monitoring data. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA A.8 4. Additional information on progress on implementation targets for Objective 1 Bridges activities under Objective 1 were implemented according to its timeline, but with some delays. The project had some delays in the rollout of activities. As shown in Table A.4, implementation targets for activities aiming to improve the WFD environment—for example, the number of firms implementing changes or regulation adopted to improve WFD— were not met until the second year of program implementation (fiscal year [FY] 2017). The delays to achieve annual targets in this objective are due to the time it took to identify practices or regulations that needed to be improved and the advocacy efforts needed so firms and local authorities could support the recommended changes to their practices. Table A.4. Progress on implementation targets for activities under Objective 1 FY2016 FY2017 FY2018 FY2019 FY2020* Achieve to date LOP target Number of firms implementing changes or new practices, as a result of Bridges project 0 14 3 23 16 56 50 Number of laws, policies, or procedures proposed or adopted to improve or increase workforce development 0 3 4 5 24 36 12 Value (in US $) in target leveraged amount (cash and in-kind) from the private sector and other donors to contribute to preparing training at￾risk youth for employment 0.5 million 1.1 million 1.6 million 2.4 million 2.7 million 8.3 million 5 million 5. Additional information on progress to implementation targets for objective 2 Bridges activities to strengthen TCs were implemented according to its timeline. Most of the activities to strengthen training centers (Objective 2) rolled out as planned—namely, implementation goals related to strengthening training providers were achieved in the first year of project implementation. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA A.9 Table A.5. Progress on implementation targets for activities under Objective 2 FY2016 FY2017 FY2018 FY2019 FY2020* Achieve to date LOP target Number of training center, NGO, CSO, private sector, and/or public sector staff/employees enrolled in Bridges training n.a 526 249 1,334 348 1,952 1,400 Number of direct partnerships between WFD service providers and private sector companies 3 19 10 32 43 107 40 Number of organizations and/or service delivery systems that serve vulnerable populations strengthened 2 13 17 11 0 43 40 Number of WFD service providers with an established tracking system that monitors employment outcomes n.a n.a 6 14 2 22 18 n.a. = not available; NGO=Nongovernmental organization; CSO= Civil society organization; WFD= Workforce development. 6. Additional information on progress to implementation targets for Objective 3 Bridges activities to increase youth readiness and improve access to employment were delayed. As mentioned in chapter III, for activities under Objective 3, the implementation of the project was delayed as a result of the delay in the approval of the grants manual and the time needed to build local capacity among grantees. However, as we will explore further, all indicators were met in the last year of project implementation (FY2020). Bridges project achieved its goal of indirect people benefitted by the project. As shown in Figure A.1, by the end of the fourth year of project implementation (FY2019), 40,863 individuals benefited from the project. Indirect beneficiaries are not project participants of training programs, who have benefited from the project activities such as attendance to Bridges recruitment or informal events or received information from a C-Orienta but did not continue to directly benefit from the project. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA A.10 Figure A.1. Number of indirect vulnerable people benefitting from Bridges project Bridges project met its goal of people benefitted by the project. In FY2019, Bridges achieved the target to benefit 18,000 individuals from the project. Bridges met the target set for the first year of project implementation and was very close to meet target in second and third year of the project. Vulnerable people refers to youth who received support from an implementing partner. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA A.11 Figure A.2. Number of vulnerable people benefitting from Bridges project Bridges project achieved its goal of youth trained in life or leadership skills in December 2019. As shown in Figure A.3, during the second year of project implementation (FY2017), 925 youth received life skills training. In the third and four year of project implementation 3,176 and 3,558 youth were trained in life or leadership skills. Figure A.3. Number of youth trained in social or leadership skills through Bridges project FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA A.12 Bridges project achieved its goal of youth trained with improved knowledge. As shown in Figure A.4, a total of 11,187 youth improved their skills or knowledge after completing a training course. Improved skills are measured by a pre-post assessment designed and develop by TCs. Youth with a higher score on the skills post-test are counted. Figure A.4. Number of youth with improved knowledge or skills following completion of Bridges training Bridges met the target for the number of previously out-of-school youth who reported to be enrolled in formal school. The project met its goal for previously out-of-school youth enrolled in formal education— including high school, vocational/ technical school, or university. By July 2019, the project met its goal of at least 7,002 youth who previously dropped out from school, enrolled in formal education. Nearly 9,775 previously out-of-school youth reported to be enrolled in formal education. FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA A.13 Figure A.5. Number of previously out-of-school youth who report enrolling in formal school Bridges achieved its goal of percent of youth with new or better employment. In March 2020, Bridges met its goal of 40 percent youth employed after completion of the training. However, during the April to July period, approximately 400 youth completed their training program, but due to COVID 19 and the subsequent mobility restrictions, the job search process was halted. As result, as of July 2020 the employment target dropped to 39.3% and is now 0.7 percentage points below the target. Bridges did not meet annual target in the previous years, employment was affected by delays in enrollment and completion of training (Figure A.6). FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA A.14 Figure A.6. Percent of individuals with new or better employment following completion of Bridges training Bridges did not meet the target to have at least half of female participants in training programs. This is the only LOP target that the project will fell short by the end of the contract. The training completion rate for women was 48% very close to the target of 50%.These results are consistent with our findings from qualitative data collected. Stakeholders agreed that the participation of female youth in these sectors was challenging due to a perception that women are not able to perform on these sectors (Figure A.7). Figure A.7. Percentage of female participants in Bridges project FINAL PERFORMANCE EVALUATION: BRIDGES TO EMPLOYMENT IN EL SALVADOR MATHEMATICA A.15 7. Additional information on strengthening provided to TCs As mentioned in chapter III, findings form the initial assessment in FY2017 of twenty service providers shaped the project’s work to improve the capacity of training centers and the quality and relevance of training available to at-risk youth. Improving institutional capacity takes time, but the assessments conducted in FY2018 for three training CAPUCOM, AGAPE, and OEF revealed that training centers made progress. As shown in Figure A.8 all three training centers improved their overall scores, CAPUCOM improved visible in student services, and equipment and material, AGAPE made important improvements in equipment and materials and organization and administration, OEF improved its finance, and overall institutional infrastructure. Figure A.8. Improvements in TVET-CAT scores 72 77 83 84 82 83 62 67 58 61 69 70 45 58 56 59 35 37 30 35 43 48 39 47 26 32 31 36 32 38 25 44 46 54 47 49 24 30 39 47 38 39 16 21 20 25 21 24 12 15 12 14 13 13 0 50 100 150 200 250 300 350 400 450 500 Baseline assessment Follow up assessment Baseline assessment Follow up assessment Baseline assessment Follow up assessment CAPUCOM AGAPE OEF TVET-CAT score Training programs Management of staff and faculty Student Services Finance Infrastructure Equipment & materials Organization and administration Philosophy and objectives Library & learning Resources 312 379 388 428 376 400 This page has been left blank for double-sided copying. APPENDIX B DESIGN REPORT This page has been left blank for double-sided copying. Final Performance Evaluation Plan for Puentes para el Empleo March 2019 This publication was produced for review by the United States Agency for International Development. It was prepared by Jorge Ugaz, Larissa Campuzano, and Nancy Murray of Mathematica Policy Research. This page has been left blank for double-sided copying. Final Performance Evaluation Plan for Puentes para el Empleo DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This page has been left blank for double-sided copying. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH vi CONTENTS I. INTRODUCTION.............................................................................................................................. 1 II. DESCRIPTION OF PUENTES AND RESEARCH QUESTIONS .................................................... 3 A. Goals and objectives ................................................................................................................. 3 B. Initial assessments and activities .............................................................................................. 3 C. Evaluation questions ................................................................................................................. 6 III. EVALUATION DESIGN: DATA AND ANALYSIS ............................................................................ 8 A. Qualitative data........................................................................................................................ 10 B. Quantitative data ..................................................................................................................... 11 C. Analysis plan............................................................................................................................ 11 IV. DATA COLLECTION PLAN ........................................................................................................... 14 A. Qualitative data collection........................................................................................................ 14 B. Quantitative data collection ..................................................................................................... 14 V. TIMELINE, DELIVERABLES, AND PERSONNEL ........................................................................ 16 A. Timeline and deliverables........................................................................................................ 16 B. Personnel................................................................................................................................. 16 This page has been left blank for double-sided copying. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH viii TABLES Table III.1. Evaluation questions and data sources for final performance evaluation of Puentes ................ 9 Table IV.1. Main outcomes or indicators from M&E data to be provided by Puentes ................................ 15 Table IV.2. Plans for qualitative and quantitative data collection................................................................ 15 Table V.1. Evaluation timeline .................................................................................................................... 16 This page has been left blank for double-sided copying. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 1 I. INTRODUCTION Youth unemployment is a critical global challenge. In 2014, almost 73 million of the world’s young people were unable to find work—an unemployment rate of 13.0 percent (International Labor Organization 2015). More than 350 million of the world’s youth were economically disengaged, and more than 500 million were underemployed (World Economic Forum 2014). There is consensus that joblessness among youth has long-lasting consequences for individuals, their families, and society as a whole. To address this, many youth workforce development (WFD) programs around the world have been established, focused on providing technical skills, soft or life skills, and workforce development services. A large body of literature documents evidence on the effectiveness of many WFD programs in both developed and developing countries. Several reviews (Kluve et al. 2016; McKenzie 2017; Olenik 2013; Tripney et al. 2013) have looked at dozens of studies and found mixed evidence on the programs’ impacts on employment outcomes. Kluve and colleagues conclude that WFD development programs have increased the employment and income of youth who participate in them, but impacts have been small and vary widely by the type of program and country context. The authors suggest that skill training programs are effective in increasing employment outcomes, whereas other programs such as subsidized employment are not. In Latin America, a few rigorous impact evaluations of skill training programs have found small effects on employment and income for at least some subgroups (Acevedo 2017; Attanasio et al. 2011; Betcherman et al. 2004; Card et al. 2011; Ibarraran et al. 2014; Ibarraran and Rosas Shady 2009). These training programs provide a combination of technical skills training, soft skills training, and internships. Although there is no rigorous evidence confirming that programs need to provide both technical and soft skills training to be effective, employers have identified soft skills training as the most valuable component (Ibarraran et al. 2015). Furthermore, researchers have also identified soft skills as an important factor in employment success (Acevedo et al. 2017; Heckman et al. 2006; J-PAL 2017; Kautz et al. 2014; Rankin et al. 2015). The United States Agency for International Development, Bureau for Latin America and the Caribbean Bureau (USAID/LAC) is playing a key role by funding the implementation of workforce development projects in Central America. The Bridges to Employment project in El Salvador (referred to throughout this document by its Spanish name, Puentes (bridges), is one of USAID/LAC’s largest investments in workforce development: it is a five-year, $42.2 million project implemented by Development Associates International (DAI) and its subcontractors, Plan International and JBS International, Inc. The Puentes program is designed to increase employment levels for at-risk youth and help them get better jobs. Puentes has adopted practices that have shown promise for achieving the latter, such as providing both technical and soft skills training. Furthermore, Puentes plans to improve the capacity of training providers and promote the link between training providers and future employers. Assessing the needs of employers will allow Puentes to provide training courses that are more aligned to employers’ needs, thereby increasing the chances that trainees will find jobs. Since 2015, Mathematica has been an active learning partner of USAID/Washington and USAID/El Salvador, conducting comprehensive reviews of the literature on youth workforce development programs and identifying gaps. At the end of 2015, through the LAC Reads EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 2 Evaluation contract, USAID/El Salvador asked Mathematica to propose rigorous impact designs and explore the possibility of conducting an impact evaluation of Puentes. In March 2016, Mathematica staff met with DAI and USAID staff in El Salvador to present our preliminary thoughts for a potential evaluation design (see Appendix C for the two meeting presentations, and a debrief presentation focused on a potential evaluation design). During early 2017, Mathematica met on different occasions with Puentes staff to discuss some of the key parameters required for an impact evaluation to be feasible and to obtain more detailed information about the rollout of the program and, in particular, of the training courses. Mathematica then updated the impact evaluation design accounting for Puentes implementation plans (see Appendix D). In June 2017, once the early program design activities were complete and Puentes had shared detailed information about its upcoming activities, USAID, Mathematica, and Puentes agreed to conduct a pilot randomized controlled trial (RCT) to test the processes and feasibility of random assignment. The pilot was successfully implemented between July and September 2017, and the process we designed for random assignment worked (see Appendix E for Mathematica’s memo summarizing the findings). Mathematica therefore concluded that it was possible to implement a three-arm RCT design and updated the power calculations (see Appendix F). However, during the first few weeks of 2018, Mathematica learned about the continuing evolution of the implementation of Puentes and identified some feasibility issues for the RCT. Mathematica shared a memo summarizing these issues, which were related to the grant-making process implemented by Puentes and the separate rollout of technical and soft-skills courses by the training centers (see Appendix F). After discussions on this topic, in April 2018 USAID/El Salvador decided to pursue instead a mid-term performance evaluation of Puentes. On July 2018, Mathematica submitted a draft of the evaluation plan for this mid-term evaluation. However, after further discussion, in November 2018 USAID/El Salvador opted to pursue a final performance evaluation, which will take place from September 2019 through June of 2020. This final performance evaluation will provide insights and generate learning on whether Puentes reached its targets and key objectives, stakeholders’ perceptions about Puentes’ implementation and impact, and what lessons have been learned that might inform the next generation of WFD programs in El Salvador and Central America. This evaluation plan lays out the design for this final performance evaluation. The rest of this document is organized as follows. In Section II, we describe the key activities implemented by Puentes and present a logic model developed to reflect our understanding of the program we are using to guide the design of the evaluation, including the research questions to be addressed by the evaluation. In Section III we discuss the evaluation design, focusing on data requirements and analysis. Section IV describes the key outcomes and data collection strategy, and Section V presents a proposed timeline, deliverables, and key personnel. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 3 II. DESCRIPTION OF PUENTES AND RESEARCH QUESTIONS In this section, we describe the goals and objectives of Puentes and the activities corresponding to them, and then present the research questions to be addressed through this evaluation. A. Goals and objectives Puentes’ central goal is to increase and improve employment of vulnerable youth1 ages 16 to 29 who live in one of 15 high-crime municipalities. 2 The project has three main objectives: (1) improve the enabling environment for youths’ WFD programs and youth employment through better laws and hiring practices, (2) improve the quality of WFD services so they effectively respond to market demand and help employ greater numbers of vulnerable youth in the targeted economic sectors, and (3) improve the workforce readiness of targeted vulnerable youth. The project focuses on working with employers and helping youth develop skills they need for the high-growth sectors of agroindustry, manufacturing (including plastics), information communications technology (IT), tourism, and renewable energy. To complement this evaluation plan, we propose a logic model to guide the Puentes project evaluation (see Figure B.1 in Appendix B). This logic model is based on Puentes’ own documentation, including its monitoring and evaluation (M&E) plan, work plans, semiannual reports, and presentations; it allows us to depict how the program’s objectives and activities are expected to result in outputs and outcomes that we can measure as part of this evaluation. B. Initial assessments and activities Puentes was designed to implement activities for each of its three key objectives in a systematic and mutually reinforcing way. During the first year of activities, Puentes carried out four needs assessments and used the findings to design the project’s main activities going forward. For Objective 1, Puentes carried out a policy assessment analysis to understand challenges the project might face during its execution, and mapped the policies, regulations, laws, and private sector practices that can hinder vulnerable youth’s access to employment or training. Based on findings from a workshop that took place in El Salvador in June 2016, Puentes developed a labor market assessment. This assessment ensured that Puentes activities were based on up-to-date information about industry growth and labor market demand for specific education and technical skills, thereby allowing Puentes to better target program training and provision of support services and to identify the type of partnerships that would work best for its goals. For Objective 2, Puentes implemented an assessment of workforce training service providers to identify their strengths, efficiencies, and areas for improvement. The assessment found that courses being offered by those providers did not match the needs of the labor market, 1 Puentes defines vulnerable youth as those “in need of technical training, education, psychosocial support, or related assistance to complete their education or to find, keep and/or improve employment” (DAI 2016b). 2 The 15 municipalities include Ahuachapán, Ciudad Arce, Ciudad Delgado, Cojutepeque, Colón, Jiquilisco, Mejicanos, San Miguel, San Salvador, Santa Ana, Santa Tecla, Sonsonate, Soyapango, Tonacatepeque, and Zacatecoluca. As of April 2018, no training has taken place in Cojutepeque; however, training has been provided in four other smaller municipalities (San Martín, Chalchuapa, Sonzacate, and Sacacoyo). EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 4 and that providers had weak organizational management capacity. Finally, for Objective 3, Puentes carried out a youth assessment in order to better understand the risks, challenges, and barriers Salvadoran youth face when trying to enter the labor market, as well as the role of violence in their daily lives. Below, we discuss some of the key activities Puentes has implemented or plans to implement in order to achieve the three primary objectives of the program. Activities for Objective 1: Improved enabling environment for workforce development and employment. Based on the findings of the policy assessment analysis and the labor market assessment, Puentes identified the economic sectors with high growth and employment potential for disadvantaged youth: agroindustry, manufacturing, IT, tourism, and renewable energy.3 Through quarterly meetings with a Stakeholder Advisory Group, Puentes developed strategies for approaching key industry associations within those sectors to understand their labor needs and build relationships for potential collaboration, including the sharing of hiring projections. Puentes also elicited the private sector’s help to engage in policy reform, raise awareness, and promote national laws to reduce legal barriers to youth employment in El Salvador. Puentes has also (1) promoted laws on the inclusion of youth and other vulnerable groups through municipal youth policies, (2) developed a collaboration with the National Council for the Attention to People with Disabilities (CONAIPD), and (3) conducted an awareness-raising workshop on LGTBI (lesbians, gay, transsexual or transgendered, bisexual, and inter-sexed) and gender inclusion for USAID/El Salvador staff members. Special attention was placed on the inclusion of young women, the LGTBI community, youth with disabilities, and ex-gang members, through the implementation of inclusion workshops for private sector firms and complementary interventions for training centers. Puentes has also helped several civil society organizations (CSOs) be better prepared to support vulnerable youth. Further, Puentes is actively working to improve perceptions about the value of young Salvadoran employees among key firms within the five high-growth economic sectors. In order to increase hiring of youth, Puentes has offered workshops on best practices for more inclusive youth employment, including a strategic communications campaign targeting businesses and the Salvadoran general public. Puentes has also been actively fostering stronger linkages between training centers (TCs) and more than 50 private firms to provide youth with internship and apprenticeship opportunities. The nature of these programs varies for each of the five key economic sectors but they all seek to strengthen the links between private firms and TCs in a way that will help more young Salvadorans enter the labor market. Activities for Objective 2: Improved quality of workforce development services that effectively respond to market demand to insert at-risk youth into target economic sectors. Puentes used the results from the local workforce development service providers’ assessment to develop a strategy to give the providers technical assistance through a grant process. As of May 2018, Puentes has worked with a total of 25 training centers, and some of them have received technical assistance and capacity building support from the project, covering areas such as contract requirements, use of pre- and post-test assessments, M&E requirements, management 3 Initially, the sectors identified were tourism/services, food and beverage, business services, and manufacturing. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 5 systems, organizational structure and culture, equipment and facility upgrades, among others. Puentes has been using the technical vocational education and training capacity assessment tool (TVET-CAT4 ) to assess the institutional capacity, strengths, and weaknesses of training centers and identify opportunities for improvement. By Project Years 4 and 5, Puentes expects to have provided technical assistance to a total of 20 service providers and also to have helped 30 service providers create partnerships with private firms for recruitment of trainees. As of April 2018, out of 3,905 youth who registered, more than 2,800 had completed one of the 188 training courses offered through project-funded training centers; 2,529 vulnerable youth received some training on life skills and/or technical training. For the year 2018, Puentes has scheduled more than 100 courses, with an important proportion of those offering training in plastics and IT. Most of the technical and soft-skills trainings being provided by the training centers last between 8 and 12 weeks, with the exception of a few courses on IT and the plastics industry that last between six and nine months. Puentes expects to provide training or support services to 16,000 youth before the project ends in 2020, with around 42 percent of youth who complete the course projected to obtain new jobs, or better ones than they had before the project. All youth participating in technical and soft skills trainings receive stipends ($5 to $7 per day) to be used for transportation, food, and child care. Activities for Objective 3: Improved workforce readiness demonstrated by targeted at￾risk youth. For this objective, Puentes conducted a participatory youth assessment that looked at barriers youth faced in accessing education; their opportunities to take advantage of training, employment, and entrepreneurship opportunities; their issues with social risk and discrimination; and so on. As noted, in addition to providing technical training that responds to the demands of the private sector and other labor market actors, Puentes offers life skills training to vulnerable youth. Puentes developed and implemented its own modules on life skills training according to the requirements and needs articulated by the private firms the project is working to serve. This new curriculum drew from content provided by the training program Elijo mi futuro (“I choose my future”), the life skills program developed by the Instituto de la Juventud (INJUVE), and the modules on creative conflict resolution developed by Fundación Pro Educación de El Salvador (FUNPRES). Puentes adapted some of those materials to its specific context and needs, and instructed training centers on how to adapt and impart the new life skills curriculum. For this objective, Puentes led and/or participated in more than 70 outreach and recruitment events where project staff shared information about the project’s training activities and screened more than 5,000 interested youth for eligibility (screening includes their completing an intake form and a psychometric test or a personal interview carried out by Puentes). Puentes also is offering career counseling and mentoring services for vulnerable youth to improve their chances of finding and performing well at a high quality job.5 Puentes is planning to offer select youth entrepreneurs seed capital and start-up mentoring services. Puentes is also planning to select and 4 The TVET-CAT tool is used to evaluate service providers; Puentes used the initial results to develop a customized capacity-building plan for each service provider. 5 Career counseling and mentoring services for beneficiaries include (1) providing a brochure on career opportunities, (2) developing “business cafes” for sharing knowledge and creating peer networks, and (3) holding job orientation sessions that will include counseling on labor market demand, mentoring with recruitment managers, and registering on online job databases. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 6 develop some service providers as “one-stop shop” activity hubs to coordinate WFD services, including additional training, information on job fairs, resume writing, interview preparation, and entrepreneurship guidance, among others. By the end of Year 5 of the project, Puentes expects to have benefitted approximately 18,400 youth through training and other services and to have directly helped 6,000 of them find new or better employment. C. Evaluation questions The evaluation is designed to describe and assess the performance of Puentes and generate knowledge on the progress Puentes made toward meeting its targets. The evaluation will assess which aspects of the program worked well, which could have been improved, what barriers or facilitators seem to have helped or hindered its implementation, and what are key stakeholders’ perceptions about the impact and benefits of the services provided by the project to prepare vulnerable youth to enter into and succeed in the labor market. USAID/El Salvador and USAID more broadly, as well as policymakers interested in developing similar programs, can benefit from the findings. The evaluation is expected to be finalized by June 2020. The objectives of the evaluation are: • To assess and generate learning on how the project met its performance targets. • To provide insights on what factors helped or hindered project results and implementation. • To assess the extent to which the project prepared vulnerable youth to face and succeed in the labor market. • To provide USAID with recommendations and lessons to inform the next generation of programs that will promote positive youth development and workforce development in El Salvador. In particular, the main research questions to be addressed include: 1. What were key barriers and facilitators to the achievement of project results? a) To what extent do these barriers and facilitators include focus on design and innovation, actors involved, program rollout, and/or external factors (environment)? b) How have the grants under contract (GUC) mechanism advanced, or detained, implementation and the strengthening of local solutions? c) How did the GUCs address sustainability? 2. To what extent has Puentes reached outcome targets in the following areas: a) Employment, improved skills, and educational outcomes among beneficiaries b) Strengthening of training centers c) New/revised certified programs d) Private sector inclusion of vulnerable youth in their staff e) Sustainability of private sector and training centers relationship to have updated curricula and trainings? 3. What positive and negative results, intended or unintended, has Puentes produced at the end of the project? a) What are stakeholders’ perceptions about those results? Which ones should be prioritized in future programs? Which ones seem to be the most sustainable? EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 7 4. What are the perceptions from employers, trainees and grantees regarding the training program and the support services provided by Puentes? a) What are employers’ perceptions about: i. Trainees regarding: Technical skills and knowledge? Were employers’ demands met with the current technical skills, or do they require more specific technical skills (formal education technical skills such as 2yr degrees, or non-formal training such as professional/vocational technical skills short term trainings)? Life/soft skills? On-the-job performance for current position or potential for upward mobility? ii. Training centers and their improved curricula regarding: Courses and curricula taught in response to private sector demand; does a sustainable demand-driven model exist between education service providers and private sector to inform relevant institutions responsible for training demand? iii. Private sector: Is the private sector more open now to hire youth from high-risk areas? b) What are Puentes’ trainees perceptions about: i. The training they received, including technical training and life/soft skills training, when applicable? ii. How useful the training/certificate has been in helping them get a job? iii. How useful the support services were in getting a job and succeeding at it? iv. How Puentes has improved their future prospects in the labor market? v. Is there more openness from the private sector to hire youth? vi. What are the perceived before and after effects given the project’s intervention? c) What are grantees’ perceptions about: i. Has engagement with private sector made a difference in the education approach? ii. In what areas was your institution most strengthened? iii. What approaches, technical assistance and activities have been most critical for vulnerable youth job placement success? 5. Additional questions to consider How were gender equality, female empowerment, and social inclusion integrated in the implementation of activity interventions? Were those efforts successful? Why or why not? In the next section, we describe the data collection and analysis plans designed to address these key research questions. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 8 III. EVALUATION DESIGN: DATA AND ANALYSIS To answer the research questions laid out in the previous section, Mathematica will conduct a performance evaluation of Puentes. Performance evaluations are not designed to detect project impacts or attribute changes in participant outcomes to the project or specific components of it, due to the lack of a suitable comparison group or counterfactual for recipients of their services. However, a performance evaluation is a powerful way to generate and share valuable learning and insights about the implementation and potential effects of Puentes as a whole. In coordination with USAID/Washington, USAID/El Salvador, and Puentes, Mathematica will carry out this performance evaluation relying on mixed methods and using both qualitative data (obtained through a desk review, key informant interviews, and focus group discussions) and quantitative data (mainly M&E data and administrative records from Puentes). In the next two sections, we describe the qualitative and quantitative data we will collect. Table III.1 details the research questions to be addressed as well as the data source or type of respondent. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 9 Table III.1. Evaluation questions and data sources for final performance evaluation of Puentes Qualitative sources Research questions Quantit ative M&E data1 KII w/ firms KII Puentes staff KII TC Directo r staff Desk review2 FGDs w/ instructors FGDs w/ trainees and graduates 1. What were key barriers/facilitators to achieve project results? a) Focus on design and innovation, actors involved, program rollout, and external factors (environment). X X X b) How have the grants under contract (GUC) mechanism advanced, or detained, implementation and the strengthening of local solutions? X X X c) How did the GUCs address sustainability? X X X 2. To what extent has Puentes reached outcome targets in these areas: a) Employment, improved skills, and educational outcomes among beneficiaries X X X X b) Strengthening of TCs X X X X c) Number of new/revised certified programs X X X X d) Private sector inclusion of vulnerable youth in their staff X X X e) Sustainability of private sector and training centers relationship to have updated curricula and trainings X X 3. What positive and negative results, intended or unintended, has Puentes produced at the end of the project? What are stakeholders’ perceptions about those results? Which ones should be prioritized in future programs? Which ones seem to be the most sustainable? X X X X X 4. Perceptions from employers, trainees and grantees: a) What are employers’ perceptions about: i. Trainees regarding technical skills and knowledge? Life or soft skills? On-the￾job performance?; ii. Training centers and their improved curricula?; iii. Private sector’s own hiring practices? X X X b) What are Puentes’ trainees perceptions about: i. The training they received, including technical training and life skills training, when applicable?; ii. How useful has the training/certificate been in helping them get a job?; iii. How useful were the support services in getting a job and succeeding at it?; How has Puentes has improved their future prospects in the labor market?; Is there more openness from the private sector to hire youth?; What are the perceived before and after effects given the project’s intervention? X X X X c) What are grantees’ perceptions about: i. Has engagement with private sector made a difference in the education approach?; ii. In what areas was your institution most strengthened?; iii. What approaches, technical assistance and activities have been most critical for vulnerable youth job placement success X X X 5. Additional questions to consider How were gender equality, female empowerment, and social inclusion integrated in the implementation of activity interventions? Were those efforts successful? Why or why not? X X X X X 1 M&E data will include inputs provided by Puentes to Mathematica for the Workforce Development report. 2 Includes review of project documents, action plans, internal reports, assessments carried out during Year 1 of the project (Policy Assessment, Labor Market Assessment, Service Provider Assessment, and Participatory Youth Assessment), and other relevant sources. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 10 A. Qualitative data We will collect and use three types of qualitative data: (1) a comprehensive desk review of Puentes quarterly and annual reports and assessments or special studies; (2) semi-structured, in￾person interviews with key informants (KIIs) including staff from Puentes, some training centers, and some large employers of Puentes trainees; and (3) focus group discussions (FGDs) with instructors from TCs and with Puentes current and former trainees. These data will help us understand how the project has been implemented, which barriers and facilitators may have affected implementation, what stakeholders perceive as the effects of the intervention, what beneficiaries think about the services and training received, and what adjustments might improve the project in the future. 1. Desk review. This review will include Puentes’ annual and semiannual reports, work plans, M&E plans and reports, and the four early assessments, as well as other documents critical to understanding Puentes’ implementation. The reports will also be important in identifying facilitators and barriers to implementation. We will consider requesting access to the reports submitted to Puentes by the TCs who are reporting aggregate information on the beneficiaries, provision of services, challenges faced, and other information; this could provide valuable insights into challenges faced by the training centers and help shape our qualitative data collection protocols. 2. Key informant interviews. We will conduct open-ended, semi-structured, in-person interviews with three of respondents; the findings will help us answer research questions 1 to 5. First, we will interview staff from Puentes in charge of project activities linked to the provision of technical assistance to TCs, technical and soft skills training, and support services to vulnerable youth, as well as ones linked to partnerships between TCs and private firms. Second, we will interview either the director or a representative from a purposively selected sample of TCs participating in the Puentes project. We will seek interviewees’ perspectives on barriers and facilitators that directly affected their work and their capacity to partner with private businesses and adjust to a changing labor market. We will choose the final sample of TCs in coordination with Puentes, making an effort to cover training across the key economic sectors, the two types of training (technical and life skills training), and as many municipalities as possible. To capture perspectives on the work done by Puentes regarding institutional strengthening, awareness raising and enabling environment for youth employment, we may also interview representatives from a few key CSOs. To address research questions 2, 3 and 4, we plan conduct interviews with representatives of private firms that have hired Puentes trainees. We will ask respondents about Puentes outreach activities, partnerships between their firms and Puentes or training centers, and how they have benefited from the project. It will also ask them to assess the job performance, technical knowledge and skills, and life or soft skills of Puentes’ trainees, perhaps asking employers to compare them with non-Puentes workers. We will define the final list of private firms in coordination with Puentes, based on the firms that have hired most Puentes trainees, and work to include firms across key economic sectors and municipalities. Ideally, the point of contact within the private firms will be a person in a managerial position, either in the Human Resources department or someone having regular, direct interaction with the trainee. See Table IV.2 for details on the number of private firms that will be approached for the interviews. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 11 3. Focus group discussions. Findings from FGDs will help us answer research questions 2, 3, 4, and 5. We will seek three types of participants for the FGDs: first, we will visit training centers and talk with instructors at the TCs to discuss the training being provided, their perceptions on trainees’ skill improvement and potential, and challenges they face during the provision of training. Second, during the visits to the training centers we will hold FGDs with youth currently receiving training under Puentes, where they will share their experiences and overall perception and satisfaction with the training (including content and organization of the courses) and the support services being received from TCs. Third, we will conduct focus group discussions with former Puentes trainees or graduates where they will share their perceptions on how Puentes training have help them in their job search, current employment, and general job performance. B. Quantitative data M&E data from Puentes. The project has been collecting M&E data on project outputs and outcomes for reporting purposes, showing how activities are being rolled out, the types of services provided, the geographic reach and gender breakdown of beneficiaries, and whether implementation targets are being met, as well as indicating whether interventions are generating their intended effects. We will use Puentes M&E data to answer research question 2; to the extent possible, we will also look at performance indicators. For the full list of indicators, please refer to Bridges for Employment’s Monitoring and Evaluation Plan (2019). C. Analysis plan In this section, we discuss the main analytical tools and data sources we will use to address the key research questions. Table III.1 breaks down all the research questions and matches them with the data type and sources we propose using to address them. The desk review, KIIs, and FGDs will provide a wealth of information for qualitative analysis. We will analyze this information and identify patterns of consensus, instances of divergent or contradictory views, and variation across dimensions of program implementation as well as across program stakeholders. We will accomplish this through two primary methods: (1) thematic framing, and (2) data triangulation. Thematic framing. To uncover patterns, themes, and issues in the qualitative data, we will develop a coding framework to capture a hierarchy of conceptual categories and classifications that are linked to the research questions and the logic model proposed in Section II.1. We will update this coding framework as we systematically review and assess the qualitative data according to Puentes’ logic model. Assigning codes to the qualitative data will enable us to efficiently access and organize information obtained through the KIIs and FGDs in order to identify themes and compile supporting evidence. For instance, representatives from different TCs may describe their experience in various ways that reflect similar underlying themes, which can be captured in our coding framework. Conversely, Puentes staff may view preparedness for the job market differently than their former trainees do; our coding framework can classify those different perspectives. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 12 Data triangulation. Because our qualitative analysis will incorporate data from several different sources (and could even include some quantitative data, like M&E indicators), we will use triangulation to test for consistency in the findings from these data sources. This process will allow us to confirm patterns or findings and identify important discrepancies. For example, when we investigate employers’ perceptions of Puentes trainees and the training centers, we will triangulate their responses with the results from focus groups with trainees, interviews with staff from Puentes and the TCs. The goal of this exercise is to develop a comprehensive understanding of the project’s potential effects by identifying instances in which qualitative findings corroborate, contradict, or help contextualize quantitative findings. Findings from the qualitative analysis using thematic framing and data triangulation will allow us to answer specific questions. For example, for questions 4.a, 4.b, and 4.c, we will combine responses from Puentes staff and TC directors with findings from the desk review to obtain an in-depth understanding on the nature, magnitude, and perceived effectiveness of the support provided by Puentes to the TCs to improve their capacity to establish and maintain partnerships with the private sector, and whether—and to what extent—Puentes has been able to transfer some of that capacity and responsibilities to the centers. This type of analysis will also allow us to identify both barriers—factors that complicated the project’s implementation—and facilitators—factors that enhanced Puentes’ effectiveness. For research question 1, we will focus on specific areas including design and rollout of training courses and support services, characteristics of training centers and trainees, and external or environmental factors. We will conduct quantitative descriptive analysis to address research questions for which quantitative data are available. For research question 2, for example, we will use M&E data to assess the extent to which Puentes reached its target objectives. M&E data will allow us to describe trends across several variables or indicators, such as graduation and employment rates. We will also use the responses to the interviews to employers to capture their perceptions about Puentes’ trainees, the curricula provided to them, and the training centers. For example, for research question 4, we will ask for employers’ satisfaction with the performance of Puentes trainees who have been hired by their firms, including not just their overall performance and technical skills and knowledge, but also their life skills and their ability to follow rules and engage with co-workers in a respectful manner. We may ask employers to compare Puentes trainees with other workers performing tasks at the same level of responsibility or technical difficulty, regardless of whether or not they have received similar training. We will use mixed-methods analysis for questions for which we have both quantitative and qualitative data. For example, we will conduct a thematic analysis of the data from interviews and focus groups, sorted by research question and by Puentes’ three objectives, to identify key themes that will help contextualize the progress shown in the M&E data and project reports. For example, in interviews with Puentes and TC staff, we may learn that logistical and contracting delays, or weak capacity of TCs affected progress towards reaching the target number of courses and the number of people receiving training. We will also use data from the key informant interviews and focus group discussions to distill stakeholders’ perceptions on the quality of implementation and to complement project reports and M&E data on the quantity of activities implemented (research question 4). EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 13 Importantly, the accounts from Puentes’ staff or instructors on the nature and quality of the trainings and support services will be compared with the accounts from trainees, who will also be asked about their overall satisfaction with the project. For research question 2 (whether Puentes reached its target outcomes), we will triangulate qualitative findings with quantitative data, including administrative and M&E data, to conclude whether Puentes has reached its target outcomes, and which barriers or limitations might have played a role in the last phase of the project. Understanding challenges, the decisions made about what work to prioritize as Puentes draws to a close, the implications of any activities that were not completed, and the lessons learned through the implementation of the project will all be valuable inputs for this evaluation. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 14 IV. DATA COLLECTION PLAN A. Qualitative data collection We will conduct one round of qualitative data collection. Mathematica staff, assisted by a local consultant, will lead the interviews and focus groups. The interviews and focus groups will be recorded, and interviewers will carefully take notes that will be transcribed later. For the focus groups, a local consultant will help with logistical support and bring important knowledge of the local context and target participants to help guide the discussion. The first round of qualitative data collection is scheduled for October 2019 and will include (1) interviews with Puentes key staff in charge of activities linked to the three Objectives; (2) interviews with representatives of a selected group of TCs; and (3) focus groups with instructors and current trainees. For the focus groups, we will carefully schedule visits to training centers in order to recruit youth currently receiving the training and capture their perceptions about the training and services provided by Puentes. Puentes has agreed to give us the necessary contact information (see Table IV.2 for the estimated number of interviews and focus groups by type of respondent and round.) The second round of data collection is tentatively scheduled for January 2020 and will include interviews with (1) CSO and NGOS, (2) focus groups with Puentes graduates, (3) interviews with employers, and (4) interviews with Puentes key staff. B. Quantitative data collection We will review all indicators for monitoring and evaluation compiled by Puentes since the beginning of the project, according to its M&E Plan (therefore, we will not conduct any primary data collection for those indicators). We expect to have access to up-to-date indicators as soon as they are available. Table IV.1 shows the main outcomes or indicators that we will be targeting using M&E data. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 15 Table IV.1. Main outcomes or indicators from M&E data to be provided by Puentes Data source Outcome or indicator M&E data (indicators) Private sector • Number of firms implementing new practices for vulnerable youth • Number of private firms with awareness training about vulnerable youth • Number of industry-recognized certifications and/or accreditations Training centers • Number of training centers with technical assistance from Puentes • Number of training centers strengthened by Puentes • Number of training centers with partnerships with private sector Vulnerable youtha • Number of eligible youth registered for training • Percentage of registered youth who are female • Number and percentage of eligible youth graduated from training • Number of individuals who benefitted from Puentes • Number of individuals who indirectly benefitted from Puentes • Number and percentage of trainees with new or better employment • Number and percentage of trainees with improved knowledge or skills • Number and percentage of trainees who report enrolling in formal school a An additional indicator may be salary or income, as reported by the trainees to the training centers. We will use this indicator depending on the data’s availability and quality and the speed with which the data are reported. For the specific case of trainees on Plastics, we will consider the fact that, after completion, trainees are expected to be hired for pasantías (internships) for a period of three to five months depending on the type of course received. This will restrict the number of trainees—especially from the plastics sector—whom we could use to track salary improvements. Table IV.2. Plans for qualitative and quantitative data collection Type of data Source of data Respondent Sample size* Qualitative data Interviews Puentes staff 5 Qualitative data Interviews CSO and NGO representatives 3 Qualitative data Interviews Training center representatives 6 Qualitative data Interviews Employers 4-5 large employers Qualitative data Focus groups Instructors 2-3 groups, 4-5 instructors each Qualitative data Focus groups Current trainees 3-4 groups, 5-10 trainees each Qualitative data Focus groups Graduates 3-4 groups, 5-10 trainees each Quantitative data M&E data Puentes n.a. Note: Interviews and focus groups will take place in February 2020. n.a. = not applicable. * Final numbers for both rounds to be determined in coordination with Puentes. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 16 V. TIMELINE, DELIVERABLES, AND PERSONNEL A. Timeline and deliverables Table V.1 shows the timeline for the evaluation, with key tasks for data collection, analysis, and reporting, by month and year. Table V.1. Evaluation timeline Task 1. Instrument development and data collection (September 2019 – January 2020). In September and October, we will prepare the instruments and protocols for the interviews and focus groups, which we will share with USAID/El Salvador and USAID/Washington, and Puentes. The first round of interviews and focus groups will be carried out in October 2019 and the second round is planned for January of 2020. Task 2. Analysis and reporting (December 2019 – March 2020). Analysis of quantitative data could begin as early as December 2019 and will continue with the qualitative analysis until March 2020. In March 2020, we will start writing the final report. We will submit a draft of the final report to USAID/El Salvador and USAID/Washington by May 2020 for comments and feedback, which we will address for the submission of the final report by June 2020. B. Personnel The performance evaluation of Puentes will require dedicated U.S.-based and in-country team members with subject matter expertise in workforce development projects, performance evaluation design and analysis, and management of data collection efforts. Our U.S.-based staff bring the experience and skills required to successfully lead this evaluation. We have strong management, communication, and coordination capabilities; deep experience in performance evaluation; subject expertise in workforce development, and a positive working relationship with Puentes and USAID/ES. The staff includes the following: • Dr. Nancy Murray will provide overall management advice and leadership, and serve as point of contact with USAID/Washington, USAID/El Salvador, and Puentes project director. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 17 • Dr. Larissa Campuzano will manage the project’s financial and staffing needs and provide technical advice to Ms. Costa on tasks related to design, data collection, analysis, and reporting. • Ms. Patricia Costa will lead the evaluation design’s implementation, analysis, and reporting tasks. She will closely coordinate with Puentes staff, and will manage the local consultant. • Mr. Camila Fernandez will lead the development of qualitative instruments and advise on qualitative analysis. • Other Mathematica staff may include a junior analyst who will support Ms. Costa in the development of instruments, help support analysis tasks, and work on final reports. • A local consultant will help coordinate and co-lead the interviews and focus groups with Ms. Costa. U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 Tel: (202) 712-0000 Fax: (202) 216-3524 www.usaid.gov APPENDIX C PROTOCOLS AND FGD GUIDES This page has been left blank for double-sided copying. 40152 PUENTES MATHEMATICA C.2 PUENTES CAPACITADOS ACTIVOS – PROTOCOLO PARA GRUPO FOCAL (2019) Presentación Gracias por tomarse el tiempo y aceptar reunirse con nosotros hoy. Mi nombre es [NOMBRE DEL ENTREVISTADOR] y soy parte del equipo que colabora con el proyecto de USAID Puentes para el Empleo (PUENTES) y que estudia programas para la juventud en América Latina. Estamos visitando diferentes centros de formación para conocer la opinión de los estudiantes sobre los programas de formación que buscan mejorar la preparación de los jóvenes para que se incorporen al mercado laboral. El propósito de esta conversación es conocer su experiencia como jóvenes beneficiarios del proyecto PUENTES. Vamos a iniciar una conversación grupal guiada que tardará más o menos 90 minutos. Les haremos algunas preguntas y la idea es que ustedes las contesten libremente y compartan sus opiniones. No hay respuestas correctas o incorrectas. Lo que nos interesa es conocer su experiencia y opinión. Tengan en cuenta que lo que ustedes compartan con nosotros en esta reunión se mantendrá en estricta confidencialidad. La información que obtengamos durante esta conversación se utilizará exclusivamente para el estudio del proyecto PUENTES. No se presentará información personal que los identifique a usted individualmente sino de manera agregada con las contribuciones de los otros jóvenes en otras zonas del país. Es decir, las opiniones que ustedes expresen en esta conversación no estarán asociadas a ustedes de forma personal o con sus nombres propios. Quisiéramos grabar la conversación, de modo que podamos representar con la mayor precisión posible la información que nos proporcionen. Si ustedes están de acuerdo grabaremos la conversación, pero el audio será compartido sólo con las personas que realicen el trabajo de transcripción y con el equipo de investigación. De igual manera, en cualquier momento me puede pedir que deje de grabar. Una vez termine el estudio, los audios serán borrados. Sus nombres serán excluidos de las transcripciones, así como de cualquier documento. ¿Están de acuerdo? __ Sí __ No ¿Tienen alguna pregunta antes de comenzar? __ Sí __ No [SI TODOS LOS PARTICIPANTES ESTÁN DE ACUERDO, INICIE LA GRABACIÓN. SI AL MENOS UNO NO ESTÁ DE ACUERDO, TOME NOTAS DETALLADAS EN LUGAR DE GRABAR] A. PERCEPCIÓN DE LA CAPACITACIÓN A JÓVENES Empecemos presentándonos con nuestro nombre y con él/los cursos o curso en los que cada uno de ustedes está estudiando. 1. ¿Dónde escucharon sobre el proyecto PUENTES? ¿Cómo supieron del proyecto? • ¿Qué los motivó a inscribirse en PUENTES? • ¿Consideraron otras opciones / cursos / programas? ¿Cuáles? • ¿Hasta el momento qué es lo que más les gusta del curso apoyado por PUENTES? y ¿lo que menos les gusta? • ¿Tuvieron alguna dificultad para inscribirse o participar en el curso? ¿cuál? ¿cómo la superaron? 2. ¿Cómo escogieron el curso en el que están inscritos (TICs/plásticos/textiles/comercio o servicios)? ¿Por qué escogieron ese sector? • ¿Tuvieron la opción de elegir cualquier tema de estudios que quisieran? ¿Por qué sí/no? 40152 PUENTES MATHEMATICA C.3 • ¿Creen que hay diferencia en los temas de estudio que escogen sus compañeros (hombres) y compañeras (mujeres)? ¿sí/no, por qué? 3. ¿Qué tipo de empleo aspiran conseguir al terminar sus cursos de formación técnica? • ¿Alguno quisiera continuar estudiando algún programa de educación superior, técnica o universitaria? ¿sí/no? ¿por qué? 4. ¿Cómo les parece la forma de enseñanza del curso que cuenta con teoría y práctica en las empresas? ¿Podrían dar unos ejemplos del vínculo entre lo que aprenden en sus clases y en la práctica (empresa, prácticas profesionales)? ¿Podrían dar algunos ejemplos? • ¿Cuáles ventajas y desventajas han observado de este modelo de formación teórico￾práctico? • ¿Consideran que el modelo funciona o creen que habría aspectos del modelo que se podrían mejorar para facilitar la transición de los jóvenes a la vida laboral? ¿Cuáles aspectos sugiere usted para mejorar el modelo? 5. ¿Además de los cursos, que otros servicios del proyecto PUENTES han utilizado? (Habilidades para la vida y el trabajo, circuitos de orientación laboral, asesoría laboral para mejorar el CV, apoyo para el fortalecimiento personal para la entrevista laboral, acompañamiento a la entrevista, inserción laboral, entre otros). • ¿Qué les ha gustado de esos servicios? ¿Qué podría mejorar? 6. ¿En qué medida creen que las actividades del proyecto PUENTES ayudan a los jóvenes con discapacidad o que pertenecen a grupos LGBTI? ¿El proyecto tiene en cuenta las necesidades de jóvenes con discapacidad? ¿Podrían dar ejemplos? Ahora hablemos más en detalle sobre los cursos a los que asisten en este centro de formación. 7. ¿Los cursos a los que asisten incluyen formación técnica y formación en habilidades para la vida y el trabajo? • ¿Cómo les ha parecido la formación técnica que recibieron? ¿Piensan ustedes que les ha ayudado o les ayudará a conseguir empleo? • ¿Cómo les han parecido los servicios o talleres para desarrollar habilidades para la vida y el trabajo? ¿Les han ayudado esos servicios? ¿Qué habilidades para la vida sienten que han adquirido específicamente? ¿Pueden dar algunos ejemplos? • ¿En qué aspectos de la formación sienten que necesitan más apoyo o ayuda? ¿Pueden dar unos ejemplos? 8. Pensando específicamente en los métodos de enseñanza que los instructores utilizan en sus cursos (por ejemplo, dar clase, explicar en la pizarra, mostrar videos, hacer trabajo en grupo, organizar experimentos, etc.) ¿qué actividades les parecen más útiles para aprender? • ¿Qué tipo de oportunidades para practicar o aplicar lo que han aprendido en clase han tenido? 9. ¿Cómo les han parecido el nivel de preparación de sus instructores? ¿Les han ayudado los cursos a adquirir nuevo conocimiento y habilidades técnicas? ¿Cómo? • ¿Cuáles creen que son las fortalezas y debilidades de los cursos? 10. ¿Creen que los cursos los están preparando bien para el trabajo que aspiran obtener en su sector? ¿Por qué sí/no? 11. ¿Pueden describir que materiales, maquinaria, equipos, o instalaciones (taller, laboratorios) utilizan para los cursos PUENTES? • ¿Han tenido suficientes oportunidades de usar esos recursos y acceso a los materiales, o instalaciones que necesitan? ¿Por qué sí/no? 40152 PUENTES MATHEMATICA C.4 • ¿Qué tanto han tenido que utilizar este tipo de recursos en sus cursos? ¿Han tenido acceso a esos recursos en el centro? ¿Por qué sí/no? • ¿Son adecuadas las instalaciones del centro para sus cursos? ¿Por qué sí/no? ¿Pueden dar algunos ejemplos? ¿Han enfrentado retos con respecto a los materiales o instalaciones? Si es así, por favor dar ejemplos. 12. ¿Han tenido oportunidades a través de PUENTES de interactuar con futuros empleadores? • ¿Qué tipo de oportunidades? ¿Pueden dar algunos ejemplos? 13. ¿Han podido atender las clases y cumplir con los requisitos de los cursos? ¿Por qué sí/no? • ¿Qué dificultades han tenido con la asistencia o con el cumplimiento de los trabajos, tareas o requisitos del curso? • ¿Alguna vez tuvieron la intención de abandonar el programa? ¿Por qué? 14. ¿Conocen ustedes los otros servicios que tienen a su disposición a través de PUENTES o de este centro de formación apoyado por PUENTES, además de los cursos? ¿Qué saben de esos servicios? • ¿Qué servicios o tipos de apoyo han utilizado o recibido? • ¿Hay algún otro tipo de apoyo que no hayan recibido, pero quisieran recibir? 15. ¿Consideran que la capacitación y los apoyos de Puentes mejorarán sus perspectivas para conseguir un trabajo? ¿Por qué? • ¿En qué medida creen que PUENTES les ayudará a cumplir sus metas futuras? ¿Pueden dar unos ejemplos? 16. ¿En qué medida creen que los jóvenes enfrentan barreras para conseguir empleo por ser hombre/mujer, por tener una discapacidad o por su lugar de residencia? ¿Pueden dar unos ejemplos? • ¿En qué medida creen que PUENTES les puede ayudar a los jóvenes a enfrentar ese tipo de barreras? ¿Pueden dar unos ejemplos? B. CIERRE 17. ¿Pensando en general en el programa PUENTES, están satisfechos? ¿Qué le cambiarían? ¿Por qué? 18. ¿Qué hubiesen hecho si no hubiera existido el programa Puentes? • ¿A qué se hubiera dedicado? ¿Hay alguna pregunta que no les haya hecho que ustedes creen que sería importante para nuestra evaluación? This page has been left blank for double-sided copying. 40152 PUENTES MATHEMATICA C.6 PUENTES DIRECTORES DE CENTROS DE FORMACION – ENTREVISTA (2019) Presentación Gracias por tomarse el tiempo y aceptar reunirse con nosotros hoy. Mi nombre es [NOMBRE DEL ENTREVISTADOR] y soy parte del equipo que colabora con el proyecto de USAID Puentes para el Empleo (PUENTES) y que estudia programas para la juventud en América Latina. Estamos visitando diferentes centros de formación para conocer la opinión de los directores y coordinadores sobre los programas de formación que buscan mejorar la preparación de los jóvenes para que se incorporen al mercado laboral. El propósito de esta conversación es conocer su experiencia como director o coordinador de este centro de formación apoyado por el proyecto PUENTES. Vamos a iniciar una entrevista que tardará más o menos 90 minutos. Le haremos algunas preguntas y la idea es que usted las conteste libremente y comparte sus opiniones. No hay respuestas correctas o incorrectas. Lo que nos interesa es conocer su experiencia y opinión. Tenga en cuenta que lo que usted comparta con nosotros en esta reunión se mantendrá en estricta privacidad. La información que obtengamos durante las entrevistas se utilizará exclusivamente para el estudio de PUENTES. No se presentará información personal que lo identifique a usted individualmente. Los resultados cualitativos se presentarán de manera agregada con las contribuciones de los otros participantes. Es decir, las opiniones que usted exprese en esta conversación no estarán asociadas a usted de forma personal o con su nombre propio. Quisiéramos grabar la conversación de modo que podamos representar con la mayor precisión posible la información que nos proporcione. Si usted está de acuerdo, grabaremos la conversación, pero el audio será compartido solo con las personas que realicen el trabajo de transcripción y con el equipo de investigación. De igual manera, en cualquier momento me puede pedir que deje de grabar. Una vez termine el estudio, los audios serán borrados. Su nombre será excluido de las transcripciones, así como de cualquier documento. ¿Está bien? __ Sí __ No ¿Tiene alguna pregunta antes de comenzar? __ Sí __ No [SI EL PARTICIPANTE ESTÁ DE ACUERDO, INICIE LA GRABACIÓN. SI NO ESTÁ DE ACUERDO, TOME NOTAS DETALLADAS EN LUGAR DE GRABAR] B. PERCEPCION SOBRE EL APOYO A LOS CENTROS DE FORMACIÓN Empecemos hablando sobre las actividades implementadas por PUENTES para fortalecer los cursos de formación. 1. ¿Cuál cree usted que fue la motivación de este centro de formación para participar en el proyecto PUENTES? • En general, ¿cuáles han sido los retos y los beneficios principales para el centro de participar en una subvención de PUENTES? 2. ¿Podría describir en qué ha consistido el apoyo o la ayuda técnica que este centro de formación ha recibido del proyecto PUENTES para mejorar los aspectos administrativos, organizacionales, o técnicos? • ¿En qué medida cree que se han fortalecido las funciones técnicas y programáticas, como planificación de lecciones, gestión del conocimiento, diseño y desarrollo del plan de estudios y monitoreo y evaluación? ¿Podría dar algunos ejemplos? • ¿En qué medida se han fortalecido la participación y alianzas del sector privado? ¿Podría dar algunos ejemplos? 40152 PUENTES MATHEMATICA C.7 • ¿En qué medida se han fortalecido y los procesos de acreditación y estándares de calidad? ¿Podría dar algunos ejemplos? • ¿En qué medida se han fortalecido el conocimiento y las prácticas de inclusión y diversidad? ¿Podría dar algunos ejemplos? 3. ¿Qué tipo de asistencia ha recibido el centro en cuanto al diseño curricular, y a la planeación de los temas y las actividades de las clases? • ¿Podría describir las etapas en que se implementa y los equipos encargados en cada etapa (Puentes o equipo interno)? 4. ¿Qué tipo de apoyo han recibido para mejorar sus recursos didácticos, equipos e instalaciones como infraestructura de Internet, computadoras, equipos técnicos y de instrucción, y suministros y muebles básicos? • ¿Recibieron apoyo específicamente para mejoras en la accesibilidad de las instalaciones para personas con discapacidad, como rampas, puertas más anchas, baños? ¿Por qué sí/no? 5. ¿Qué tipo de apoyo que han recibido del proyecto PUENTES para la asistencia y seguimiento de los jóvenes graduados de los cursos, por ejemplo, bases de datos para darle seguimiento a la colocación laboral o para conectar a los jóvenes con oportunidades laborales o de emprendimiento? • ¿En qué medida se han fortalecido estos aspectos del trabajo del centro de formación? 6. ¿Cómo ha sido el proceso de evaluación del centro antes del proyecto PUENTES y durante la implementación de las actividades de asistencia técnica? • ¿Qué ventajas y desventajas ha tenido del proceso de evaluación para el centro? • ¿Han conocido los resultados de la evaluación del centro? (por qué sí/no) ¿Qué perspectiva tiene sobre esos resultados? ¿En qué medida le ha ayudado a usted como director del centro conocer esos resultados? • ¿Qué elementos de gestión organizacional aún se necesitan fortalecer en este centro? 7. ¿Pueden describir los métodos o enfoques de instrucción que utilizan en sus cursos? ¿En qué aspectos se diferencian los cursos apoyados por PUENTES de otros cursos de educación técnica vocacional que usted conozca o haya enseñado? • ¿Cuáles son las fortalezas y debilidades de los cursos de PUENTES? • ¿Qué retos han enfrentado enseñando los cursos de PUENTES? 8. ¿Qué tipo de seguimiento se le ha hecho a este centro de formación, como por ejemplo: 1) visitas de campo para darles orientación, 2) verificar la implementación adecuada y recomendar medidas correctivas, 3) oportunidad para capacitaciones y reforzamiento de los temas (M&E, gestores de empleo, liquidaciones, etc)? ¿Qué organización ofreció ese apoyo? • ¿Qué ventajas y desventajas ha tenido ese seguimiento? • ¿En qué medida se han beneficiado usted como director o coordinador, o los instructores del seguimiento? 9. ¿Sabe de alguna colaboración del centro de formación con el sector privado? ¿Cómo se ha reflejado esta colaboración con el sector privado en el enfoque de capacitación? 40152 PUENTES MATHEMATICA C.8 C. PERCEPCIONES SOBRE LA CAPACITACION A JÓVENES Ahora, hablemos sobre las actividades desarrolladas en este centro de formación para ayudar a los jóvenes desarrollar habilidades para la vida y el trabajo. 10. ¿En los cursos que se dictan como parte del proyecto PUENTES, cómo se implementan los cursos que cuentan con teoría y práctica en las empresas? ¿Podrían dar algunos ejemplos? • ¿Cuáles ventajas y desventajas han observado de estos cursos que cuentan con teoría y práctica en las empresas? • ¿Qué aspectos del modelo se podrían mejorar para facilitar la transición de los jóvenes a la vida laboral? • ¿En qué medida cree que este centro utiliza un enfoque de inclusión y diversidad? ¿Se adecúa o tiene en cuenta las necesidades de jóvenes con discapacidad o las poblaciones LGBTI? 11. Ustedes tal vez saben que antes de ingresar a PUENTES los jóvenes participan en un proceso intensivo de selección para procurar que los beneficiarios del proyecto estén totalmente comprometidos con el curso, lo completen y logren vincularse al mercado laboral. ¿Según el trabajo que han observado de los jóvenes que llegan a este centro, qué ha observado ustedes sobre las características, la motivación y el nivel de compromiso de los jóvenes? ¿Podría dar unos ejemplos? 12. ¿Cómo se ha utilizado la valoración psicosocial de ingreso de los jóvenes al inicio del proyecto para la selección de cursos, actividades o talleres complementarios para ayudarles a desarrollar habilidades para la vida? • ¿Qué tan útil cree usted que ha sido esa valoración inicial? ¿Qué ventajas o desventajas tiene? 13. ¿Están preparados los jóvenes que ingresan al centro para aprender y desarrollar las habilidades requeridas por el plan de estudios/currículo? ¿Ha notado deficiencias en los conocimientos, habilidades básicas, o en el nivel de desempeño requerido para sus cursos? 14. ¿Qué rol ha jugado este centro de formación en la identificación de empresas o firmas que puedan participar en el diseño de los programas de formación para el trabajo y en los procesos de inserción laboral? ¿Qué rol ha jugado Puentes en este sentido? 15. ¿Cómo se coordina en este centro las actividades con los empleadores para facilitar la extensión del aprendizaje académico o teórico con las empresas involucradas en el proyecto? 16. ¿Cómo se monitorea y evalúa en este centro de formación el desempeño de los jóvenes que participan en programas de capacitación en el trabajo? 17. ¿Se han impartido servicios sobre habilidades para la vida y el trabajo? • ¿Cómo se ha implementado, por ejemplo, se han dado cursos o talleres? ¿Se ofrecen como parte de la formación técnica o son cursos separados? 18. ¿Qué otros servicios ofrece este centro (además de los cursos) a los jóvenes de PUENTES para ayudarles a encontrar empleo? ¿Podría dar unos ejemplos (especificar si los ofrece PUENTES o el centro de formación)? • ¿En qué medida creen que esos servicios complementan sus cursos? ¿Pueden dar unos ejemplos? • ¿En qué aspectos han notado ustedes que los jóvenes requieren más apoyo? ¿Pueden dar unos ejemplos? 40152 PUENTES MATHEMATICA C.9 19. ¿En qué medida se tienen en cuenta los servicios prestados por el centro las necesidades específicas de jóvenes con discapacidad o pertenecientes a la comunidad LGBTI? ¿Podría dar algunos ejemplos? 20. ¿Qué actividades han sido claves en la colocación de los jóvenes capacitados en algún empleo? 21. ¿Qué planes tienen para continuar los programas de formación cuando finalice el apoyo de PUENTES? • ¿Se van a mantener los mismos cursos que están ofreciendo con PUENTES? ¿Por qué sí/no? • ¿Cuáles cree que serán los principales retos cuando termine el apoyo de PUENTES? ¿Qué opciones han considerado para afrontar esos retos? D. CIERRE 22. ¿En qué áreas cree que el centro de formación se ha beneficiado del apoyo de PUENTES? 23. Pensando en el diseño e implementación de PUENTES, ¿qué cambiaría? ¿Por qué? ¿Hay alguna pregunta que no le haya hecho que usted considere que sería importante para nuestra evaluación? 40152 PUENTES MATHEMATICA C.10 PUENTES – Protocolo para entrevistas con empleadores Presentación Gracias por tomarse el tiempo y aceptar reunirse con nosotros hoy. Mi nombre es [NOMBRE DEL ENTREVISTADOR] y soy parte del equipo que colabora con el proyecto de USAID Puentes para el Empleo (PUENTES) y una organización internacional que estudia programas para la juventud en América Latina. El propósito de la reunión es recolectar información cualitativa sobre la implementación del proyecto Puentes para el empleo (PUENTES) como parte de la evaluación final de desempeño del proyecto. Hemos estructurado esta entrevista como una conversación guiada que tomará aproximadamente 30 minutos. Le haré algunas preguntas y la idea es que conteste libremente y comparta sus opiniones. No hay respuestas correctas o incorrectas. Lo que nos interesa es conocer su experiencia y opinión. Tenga en cuenta que lo que comparta con nosotros en esta reunión se utilizará exclusivamente para la evaluación de desempeño del proyecto PUENTES. La información que obtengamos en esta entrevista no lo identificará de forma personal en el reporte, no obstante, sus aportes serán identificados como representante del sector privado. Quisiera grabar la conversación para representar con la mayor precisión posible la información que nos proporcionen. Si está de acuerdo, grabaré la conversación, pero el audio será compartido sólo con las personas que realicen el trabajo de transcripción y con el equipo de investigación. De igual manera, en cualquier momento me puede pedir que deje de grabar. Una vez que termine el estudio, los audios serán borrados. Sus nombres serán excluidos de las transcripciones, así como de cualquier documento. ¿Está de acuerdo? __ Sí __ No ¿Tiene alguna pregunta antes de comenzar? __ Sí __ No A. TIPO DE EMPRESA Y MOTIVACIÓN PARA TRABAJAR CON PUENTES Para comenzar, me gustaría hablar sobre el involucramiento de su empresa con el proyecto PUENTES 1. ¿A qué se dedica su empresa? ¿En qué sector desarrolla sus actividades? ● ¿Aproximadamente cuántas personas trabajan actualmente en su empresa? 2. ¿Me puede explicar cómo conoció el proyecto PUENTES? ● ¿Tuvo alguna reunión en persona con algún representante de PUENTES antes de formalizar su relación? 3. ¿Cuáles aspectos del proyecto lo motivaron a colaborar con PUENTES? ● ¿Consideró trabajar con otros programas de formación? ¿Cuáles y por qué? ● ¿Qué actividades ha desarrollado con el proyecto PUENTES? ¿Podría dar algunos ejemplos? 4. ¿Ha colaborado con centros de formación fortalecidos por PUENTES? □ SÍ □ NO PASE A LA PREGUNTA 9 EN LA SECCION C 40152 PUENTES MATHEMATICA C.11 B. EXPERIENCIA COLABORANDO CON LOS CENTROS DE FORMACIÓN APOYADOS POR EL PROYECTO PUENTES Ahora, me gustaría que platiquemos sobre su experiencia colaborando con los centros de formación y su opinión respecto a los servicios que brindan 5. ¿Ha colaborado con algún centro de formación en el desarrollo o actualización de cursos de formación o en desarrollo de la currícula de los cursos? ● ¿De qué manera ha colaborado con los centros de formación? ¿Podría dar algunos ejemplos? 6. ¿En qué medida cree que la formación de jóvenes responde a las necesidades de empleo de su empresa? ● ¿Ha podido colaborar con PUENTES para diseñar las capacitaciones (o darles un enfoque) para reflejar las necesidades de su empresa? ¿sí/no, por qué? ● Si sí ¿De qué manera cree que la colaboración entre su empresa y el proyecto PUENTES ha contribuido al enfoque de la capacitación que ofrece PUENTES? ● ¿Los cursos y su currícula atienden las necesidades de trabajo de su empresa? ¿Por qué sí/no? ● ¿Qué condiciones cree que son necesarias para que la formación de jóvenes y las oportunidades de empleo del sector privado estén alineados en el corto menos de un año y el largo plazo (más de 3 años)? 7. ¿Si pudiera, cambiaría algo de los cursos de formación que ofrece el proyecto PUENTES? ¿Por qué sí/no? ¿Qué aspectos cambiaría? ¿Es el bachillerato un prerrequisito de contratación en su empresa? ¿Y alguna certificación técnica? C. PERCEPCIÓN SOBRE LOS JÓVENES GRADUADOS DE LOS CENTROS DE FORMACIÓN APOYADOS POR EL PROYECTO PUENTES Ahora, platiquemos sobre su percepción sobre los jóvenes graduados de los centros de formación apoyados por el proyecto PUENTES 8. ¿Ha contratado a jóvenes graduados de los centros de formación apoyados por el proyecto PUENTES? ● ¿Aproximadamente cuántos jóvenes ha contratado en los últimos 12 meses? ● ¿Aproximadamente cuánto tiempo han trabajado o trabajaron en su empresa? ● ¿Los jóvenes fueron contratados como empleados o pasantes? ● En el caso de pasantes, ¿aproximadamente cuánto tiempo dura la pasantía? ¿qué porcentaje o cuántos de los pasantes han sido contratados como empleados? ¿Si no ha contratado a los pasantes, por qué no? 9. ¿Cómo fue el proceso de selección y contratación de los jóvenes del proyecto PUENTES como empleados o pasantes? ● ¿Tuvo oportunidad de interactuar con los centros de formación y/o los jóvenes a través de actividades facilitadas por el proyecto PUENTES antes de contratarlos? ¿De qué manera? ● ¿Qué aspectos del proyecto cree que favorecieron o fueron claves en la contratación de estos jóvenes? ¿Podría dar algunos ejemplos? ● ¿Qué tipo de servicios adicionales hicieron falta para facilitar la contratación de jóvenes del proyecto? ¿Puede dar unos ejemplos? 40152 PUENTES MATHEMATICA C.12 ● ¿Qué dificultades o retos tuvo para la contratación de los jóvenes del proyecto? 10. ¿Cómo ha sido el desempeño de los jóvenes del proyecto PUENTES (empleados y pasantes) en su ocupación o empleo hasta el momento? ● ¿Considera que los jóvenes del proyecto PUENTES cuentan con las habilidades técnicas para cumplir con las responsabilidades de su empleo? ¿sí/no, por qué? ● ¿Considera que los jóvenes del proyecto PUENTES tienen las habilidades interpersonales (habilidades blandas) necesarias para desempeñar sus tareas en la empresa? ¿sí/no, por qué? (si contesta no, ¿cuáles habilidades cree que se deben fortalecer específicamente?) ● ¿Ha identificado alguna debilidad importante en la formación de los jóvenes del proyecto PUENTES para que puedan desempeñarse en los cargos/ocupaciones esperadas? ¿Podría dar algunos ejemplos? ● ¿Considera que los jóvenes del proyecto PUENTES tienen potencial para crecer en la empresa? Por qué sí/ por qué no? (si contesta sí, ¿podría dar unos ejemplos sobre las trayectorias que podrían tener dentro de esta empresa? ¿Considera que tienen el mismo potencial que otros jóvenes no beneficiados de PUENTES? 11. ¿Cómo compararía a los jóvenes graduados del proyecto PUENTES con otros jóvenes empleados que no han participado en este proyecto de capacitación? ● ¿Considera que las aptitudes técnicas son similares o destacan por sus diferencias? ● ¿Y las habilidades blandas? ● ¿Qué otra diferencia destacaría (de haber diferencias)? Actualmente, ¿en qué medida cree que los jóvenes enfrentan barreras para conseguir empleo por ser hombre/mujer, por tener una discapacidad o por su lugar de residencia? ¿Pueden dar unos ejemplos? ● ¿Cuáles han sido las principales barreras que el proyecto PUENTES ha permitido sobrepasar? ● ¿Persisten barreras que no han sido atendidas por el proyecto PUENTES? 12. Antes del proyecto PUENTES, ¿había contratado jóvenes capacitados de zonas de alto riesgo? ¿Por qué sí/no? 13. ¿Considera que gracias a su colaboración con el proyecto PUENTES, su empresa está más abierta a contratar jóvenes vulnerables? ¿Por qué sí/no? 14. En general, ¿cuáles diría usted que han sido los aspectos positivos y negativos de su experiencia colaborando con PUENTES? ● ¿Podría mencionar algunos resultados positivos que haya generado el proyecto PUENTES para que estén alineados la formación de jóvenes y las oportunidades de empleo? ● ¿Podría mencionar algunos resultados inesperados que haya generado PUENTES para que estén alienados la formación de jóvenes y las oportunidades de empleo? ¿Hay alguna pregunta que no le haya hecho que usted considere que sería importante para nuestra evaluación? This page has been left blank for double-sided copying. MATHEMATICA C.14 PUENTES – Protocolo para grupos focales con graduados Presentación Gracias por tomarse el tiempo y aceptar participar en esta ENTREVISTA/GRUPO FOCAL con nosotros hoy. Mi nombre es [NOMBRE DEL ENTREVISTADOR] y soy parte del equipo que colabora con el proyecto de USAID Puentes para el Empleo (PUENTES) y una organización internacional que estudia programas para la juventud en América Latina. El propósito de la reunión es recolectar información cualitativa sobre la implementación del proyecto Puentes para el empleo (PUENTES) como parte de la evaluación final de desempeño del proyecto. Hemos estructurado esta entrevista como una conversación guiada que tomará aproximadamente 20 minutos. Les haré algunas preguntas y la idea es que contesten libremente y compartan sus opiniones. No hay respuestas correctas o incorrectas. Lo que nos interesa es conocer su experiencia y opinión. Tenga en cuenta que lo que comparta con nosotros en esta reunión se utilizará exclusivamente para la evaluación de desempeño del proyecto PUENTES. La información que obtengamos en esta entrevista no lo identificará de forma personal en el reporte, no obstante, sus aportes serán identificados como beneficiarios de proyecto PUENTES. Quisiera grabar la conversación para representar con la mayor precisión posible la información que nos proporcionen. Si está de acuerdo, grabaré la conversación, pero el audio será compartido sólo con las personas que realicen el trabajo de transcripción y con el equipo de investigación. De igual manera, en cualquier momento me puede pedir que deje de grabar. Una vez que termine el estudio, los audios serán borrados. Sus nombres serán excluidos de las transcripciones, así como de cualquier documento. ¿Están de acuerdo todos ustedes? __ Sí __ No ¿Tienen alguna pregunta antes de comenzar? __ Sí __ No Para comenzar, me gustaría hablar sobre su experiencia en los cursos de formación del proyecto PUENTES 1. ¿El curso del proyecto PUENTES para el empleo del que se graduó incluyó formación técnica y formación en habilidades para la vida? ● SI EL CURSO INCLUYO FORMACIÓN TÉCNICA PREGUNTE: ¿Cuáles fueron los aspectos más útiles de la formación técnica que recibió? ● SI EL CURSO INCLUYO HABILIDADES PARA LA VIDA PREGUNTE: ¿Cuáles fueron los aspectos más útiles de los talleres o cursos para desarrollar habilidades para la vida? 2. ¿Qué conocimientos técnicos o habilidades para la vida ha adquirido o mejorado como resultado de la formación del proyecto PUENTES? ¿Puede dar algunos ejemplos? ● ¿Existen conocimientos técnicos o habilidades interpersonales en los que necesite más formación, apoyo y/o práctica? 3. Además de los cursos, ¿qué otros servicios del proyecto PUENTES utilizó? Por ejemplo, asesoría laboral para mejorar el CV, apoyo para el fortalecimiento personal para la entrevista laboral, acompañamiento a la entrevista ● ¿De qué manera le han ayudado esos servicios? ¿Podría dar algunos ejemplos? 4. Pensando en general en el proyecto PUENTES, ¿Qué fue lo que más le gustó del curso del proyecto PUENTES? y ¿lo que menos le gustó? ¿qué cambiaría? ¿por qué? 5. Antes de participar en el curso de formación del proyecto PUENTES, ¿había participado en otros cursos de formación? ● ¿En qué se diferencia el curso de PUENTES de otros cursos de formación? MATHEMATICA C.15 6. ¿Tuvo oportunidades a través de PUENTES de interactuar con futuros empleadores durante los cursos? ● ¿Qué tipo de oportunidades? ¿Puede dar algunos ejemplos? Ahora, me gustaría que platiquemos sobre su empleo actual o el último empleo que tuvo con apoyo de PUENTES después de graduarse 7. ¿Tiene o ha tenido un empleo gracias al apoyo del proyecto PUENTES? □ SÍ □ NO PASE A LA PREGUNTA 19 8. a. EN CASO DE QUE ACTUALMENTE ESTE EMPLEADO PREGUNTAR: ¿Qué ocupación o empleo tiene? ¿Cuánto tiempo lleva en esa ocupación o empleo? ¿Logró conseguir empleo en la misma zona donde vive o consiguió empleo en otro municipio? ¿Es este su único empleo después de graduarse o ha tenido otros empleos con el apoyo de PUENTES entre la graduación del curso de formación y el trabajo actual? b. EN CASO DE QUE ACTUALMENTE NO ESTE EMPLEADO PREGUNTAR: ¿Cuál fue su última ocupación o empleo? ¿Cuánto tiempo estuvo trabajando en su última ocupación o empleo? ¿Su ultimo empleo estaba en la misma zona donde vive o en otro municipio? 9. ¿Cómo logró conseguir una oportunidad de trabajo al terminar el curso? ● ¿Qué dificultades o retos tuviste para encontrar trabajo? 10. ¿Qué aspectos del proyecto PUENTES cree que le favorecieron para encontrar trabajo? 11. ¿Consiguió el tipo de empleo al que aspiraba conseguir al terminar los cursos en el centro de formación apoyado por PUENTES? ¿sí/no? ¿por qué? ● ¿Cuánto tiempo le tomó conseguir empleo? 12. ¿Está bien preparado para cumplir con las responsabilidades de su empleo? ¿sí/no, por qué? ● ¿En qué medida los cursos de formación de PUENTES le ayudaron a prepararse para su empleo actual? ¿Podría dar algunos ejemplos? 13. ¿Cuáles de los servicios o apoyos del proyecto PUENTES le sirvieron más para encontrar empleo? ¿Cuáles le sirvieron menos? 14. ¿Qué tipo de servicios adicionales hicieron falta para facilitar el encontrar empleo o para desempeñarse mejor en su trabajo actual? ¿Puede dar unos ejemplos? 15. Además de los conocimientos y habilidades adquiridas, ¿considera que hay otros beneficios de haber participado en proyecto PUENTES? ¿Puede dar unos ejemplos? 16. ¿Considera que Puentes les apoyó de manera diferenciada por su sexo, discapacidad, o lugar de vivienda y así tener igual oportunidad de conseguir empleo? 17. ¿Qué expectativa tienen de hacer carrera en la empresa donde trabajan o en otra empresa en El Salvador que les permita tener una vida plena en el país? ¿Qué deseos profesionales de futuro tiene? 18. ¿Hay alguna pregunta que no le haya hecho que usted considere que sería importante para nuestra evaluación? ¡Gracias! MATHEMATICA C.16 ESTAS PREGUNTAS SE ADMINISTRAN A GRADUADOS QUE NO HAN TENIDO UN EMPLEO CON EL APOYO DE PUENTES 19. Al término del curso de formación de PUENTES, ¿participó en una pasantía con alguna empresa? ● ¿Cuánto tiempo duró la pasantía? 20. Actualmente, ¿está buscando empleo? □ SÍ □ NO PREGUNTAR POR QUÉ y PASAR A LA PREGUNTA 22 21. ¿Cuánto tiempo lleva buscando empleo sin conseguir? ¿Cuál es su plan en caso de no encontrar empleo en los próximos 6 meses? 22. Después de la graduación del curso de formación, ¿ha tenido oportunidades a través de PUENTES de participar en procesos de selección de personal? ● ¿En cuantos procesos ha participado en los últimos 12 meses y a qué etapas ha llegado en los procesos? 23. ¿Cuáles de los servicios o apoyos de PUENTES le han servido en la búsqueda de empleo? ¿Cuáles sirvieron menos? 24. ¿Qué tipo de servicios adicionales hacen falta para facilitar la búsqueda de empleo? ¿Puede dar unos ejemplos? 25. ¿Consideran que la capacitación y los apoyos de PUENTES mejoraron sus posibilidades para conseguir un trabajo? ● ¿En qué medida creen que PUENTES le ha ayudado a progresar hacia sus metas futuras? ¿Pueden dar unos ejemplos? 26. ¿Considera que Puentes les apoyó de manera diferenciada por su sexo, discapacidad, o lugar de vivienda y así tener igual oportunidad de conseguir empleo? 27. Además de los conocimientos y habilidades adquiridas, ¿considera que hay otros beneficios de haber participado en proyecto PUENTES? ¿Puede dar unos ejemplos? 28. ¿Hay alguna pregunta que no le haya hecho que usted considere que sería importante para nuestra evaluación? ¡Gracias! This page has been left blank for double-sided copying. 40152 PUENTES MATHEMATICA C.18 PUENTES - Protocolo para entrevista con personal de INSAFORP Presentación Gracias por tomarse el tiempo y aceptar reunirse con nosotros hoy. Mi nombre es [NOMBRE DEL ENTREVISTADOR] y soy parte del equipo que colabora con el proyecto de USAID Puentes para el Empleo (PUENTES) y una organización internacional que estudia programas para la juventud en América Latina. El propósito de la reunión es recolectar información cualitativa sobre la implementación del proyecto Puentes para el empleo (PUENTES) como parte de la evaluación final de desempeño del proyecto. Hemos estructurado esta entrevista como una conversación guiada que tomará aproximadamente 30-45 minutos. Tenemos planeadas algunas preguntas, y la idea es que usted(es) las conteste(n) y comente(n) libremente ofreciendo la información que nos pueda ayudar a entender cómo ha evolucionado el proyecto, a qué retos se han enfrentado y qué se ha logrado hasta el momento. Tenga en cuenta que lo que comparta con nosotros en esta reunión se utilizará exclusivamente para la evaluación de desempeño del proyecto PUENTES. La información que obtengamos en esta entrevista no lo identificará de forma personal en el reporte, no obstante, sus aportes serán identificados como representantes de INSAFORP. Quisiéramos grabar la conversación de modo que podamos representar con la mayor precisión posible la información que nos proporcione(n). Si usted está de acuerdo, grabaremos la conversación, pero el audio será compartido solo con las personas que realicen el trabajo de transcripción y con el equipo de investigación. De igual manera, en cualquier momento me puede pedir que deje de grabar. Una vez se haya terminado el estudio, los audios serán borrados. Su(s) nombre(s) será(n) excluido(s) de las transcripciones, así como de cualquier documento. ¿Está de acuerdo? __ Sí __ No ¿Tiene alguna pregunta antes de comenzar? __ Sí __ No [SI EL/LOS PARTICIPANTE(S) ESTÁ(N) DE ACUERDO, INICIE LA GRABACIÓN. DE LO CONTRARIO, TOME NOTAS DETALLADAS EN LUGAR DE GRABAR] Me gustaría empezar por platicar sobre la colaboración de INSAFORP con el proyecto PUENTES 1. ¿Cómo ha colaborado el INSAFORP con el proyecto Puentes para el Empleo? ● En general, ¿cuáles han sido los retos y los beneficios principales en su colaboración con PUENTES? 2. ¿Cómo ha sido la colaboración para la revisión y/o creación de programas certificados? ● ¿Cómo ha colaborado con el proyecto PUENTES para el diseño o actualización del currículo para la formación de capacidades técnicas en sector productivos (TICs, textil, manufactura)? ● ¿Cuáles fueron los retos y los aprendizajes principales de este proceso? 3. Desde su punto de vista, ¿cuál ha sido el mayor beneficio de la colaboración entre INSAFORP y PUENTES? ● ¿Cuáles son las lecciones aprendidas de su colaboración con PUENTES hasta ahora? 40152 PUENTES MATHEMATICA C.19 Ahora, me gustaría platicar sobre la oferta de formación profesional y su coordinación con el sector privado 4. ¿Qué resultados positivos en la oferta de servicios de formación cree que ha producido el proyecto PUENTES? 5. ¿Qué resultados no deseados en la oferta de servicios de formación cree que ha producido el proyecto PUENTES? ¿Por qué cree que ha pasado esto? 6. ¿En qué procesos de actualización curricular les ha apoyado PUENTES? Ahora, vamos a enfocarnos en los planes a futuro para darle continuidad a las actividades del proyecto PUENTES 7. ¿Ha colaborado con PUENTES y/o los centros de formación en un plan de sostenibilidad para los cursos de formación que actualmente ofrece el proyecto PUENTES? ● ¿Me podría describir las actividades que han considerado para el plan de sostenibilidad? ¿Nos podría dar más información sobre quien en INSAFORP participó en la elaboración de este plan? ● ¿El INSAFORP cuenta con los recursos para apoyar este plan de sostenibilidad? Si sí, ¿de dónde han obtenido los recursos? Si no, ¿de dónde prevén obtener los recursos para implementar el plan? 8. ¿En INSAFORP han considerado incorporar en la currícula de la formación técnica, formación en habilidades para la vida y el trabajo cuando termine el apoyo de PUENTES? ¿Por qué sí/no? ● ¿Cuáles cree que serán los principales retos y facilitadores para incorporar formación en habilidades para la vida y el trabajo en los cursos de formación técnica? 9. ¿En INSAFORP han considerado incorporar el sistema SisPuentes para darle seguimiento a los centros de formación autorizados? Si la respuesta es no, ¿Por qué? Si la respuesta es sí, ¿de qué manera? ● ¿Cuáles cree que serán los principales retos y facilitadores para incorporar el sistema SisPuentes como herramienta para la gestión de datos? 10. ¿Qué otros sistemas o herramientas elaborados por PUENTES creen que va a seguir utilizando INSAFORP? 11. ¿Considera que los vínculos entre el sector privado y los centros de capacitación para tener currículos y capacitaciones actualizados se lograrán mantener aún sin el apoyo del proyecto PUENTES? ¿por qué si/no? ¿Hay alguna pregunta que no le haya que Ud. cree que sería importante para nuestra evaluación? 40152 PUENTES MATHEMATICA C.20 PUENTES INSTRUCTOR – PROTOCOLO PARA ENTREVISTA Y GRUPO FOCAL (2019) Presentación Gracias por tomarse el tiempo y aceptar reunirse con nosotros hoy. Mi nombre es [NOMBRE DEL ENTREVISTADOR] y soy parte del equipo que colabora con el proyecto de USAID Puentes para el Empleo (PUENTES) y que estudia programas para la juventud en América Latina. Estamos visitando diferentes centros de formación para conocer la opinión de los instructores sobre los programas de formación que buscan mejorar la preparación de los jóvenes para que se incorporen al mercado laboral. El propósito de esta conversación es conocer su experiencia como instructores de este tipo de programas de formación. Vamos a iniciar una conversación grupal guiada que tardará entre 60 y 90 minutos. Les haremos algunas preguntas y la idea es que ustedes las contesten libremente y compartan sus opiniones. No hay respuestas correctas o incorrectas. Lo que nos interesa es conocer su experiencia y opinión. Tengan en cuenta que lo que ustedes compartan con nosotros en esta reunión se mantendrá en estricta privacidad. La información que obtengamos durante esta conversación se utilizará exclusivamente para el estudio del programa PUENTES. No se presentará información personal que los identifique individualmente sino de manera agregada con las contribuciones de los otros instructores en otras zonas del país. Es decir, las opiniones que ustedes expresen en esta conversación no estarán asociadas a ustedes de forma personal o con sus nombres propios. Quisiéramos grabar la conversación, de modo que podamos representar con la mayor precisión posible la información que nos proporcionen. Si ustedes están de acuerdo, grabaremos la conversación, pero el audio será compartido sólo con las personas que realicen el trabajo de transcripción y con el equipo de investigación. De igual manera, en cualquier momento me puede pedir que deje de grabar. Una vez termine el estudio, los audios serán borrados. Sus nombres serán excluidos de las transcripciones, así como de cualquier documento. ¿Están de acuerdo todos ustedes? __ Sí __ No ¿Tiene alguna pregunta antes de comenzar? __ Sí __ No [SI TODOS LOS PARTICIPANTES ESTÁN DE ACUERDO, INICIE LA GRABACIÓN. SI AL MENOS UNO NO ESTÁ DE ACUERDO, TOME NOTAS DETALLADAS EN LUGAR DE GRABAR] Empecemos presentándonos con nuestro nombre y el curso que cada uno de ustedes enseña en el programa PUENTES. D. PERCEPCION SOBRE EL APOYO A LOS CENTROS DE FORMACIÓN Primero me gustaría hablar sobre el fortalecimiento de los centros de formación y el apoyo que han recibido para mejorar su enseñanza. 1. ¿Han recibido formación o recibieron apoyo de PUENTES para mejorar sus prácticas de enseñanzas? • ¿En qué medida sus prácticas de enseñanza han cambiado después de haber recibido el apoyo de PUENTES? ¿Podrían darme un ejemplo? • ¿Cuáles fueron los aspectos más útiles del apoyo que recibió de PUENTES? ¿Cuáles aspectos considera que se podría mejorar? 2. ¿Se sintieron bien preparados para impartir los cursos de PUENTES? ¿Por qué sí/no? • ¿Qué tan preparados se sienten para apoyar a los jóvenes con discapacidades, o de grupos LGBTI? 40152 PUENTES MATHEMATICA C.21 3. ¿Pueden describir qué materiales, equipos, o instalaciones (taller, laboratorios) utilizan para los cursos PUENTES? • ¿Han tenido suficiente acceso a los materiales, o instalaciones que necesitan? ¿Han enfrentado retos con respecto a los materiales o instalaciones? • ¿Saben si los materiales/equipos utilizados fueron proporcionados a través de las donaciones del proyecto PUENTES al centro de formación? • ¿Qué tan alineados están los textos y guías de enseñanza con los cursos que imparten? ¿Esos materiales son relevante para los cursos? ¿Por qué sí/no? 4. ¿Sabe de alguna colaboración del centro de formación con la empresa privada? ¿Cómo se ha reflejado esta colaboración con el sector privado en el enfoque de capacitación? E. PERCEPCIONES SOBRE LA CAPACITACION A JÓVENES Ahora me gustaría que hablemos sobre los cursos de formación que se imparten para preparar a los jóvenes para ingresar al mercado laboral. 5. ¿Pueden describir los métodos o enfoques de instrucción que utilizan en sus cursos? ¿En qué aspectos se diferencian los cursos apoyados por PUENTES de otros cursos de educación técnica vocacional que usted conozca o haya enseñado? • ¿Cuáles cree que son las fortalezas y debilidades de los cursos de PUENTES? • ¿Han enfrentado retos enseñado los cursos de PUENTES? ¿Cuáles? 6. ¿Qué tan alineado cree usted que está su curso con las oportunidades laborales en el sector económico respectivo? ¿Qué tipo de ocupación (o puestos laborales) están en capacidad de ejercer los jóvenes que completan su curso? • ¿En qué medida cubre sus cursos los conocimientos y destrezas/habilidades laborales que los empleadores requieren en sus respectivos campos? • ¿Hay vacíos o áreas que considera usted que falta cubrir en el plan de estudios/currículo? ¿Pueden dar algunos ejemplos? 7. ¿Cómo se implementan los cursos que cuentan con teoría y práctica en las empresas? ¿Podrían dar algunos ejemplos? • ¿Cuáles ventajas y desventajas han observado de estos cursos que cuentan con teoría y práctica en las empresas? • ¿Qué aspectos del modelo de formación teórico-práctico se podrían mejorar para facilitar la transición de los jóvenes a la vida laboral? • ¿En qué medida cree que su curso y el proyecto PUENTES en general utilizan un enfoque de inclusión y diversidad? ¿El curso se adecúa o tiene en cuenta las necesidades de los jóvenes con discapacidad? 8. Ustedes tal vez saben que antes de ingresar a PUENTES los jóvenes participan en un proceso intensivo de selección para procurar que los beneficiarios del programa estén totalmente comprometidos con el programa, lo completen y logren vincularse al mercado laboral. ¿Según el 40152 PUENTES MATHEMATICA C.22 trabajo de los jóvenes en sus cursos, qué han observado ustedes sobre las características, la motivación y el nivel de compromiso de los jóvenes? ¿Podrían dar unos ejemplos? 9. ¿Los jóvenes que ingresan están preparados a sus cursos para aprender y desarrollar las habilidades requeridas por el plan de estudios/currículo? ¿Han notado deficiencias en los conocimientos, habilidades básicas, o en el nivel de desempeño requerido para sus cursos? 10. ¿Conocen ustedes los otros servicios que tiene a disposición los jóvenes a través de PUENTES además de los cursos? (orientación laboral, inserción laboral, habilidades para la vida) ¿qué saben de esos servicios? • ¿En qué medida creen que esos servicios complementan sus cursos? • ¿En qué aspectos han notado ustedes que los jóvenes requieren más apoyo? ¿Pueden dar unos ejemplos? 11. ¿En qué medida creen que los cursos de PUENTES les dan a los alumnos una ventaja para acceder a oportunidades de trabajo? • ¿Cuáles actividades han sido claves en la colocación de los jóvenes capacitados en algún empleo? C. CIERRE 12. ¿En qué áreas cree que el centro de formación se ha beneficiado del apoyo de PUENTES? 13. Pensando en general en el programa PUENTES, ¿qué cambiaría? ¿Por qué? 14. ¿Hay alguna pregunta que no le haya hecho que usted cree que sería importante para nuestra evaluación? This page has been left blank for double-sided copying. 40152 PUENTES MATHEMATICA C.24 PUENTES-PROTOCOLO PARA ENTREVISTA O GRUPO FOCAL CON EL PERSONAL DE PUENTES (2019) Presentación Gracias por tomarse el tiempo y aceptar reunirse con nosotros hoy. El propósito de la llamada es recoger información cualitativa sobre la implementación y los logros del PUENTES hasta el momento. Además de esta reunión con usted(es), vamos a conversar con diferentes personas involucradas directa o indirectamente con el proyecto para conocer diferentes aspectos de la implementación. Esta conversación guiada tardará entre 60 y 90 minutos. Tenemos planeadas algunas preguntas amplias, y la idea es que usted(es) las conteste(n) y comente(n) libremente ofreciendo la información que nos pueda ayudar a entender cómo ha evolucionado el proyecto, a qué retos se han enfrentado y qué se ha logrado hasta el momento. Tenga(n) en cuenta que lo que usted(es) comparta(n) con nosotros en esta reunión se mantendrá en estricta privacidad. La información que obtengamos durante las entrevistas se utilizará exclusivamente para el estudio de PUENTES. No se presentará información personal que lo(s) identifique a usted(es) individualmente. Los resultados cualitativos se presentarán de manera agregada con las contribuciones de los otros participantes. Es decir, las opiniones que usted(es) exprese(n) en esta conversación no estarán asociadas a usted(es) de forma personal o con su nombre propio. Quisiéramos grabar la conversación de modo que podamos representar con la mayor precisión posible la información que nos proporcione(n). Si usted está de acuerdo, grabaremos la conversación, pero el audio será compartido solo con las personas que realicen el trabajo de transcripción y con el equipo de investigación. De igual manera, en cualquier momento me puede pedir que deje de grabar. Una vez se haya terminado el estudio, los audios serán borrados. Su(s) nombre(s) será(n) excluido(s) de las transcripciones, así como de cualquier documento. ¿Está bien con usted(es)? __ Si __ No ¿Tiene(n) alguna pregunta antes de comenzar? __ Si __ No [SI EL/LOS PARTICIPANTE(S) ESTÁ(N) DE ACUERDO, INICIE LA GRABACIÓN. DE LO CONTRARIO, TOME NOTAS DETALLADAS EN LUGAR DE GRABAR] F. PERCEPCION SOBRE EL ENTORNO PARA EL CRECIMIENTO DE LA FUERZA LABORAL Y DEL EMPLEO Empecemos hablando sobre las actividades del proyecto enfocadas en crear un ambiente más favorable para el crecimiento de la fuerza laboral y del empleo. 1. ¿Cómo se desarrolló el diagnóstico con empresas para conocer la demanda e integrar iniciativas para desarrollar oportunidades en el mercado laboral? • ¿Cuáles fueron los retos y los aprendizajes principales de ese diagnóstico? ¿Cuáles consideran que fueron los logros más importantes de esta actividad? 2. ¿Cómo se implementó la asistencia técnica para la sensibilización de mejores prácticas de inclusión? • Podrían describir el proceso de diseño y ejecución de los talleres de inclusión. ¿Cómo recibieron las empresas esos talleres? • ¿Qué retos tuvieron en el diseño o implementación de las actividades de sensibilización para mejorar las prácticas de inclusión? 40152 PUENTES MATHEMATICA C.25 • ¿Cómo se ha integrado la igualdad de género, el empoderamiento femenino y la inclusión social? • ¿Cuáles consideran que fueron los logros más exitosos las actividades de sensibilización? ¿Qué aspectos no salieron como se esperaba? Según el informe de resultados de junio (2019), 38 empresas implementaron cambios en las prácticas de inclusión. ¿Cuáles son los planes para llegar a la meta de 50 empresas al final del proyecto? 3. ¿Cómo se implementó la campaña de comunicación a nivel nacional? ¿Cuáles fueron los mensajes principales que se transmitieron a través de la campaña? • ¿Qué retos tuvieron en el diseño o implementación de la campaña de comunicación nacional? • ¿Qué alcances tuvo la campaña? ¿Cómo se midieron los alcances? ¿Cuáles consideran que fueron los logros más importantes de la campaña? ¿Qué aspectos de la campaña no salieron como se esperaba? 4. ¿Cómo han utilizado los resultados de la evaluación de las regulaciones que afectan el desarrollo de la fuerza laboral y el empleo de jóvenes en riesgo? • ¿Cuáles han sido los retos en gestionar las leyes o reglamentos que faltan? 5. ¿Cuáles han sido las actividades implementadas para construir asociaciones con el sector privado y asociaciones de la industria? • ¿Cómo se lograron las contribuciones del sector privado para mejorar la formación de jóvenes vulnerables (de mayor riesgo) por el valor de $4,238,369 USD? ¿Esperan inversiones adicionales a dentro de los próximos seis a doce meses? 6. ¿Qué resultados (positivos y negativos) ha generado la implementación de este componente? • ¿Cuáles son las percepciones de los actores involucrados (empresas del sector privado, representantes de centros de formación, jóvenes) sobre esos resultados? • ¿Cuáles de las actividades implementadas en este componente considera(n) que deberían priorizarse en futuros proyectos? • ¿Considera(n) que las actividades de este componente son sostenibles? ¿Por qué si o por qué no? G. PERCEPCION SOBRE EL APOYO A LOS CENTROS DE FORMACIÓN Ahora hablemos sobre el fortalecimiento de los centros de formación. 7. ¿Cómo se adaptó la metodología TVET/CAT al contexto de El Salvador y en particular a los centros de formación? • ¿Cuáles fueron los retos y los aprendizajes principales de esa evaluación? ¿Cuáles consideran que fueron los logros más importantes de la evaluación? • ¿Cambiaría algo más de la metodología TVET/CAT después de haberla utilizado varias veces? • ¿Cómo han utilizado los resultados de la evaluación? 8. ¿Cómo se seleccionaron los centros de formación, y cómo se implementaron las actividades para fortalecer los centros? • ¿Cuáles son los aspectos centrales del proyecto? ¿Cómo recibieron el proyecto los centros de formación? • ¿Qué retos tuvieron en el diseño o implementación del programa de formación? • ¿Cuáles considera(n) que fueron los logros más importantes del programa de formación? ¿Qué aspectos no han salido como se esperaba? 40152 PUENTES MATHEMATICA C.26 • ¿En qué medida considera(n) que el mecanismo de subvenciones (grants under contract [GUC]) ha beneficiado o afectado la implementación del proyecto? 9. ¿Cómo ha sido el proceso de revisión y/o creación de programas certificados? • ¿Cuáles fueron los retos y los aprendizajes principales de este proceso? • ¿Considera que los vínculos entre el sector privado y los centros de capacitación para tener currículos y capacitaciones actualizados se lograrán mantener aún sin el apoyo de USAID cuando Puentes finalice? ¿por qué si/no? 10. ¿En qué medida PUENTES ha alcanzado sus metas de resultados en cuanto a: • el fortalecimiento de los centros de capacitación? • los programas certificados nuevos / revisados? H. PERCEPCIONES SOBRE LA CAPACITACION A JÓVENES Para finalizar, hablemos sobre las actividades para ayudar a los jóvenes a mejorar su preparación para el empleo. 11. ¿Cómo ha sido el desarrollo y la implementación de la estrategia para que los jóvenes permanezcan o regresen al sistema educativo formal a través de bachilleratos flexibles? • ¿Qué retos tuvieron en el diseño o implementación de esta estrategia? • ¿Cuáles consideran que fueron los logros más importantes en la implementación de la estrategia? ¿Qué aspectos no salieron como se esperaba? 12. ¿Cómo ha sido el desarrollo y la implementación los cursos de formación ofrecidos en bachilleratos técnicos vocacionales (BTV)? • ¿Cómo ha sido el diseño e implementación del nuevo currículo para cursos de BTV? • ¿Qué retos tuvieron en el diseño o implementación de esta actividad? • ¿Cuáles consideran que fueron los logros más importantes en la implementación de la actividad? ¿Qué aspectos no salieron como se esperaba? 13. ¿Cómo se diseñó e implementó el currículo para las habilidades para la vida? ¿Cuáles son los aspectos centrales del currículo? • ¿Qué retos tuvieron en el diseño o implementación del currículo para los cursos enfocados en habilidades para la vida? • ¿Cómo han recibido los jóvenes la formación para fortalecer las habilidades para la vida? ¿Cómo se están monitoreando la mejora en las habilidades blandas y conocimiento de los jóvenes? • ¿Cuáles consideran que fueron los logros más importantes en la implementación del currículo para las habilidades para la vida? ¿Qué aspectos no salieron como se esperaba? • Según el informe de resultados de junio (2019), 5,801 jóvenes han recibido formación en habilidades para la vida. ¿Cuáles son los planes para llegar a meta actualizada después del recorte presupuestal (referirse a la nueva meta)? 14. ¿Cómo se está promoviendo el acceso de los jóvenes en riesgo a las capacitaciones? • ¿Qué retos han tenido en los esfuerzos para aumentar el acceso de los jóvenes en riesgo a las capacitaciones? • ¿Cómo han recibido los jóvenes en riesgo los esfuerzos para aumentar el acceso que tienen a los programas de formación a través de PUENTES? 40152 PUENTES MATHEMATICA C.27 15. ¿Cómo está promoviendo el acceso de los jóvenes en riesgo al empleo? • ¿Qué retos han tenido en los esfuerzos para aumentar el acceso de los jóvenes en riesgo al empleo? • ¿Cuáles consideran han sido los logros principales en el acceso de los jóvenes en riesgo al empleo? ¿Qué aspectos no han salido como se esperaba? • Según el informe de resultados de junio (2019), 2,168 jóvenes han adquirido nuevo o mejor empleo después de participar en las actividades de formación. ¿Cuáles son los planes para llegar a la meta actualizada después del recorte presupuestal (referirse a la nueva meta)? 16. ¿Cómo están apoyando las iniciativas de emprendimiento? • ¿Qué retos han tenido en la implementación de las iniciativas de emprendimiento? • ¿Cómo han recibido los jóvenes las iniciativas de emprendimiento? • ¿Cuáles considera(n) que han sido los logros principales en las iniciativas de emprendimiento? ¿Qué aspectos no han salido como se esperaba? 17. ¿Cómo se diseñaron e implementaron los servicios de colocación laboral? • ¿Cómo han recibido los jóvenes los servicios de colocación laboral? ¿Cuáles servicios han tenido una mayor acogida? ¿Cuáles no se han acogido de la manera esperada? • ¿Cuáles considera(n) que han sido los logros más importantes de los servicios de colocación laboral? • ¿Qué aspectos no han salido como se planeaba? 18. ¿Cómo se ha integrado la igualdad de género, el empoderamiento femenino y la inclusión social en la implementación de este componente? • ¿Fueron exitosos esos esfuerzos? ¿Por qué si o por qué no? 19. ¿En qué medida ha alcanzado PUENTES sus metas respecto a la inclusión jóvenes vulnerables en el sector privado? 20. ¿En qué medida ha alcanzado PUENTES sus metas de resultados en cuanto a: • empleo, mejores habilidades y resultados educativos entre los beneficiarios? D. CIERRE 21. Pensando en el diseño e implementación de PUENTES, ¿qué cambiaría? ¿por qué? 22. ¿Hay alguna pregunta que no le haya que Ud. cree que sería importante para nuestra evaluación? C.28 PUENTES- Protocolo para entrevistas con personal clave de USAID/El Salvador Presentación Gracias por tomarse el tiempo y aceptar reunirse con nosotros hoy. Como saben, el propósito de la reunión es recolectar información cualitativa sobre la implementación del proyecto Puentes para el empleo (PUENTES) como parte de la evaluación final de desempeño del proyecto. Ustedes son actores clave con una perspectiva única e importante sobre la implementación del proyecto. Hemos estructurado esta entrevista como una conversación guiada que tomará aproximadamente 60 minutos. Le haré algunas preguntas y la idea es que conteste libremente y comparta sus opiniones. No hay respuestas correctas o incorrectas. Lo que nos interesa es conocer su experiencia y opinión. Tenga en cuenta que lo que comparta con nosotros en esta reunión se utilizará exclusivamente para la evaluación de desempeño del proyecto PUENTES. La información que obtengamos en esta entrevista no lo identificará de forma personal en el reporte, no obstante, sus aportes serán identificados como representante de USAID/El Salvador encargados de la administración del proyecto PUENTES. Quisiera grabar la conversación para representar con la mayor precisión posible la información que nos proporcionen. Si está de acuerdo, grabaré la conversación, pero el audio será compartido sólo con las personas que realicen el trabajo de transcripción y con el equipo de investigación. De igual manera, en cualquier momento me puede pedir que deje de grabar. Una vez que termine el estudio, los audios serán borrados. Sus nombres serán excluidos de las transcripciones, así como de cualquier documento. ¿Está de acuerdo? __ Sí __ No ¿Tiene alguna pregunta antes de comenzar? __ Sí __ No [SI EL/LOS PARTICIPANTE(S) ESTÁ(N) DE ACUERDO, INICIE LA GRABACIÓN. DE LO CONTRARIO, TOME NOTAS DETALLADAS EN LUGAR DE GRABAR] Antes de empezar, cuénteme un poco sobre usted: 1. ¿Cuál es su cargo en la organización? • ¿Hace cuánto que ha estado involucrado en el proyecto PUENTES (incluso desde la fase del diseño)? Ahora, me gustaría platicar sobre el diseño y la implementación del proyecto PUENTES 2. ¿Desde su punto de vista, cuáles son las barreras que existen en El Salvador para incrementar y mejorar el empleo de jóvenes vulnerables de zonas de alto riesgo? 3. Cuando el proyecto PUENTES fue diseñado, ¿cuál era la visión del proyecto en relación con las barreras que mencionó? • ¿Cuáles eran las expectativas del proyecto cuando fue diseñado? Es decir, ¿Qué se lograría con el proyecto y cómo? • ¿Cómo compara esa visión con lo que se ha logrado hasta el momento? C.29 4. En términos generales, ¿podría describir las principales actividades implementadas como parte del proyecto PUENTES? 5. ¿Cuáles fueron las principales barreras y facilitadores en la implementación de las actividades que acaba de mencionar? Las barreras pueden ser internas o externas al proyecto o al país. 6. ¿Puede mencionar los éxitos que resultaron de los facilitadores que acaba de mencionar 7. ¿Podría mencionar cambios en la implementación que ocurrieron durante la implementación del proyecto? 8. ¿Con el apoyo y servicios que ofrece PUENTES se están alcanzando las metas de implementación del proyecto? ¿Por qué si o por qué no? 9. ¿Cómo ha facilitado o entorpecido el mecanismo de subvenciones bajo contrato (Grants under contract) la implementación en general del proyecto y el fortalecimiento de soluciones locales? 10. ¿Cómo se vio afectada la implementación de actividades por la incertidumbre del futuro de los proyectos en el triángulo norte de América Central? • ¿Como se ha visto afectado el financiamiento de las actividades del proyecto? • ¿Se han generado retrasos o demoras en la implementación? • ¿Se han llevado a cabo cambios en la implementación? Para finalizar, me gustaría que platicáramos sobre los resultados del proyecto y su visión para las actividades una vez que termine el apoyo de USAID 11. ¿Qué resultados positivos ha producido el proyecto PUENTES al final del proyecto? ¿Qué tal resultados negativos producidos por el proyecto PUENTES? ¿Ha habido resultados no deseados debido al proyecto PUENTES? 12. Desde su perspectiva, ¿cuáles considera algunas de las lecciones más importantes aprendidas hasta ahora? • ¿Cuáles deberían priorizarse en futuros programas en El Salvador? ¿Cuáles deberían priorizarse en otros proyectos de desarrollo de la fuerza laboral? 13. Si pudiera regresar, ¿hay algo en el diseño o la implementación de PUENTES que cambiaría? ¿Por qué? 14. ¿Qué actividades se han implementado para garantizar la sostenibilidad del proyecto PUENTES? • ¿Cuáles de las actividades del programa cree que pueden perdurar y terminarán siendo sostenibles sin el apoyo continuo de USAID? ¿Hay alguna pregunta que no le haya que Ud. cree que sería importante para nuestra evaluación? APPENDIX D LIST OF INFORMATION SOURCES This page has been left blank for double-sided copying. LIST OF INFORMATION SOURCES D.2 Key Informant Interviews and Focus Group Participants We do not provide names of all people interviewed since this goes against the informed consent statement approved by USAID. The types of people interviewed is included in the body of the report. Documents review All documents reviewed for this evaluation are included under the list of references as part of the main report of the evaluation. Sites visited Sites visited Depto/Mun Training centers Escuela Superior Franciscana Especializada/AGAPE Sonsonate Fe y Alegría El Salvador Santa Ana Fundación Gloria de Kriete (FGK) Ahuachapán Universidad Alberto Masferrer / USAM San Salvador FUNDEPLAST San Salvador Fe y Alegría - FyA San Salvador Universidad Evangélica de El Salvador Santa Ana Centro de Formación Profesional - CFP AGAPE Textiles San Salvador Firms Grupo Monge Soyapango Intradesa S.A. de C.V. Soyapango Central American Software Services (CASS) San Salvador Empresas ADOC S.A de C.V. Soyapango DECOTEX INTERNATIONAL Limitada de C.V. Ciudad Arce Union Comercial de El Salvador S.A de C.V. (UNICOMER) San Salvador Dymel SA de CV San Salvador Fruit of the Loom (FOL) Ciudad Arce Termoencogibles Santa Tecla Grupo Lorena, S.A. de C.V. San Miguel Other sites visited USAID San Salvador Bridges to Employment San Salvador INSAFORP San Salvador This page has been left blank for double-sided copying. APPENDIX E DISCLOSURES OF COI This page has been left blank for double-sided copying. DISCOLSOURES OF COI MATHEMATICA E.2 Disclosure of Real or Potential Conflict of Interest for USAID Evaluations Instructions: Evaluations of USAID projects will be undertaken so that they are not subject to the perception or reality of biased measurement or reporting due to conflict of interest. 1 For external evaluations, all evaluation team members will provide a signed statement attesting to a lack of conflict of interest or describing an existing conflict of interest relative to the project being evaluated.2 Evaluators of USAID projects have a responsibility to maintain independence so that opinions, conclusions, judgments, and recommendations will be impartial and will be viewed as impartial by third parties. Evaluators and evaluation team members are to disclose all relevant facts regarding real or potential conflicts of interest that could lead reasonable third parties with knowledge of the relevant facts and circumstances to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Operating Unit leadership, in close consultation with the Contracting Officer, will determine whether the real or potential conflict of interest is one that should disqualify an individual from the evaluation team or require recusal by that individual from evaluating certain aspects of the project(s). In addition, if evaluation team members gain access to proprietary information of other companies in the process of conducting the evaluation, then they must agree with the other companies to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished.3 Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. 1 USAID Evaluation Policy (p. 8); USAID Contract Information Bulletin 99-17; and Federal Acquisition Regulations (FAR) Part 9.5, Organizational Conflicts of Interest, and Subpart 3.10, Contractor Code of Business Ethics and Conduct. 2 USAID Evaluation Policy (p. 11) 3 FAR 9.505-4(b) DISCOLSOURES OF COI MATHEMATICA E.3 DISCOLSOURES OF COI MATHEMATICA E.4 DISCOLSOURES OF COI MATHEMATICA E.5 DISCOLSOURES OF COI MATHEMATICA E.6 This page has been left blank for double-sided copying. APPENDIX F SUMMARY OF INFORMATION ABOUT THE EVALUATION TEAM This page has been left blank for double-sided copying. SUMMARY OF INFORMATION ABOUT THE EVALUATION TEAM MATHEMATICA F.2 Patricia Costa brings more than fifteen years of international development experience both implementing and evaluating projects in the economic growth and workforce development space. Ms. Costa is the project director of a mixed-methods performance evaluation of an investment climate project in El Salvador for MCC. She is also a senior researcher on the LAC Reads project where she has co-authored the Central America Workforce Development Report three years in a row, where she facilitated learnings and developed lessons learned based on the implementation of six workforce development reports in Central America funded by USAID. She has also served as qualitative researcher on several mixed-methods evaluations in Africa. She holds an M.P.P. in public policy from the University of Michigan. For the Bridges Performance Evaluation, she served as one of the primary authors and provided overall leadership to the evaluation. Ivonne Padilla has supported education evaluations for LAC Reads since 2014. She led the analysis and co-authored randomized control trial (RCT) evaluation reports of early reading programs in Guatemala and Peru. She also conducted data collection, analysis and report writing of an experimental evaluation of a set of pedagogical interventions focused on training and coaching teachers to use the best early reading practices in two departments of the Peruvian Amazonian region. In addition, Ms. Padilla is working on two education interventions funded by the Millennium Challenge Corporation (MCC) in El Salvador, where she has developed protocols, conducted data collection and served as co-author of the baseline report of an impact evaluation of the Full Time Inclusive schools. She holds an M.P.A. in international development from John F. Kennedy School of Government, Harvard University. For the Bridges Performance Evaluation, she led data collection, conducted the analysis of qualitative data and co-authored the evaluation report. Dr. Larissa Campuzano has extensive experience in the design and implementation of experimental and nonexperimental evaluations as well as conducting systematic reviews and providing evaluation technical assistance and training. Her main research areas are education and international development. Dr. Campuzano is a senior researcher on the USAID-funded LAC Reads project, a large multi-country evaluation for USAID that is evaluating early-reading interventions programs in Latin America. She is the project director and principal investigator of the evaluation of the education interventions that the Millennium Challenge Corporation’s (MCC) is funding under the second compact in El Salvador. She holds a Ph.D. in economics from the University of Rochester. For the Bridges Performance Evaluation, Dr. Campuzano led the initial design stages of the evaluation as well as data collection efforts. She served as a quality assurance reviewer on the report. Dr. Nancy Murray is the current project director of the Latin America and the Caribbean (LAC) Reads project which she has led since 2012. As part of her role she provides leadership and technical support and manages multiple impact evaluations and cost-effectiveness analysis of programs addressing early grade reading and access to education in post-conflict zones including: a two-country randomized control trial (RCT) in Guatemala and Peru of an early grade reading and community outreach program (Read Together, Learn Together) for indigenous speaking populations; an RCT of a set of pedagogical interventions focusing on teacher training and coaching in best early grade reading practices in the Amazon region of Peru (Amazonia Reads); an RCT of the Community Action for Reading and Security project in Nicaragua’s Room to Grow project for children out of school or at risk of dropping out; an RCT in Honduras SUMMARY OF INFORMATION ABOUT THE EVALUATION TEAM MATHEMATICA F.3 examining the use of end of grade and formative assessments on early grade reading and other learning outcomes; and a four-country descriptive study of the achievements of youth workforce development programs in El Salvador, Guatemala, Nicaragua, and Honduras. She holds a Ph.D. in population dynamics from The Johns Hopkins University. For the Bridges Performance Evaluation, she oversaw all aspects of the evaluation during both the design and implementation phase. This page has been left blank for double-sided copying. A.2 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 Tel: (202) 712-0000 Fax: (202) 216-3524 www.usaid.gov