Final Performance Evaluation Plan for Puentes para el Empleo March 2019 This publication was produced for review by the United States Agency for International Development. It was prepared by Jorge Ugaz, Larissa Campuzano, and Nancy Murray of Mathematica Policy Research. This page has been left blank for double-sided copying. This page has been left blank for double-sided copying. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH v CONTENTS I. INTRODUCTION.............................................................................................................................. 1 II. DESCRIPTION OF PUENTES AND RESEARCH QUESTIONS .................................................... 3 A. Goals and objectives ................................................................................................................. 3 B. Initial assessments and activities .............................................................................................. 3 C. Evaluation questions ................................................................................................................. 6 III. EVALUATION DESIGN: DATA AND ANALYSIS ............................................................................ 9 A. Qualitative data........................................................................................................................ 11 B. Quantitative data ..................................................................................................................... 12 C. Analysis plan............................................................................................................................ 12 IV. DATA COLLECTION PLAN ........................................................................................................... 15 A. Qualitative data collection........................................................................................................ 15 B. Quantitative data collection ..................................................................................................... 15 V. TIMELINE, DELIVERABLES, AND PERSONNEL ........................................................................ 19 A. Timeline and deliverables........................................................................................................ 19 B. Personnel................................................................................................................................. 19 APPENDIX A LOGIC MODEL FOR THE PUENTES PROJECT............................................................... 21 APPENDIX B PRESENTATIONS FROM 2016 ......................................................................................... 25 APPENDIX C IMPACT EVALUATION DESIGN, 2017.............................................................................. 45 APPENDIX D SUMMARY OF PILOT OF RANDOM ASSIGNMENT ........................................................ 51 APPENDIX A (OF SUMMARY OF THE PILOT) POTENTIAL TIMELINE FOR EVALUATION OF PUENTES........................................................................................................................................ 65 APPENDIX B (OF SUMMARY OF THE PILOT) PROCESS FOR RANDOM ASSIGNMENT ............. 69 APPENDIX E UPDATED POWER CALCULATIONS FOR IMPACT EVALUATION................................. 71 This page has been left blank for double-sided copying. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH vii TABLES III.1 Evaluation questions and data sources for final performance evaluation of Puentes ................... 10 IV.1 Main outcomes or indicators from M&E data to be provided by Puentes...................................... 16 IV.2 Plans for qualitative and quantitative data collection ..................................................................... 17 V.1 Evaluation timeline......................................................................................................................... 19 This page has been left blank for double-sided copying. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 1 I. INTRODUCTION Youth unemployment is a critical global challenge. In 2014, almost 73 million of the world’s young people were unable to find work—an unemployment rate of 13.0 percent (International Labor Organization 2015). More than 350 million of the world’s youth were economically disengaged, and more than 500 million were underemployed (World Economic Forum 2014). There is consensus that joblessness among youth has long-lasting consequences for individuals, their families, and society as a whole. To address this, many youth workforce development (WFD) programs around the world have been established, focused on providing technical skills, soft or life skills, and workforce development services. A large body of literature documents evidence on the effectiveness of many WFD programs in both developed and developing countries. Several reviews (Kluve et al. 2016; McKenzie 2017; Olenik 2013; Tripney et al. 2013) have looked at dozens of studies and found mixed evidence on the programs’ impacts on employment outcomes. Kluve and colleagues conclude that WFD development programs have increased the employment and income of youth who participate in them, but impacts have been small and vary widely by the type of program and country context. The authors suggest that skill training programs are effective in increasing employment outcomes, whereas other programs such as subsidized employment are not. In Latin America, a few rigorous impact evaluations of skill training programs have found small effects on employment and income for at least some subgroups (Acevedo 2017; Attanasio et al. 2011; Betcherman et al. 2004; Card et al. 2011; Ibarraran et al. 2014; Ibarraran and Rosas Shady 2009). These training programs provide a combination of technical skills training, soft skills training, and internships. Although there is no rigorous evidence confirming that programs need to provide both technical and soft skills training to be effective, employers have identified soft skills training as the most valuable component (Ibarraran et al. 2015). Furthermore, researchers have also identified soft skills as an important factor in employment success (Acevedo et al. 2017; Heckman et al. 2006; J-PAL 2017; Kautz et al. 2014; Rankin et al. 2015). The United States Agency for International Development, Bureau for Latin America and the Caribbean Bureau (USAID/LAC) is playing a key role by funding the implementation of workforce development projects in Central America. The Bridges to Employment project in El Salvador (referred to throughout this document by its Spanish name, Puentes (bridges), is one of USAID/LAC’s largest investments in workforce development: it is a five-year, $42.2 million project implemented by Development Associates International (DAI) and its subcontractors, Plan International and JBS International, Inc. The Puentes program is designed to increase employment levels for at-risk youth and help them get better jobs. Puentes has adopted practices that have shown promise for achieving the latter, such as providing both technical and soft skills training. Furthermore, Puentes plans to improve the capacity of training providers and promote the link between training providers and future employers. Assessing the needs of employers will allow Puentes to provide training courses that are more aligned to employers’ needs, thereby increasing the chances that trainees will find jobs. Since 2015, Mathematica has been an active learning partner of USAID/Washington and USAID/El Salvador, conducting comprehensive reviews of the literature on youth workforce development programs and identifying gaps. At the end of 2015, through the LAC Reads EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 2 Evaluation contract, USAID/El Salvador asked Mathematica to propose rigorous impact designs and explore the possibility of conducting an impact evaluation of Puentes. In March 2016, Mathematica staff met with DAI and USAID staff in El Salvador to present our preliminary thoughts for a potential evaluation design (see Appendix C for the two meeting presentations, and a debrief presentation focused on a potential evaluation design). During early 2017, Mathematica met on different occasions with Puentes staff to discuss some of the key parameters required for an impact evaluation to be feasible and to obtain more detailed information about the rollout of the program and, in particular, of the training courses. Mathematica then updated the impact evaluation design accounting for Puentes implementation plans (see Appendix D). In June 2017, once the early program design activities were complete and Puentes had shared detailed information about its upcoming activities, USAID, Mathematica, and Puentes agreed to conduct a pilot randomized controlled trial (RCT) to test the processes and feasibility of random assignment. The pilot was successfully implemented between July and September 2017, and the process we designed for random assignment worked (see Appendix E for Mathematica’s memo summarizing the findings). Mathematica therefore concluded that it was possible to implement a three-arm RCT design and updated the power calculations (see Appendix F). However, during the first few weeks of 2018, Mathematica learned about the continuing evolution of the implementation of Puentes and identified some feasibility issues for the RCT. Mathematica shared a memo summarizing these issues, which were related to the grant-making process implemented by Puentes and the separate rollout of technical and soft-skills courses by the training centers (see Appendix F). After discussions on this topic, in April 2018 USAID/El Salvador decided to pursue instead a mid-term performance evaluation of Puentes. On July 2018, Mathematica submitted a draft of the evaluation plan for this mid-term evaluation. However, after further discussion, in November 2018 USAID/El Salvador opted to pursue a final performance evaluation, which will take place from September 2019 through June of 2020. This final performance evaluation will provide insights and generate learning on whether Puentes reached its targets and key objectives, stakeholders’ perceptions about Puentes’ implementation and impact, and what lessons have been learned that might inform the next generation of WFD programs in El Salvador and Central America. This evaluation plan lays out the design for this final performance evaluation. The rest of this document is organized as follows. In Section II, we describe the key activities implemented by Puentes and present a logic model developed to reflect our understanding of the program we are using to guide the design of the evaluation, including the research questions to be addressed by the evaluation. In Section III we discuss the evaluation design, focusing on data requirements and analysis. Section IV describes the key outcomes and data collection strategy, and Section V presents a proposed timeline, deliverables, and key personnel. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 3 II. DESCRIPTION OF PUENTES AND RESEARCH QUESTIONS In this section, we describe the goals and objectives of Puentes and the activities corresponding to them, and then present the research questions to be addressed through this evaluation. A. Goals and objectives Puentes’ central goal is to increase and improve employment of vulnerable youth1 ages 16 to 29 who live in one of 15 high-crime municipalities. 2 The project has three main objectives: (1) improve the enabling environment for youths’ WFD programs and youth employment through better laws and hiring practices, (2) improve the quality of WFD services so they effectively respond to market demand and help employ greater numbers of vulnerable youth in the targeted economic sectors, and (3) improve the workforce readiness of targeted vulnerable youth. The project focuses on working with employers and helping youth develop skills they need for the high-growth sectors of agroindustry, manufacturing (including plastics), information communications technology (IT), tourism, and renewable energy. To complement this evaluation plan, we propose a logic model to guide the Puentes project evaluation (see Figure B.1 in Appendix B). This logic model is based on Puentes’ own documentation, including its monitoring and evaluation (M&E) plan, work plans, semiannual reports, and presentations; it allows us to depict how the program’s objectives and activities are expected to result in outputs and outcomes that we can measure as part of this evaluation. B. Initial assessments and activities Puentes was designed to implement activities for each of its three key objectives in a systematic and mutually reinforcing way. During the first year of activities, Puentes carried out four needs assessments and used the findings to design the project’s main activities going forward. For Objective 1, Puentes carried out a policy assessment analysis to understand challenges the project might face during its execution, and mapped the policies, regulations, laws, and private sector practices that can hinder vulnerable youth’s access to employment or training. Based on findings from a workshop that took place in El Salvador in June 2016, Puentes developed a labor market assessment. This assessment ensured that Puentes activities were based on up-to-date information about industry growth and labor market demand for specific education and technical skills, thereby allowing Puentes to better target program training and provision of support services and to identify the type of partnerships that would work best for its goals. For Objective 2, Puentes implemented an assessment of workforce training service providers to identify their strengths, efficiencies, and areas for improvement. The assessment found that courses being offered by those providers did not match the needs of the labor market, 1 Puentes defines vulnerable youth as those “in need of technical training, education, psychosocial support, or related assistance to complete their education or to find, keep and/or improve employment” (DAI 2016b). 2 The 15 municipalities include Ahuachapán, Ciudad Arce, Ciudad Delgado, Cojutepeque, Colón, Jiquilisco, Mejicanos, San Miguel, San Salvador, Santa Ana, Santa Tecla, Sonsonate, Soyapango, Tonacatepeque, and Zacatecoluca. As of April 2018, no training has taken place in Cojutepeque; however, training has been provided in four other smaller municipalities (San Martín, Chalchuapa, Sonzacate, and Sacacoyo). EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 4 and that providers had weak organizational management capacity. Finally, for Objective 3, Puentes carried out a youth assessment in order to better understand the risks, challenges, and barriers Salvadoran youth face when trying to enter the labor market, as well as the role of violence in their daily lives. Below, we discuss some of the key activities Puentes has implemented or plans to implement in order to achieve the three primary objectives of the program. Activities for Objective 1: Improved enabling environment for workforce development and employment. Based on the findings of the policy assessment analysis and the labor market assessment, Puentes identified the economic sectors with high growth and employment potential for disadvantaged youth: agroindustry, manufacturing, IT, tourism, and renewable energy.3 Through quarterly meetings with a Stakeholder Advisory Group, Puentes developed strategies for approaching key industry associations within those sectors to understand their labor needs and build relationships for potential collaboration, including the sharing of hiring projections. Puentes also elicited the private sector’s help to engage in policy reform, raise awareness, and promote national laws to reduce legal barriers to youth employment in El Salvador. Puentes has also (1) promoted laws on the inclusion of youth and other vulnerable groups through municipal youth policies, (2) developed a collaboration with the National Council for the Attention to People with Disabilities (CONAIPD), and (3) conducted an awareness-raising workshop on LGTBI (lesbians, gay, transsexual or transgendered, bisexual, and inter-sexed) and gender inclusion for USAID/El Salvador staff members. Special attention was placed on the inclusion of young women, the LGTBI community, youth with disabilities, and ex-gang members, through the implementation of inclusion workshops for private sector firms and complementary interventions for training centers. Puentes has also helped several civil society organizations (CSOs) be better prepared to support vulnerable youth. Further, Puentes is actively working to improve perceptions about the value of young Salvadoran employees among key firms within the five high-growth economic sectors. In order to increase hiring of youth, Puentes has offered workshops on best practices for more inclusive youth employment, including a strategic communications campaign targeting businesses and the Salvadoran general public. Puentes has also been actively fostering stronger linkages between training centers (TCs) and more than 50 private firms to provide youth with internship and apprenticeship opportunities. The nature of these programs varies for each of the five key economic sectors but they all seek to strengthen the links between private firms and TCs in a way that will help more young Salvadorans enter the labor market. Activities for Objective 2: Improved quality of workforce development services that effectively respond to market demand to insert at-risk youth into target economic sectors. Puentes used the results from the local workforce development service providers’ assessment to develop a strategy to give the providers technical assistance through a grant process. As of May 2018, Puentes has worked with a total of 25 training centers, and some of them have received technical assistance and capacity building support from the project, covering areas such as contract requirements, use of pre- and post-test assessments, M&E requirements, management 3 Initially, the sectors identified were tourism/services, food and beverage, business services, and manufacturing. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 5 systems, organizational structure and culture, equipment and facility upgrades, among others. Puentes has been using the technical vocational education and training capacity assessment tool (TVET-CAT4 ) to assess the institutional capacity, strengths, and weaknesses of training centers and identify opportunities for improvement. By Project Years 4 and 5, Puentes expects to have provided technical assistance to a total of 20 service providers and also to have helped 30 service providers create partnerships with private firms for recruitment of trainees. As of April 2018, out of 3,905 youth who registered, more than 2,800 had completed one of the 188 training courses offered through project-funded training centers; 2,529 vulnerable youth received some training on life skills and/or technical training. For the year 2018, Puentes has scheduled more than 100 courses, with an important proportion of those offering training in plastics and IT. Most of the technical and soft-skills trainings being provided by the training centers last between 8 and 12 weeks, with the exception of a few courses on IT and the plastics industry that last between six and nine months. Puentes expects to provide training or support services to 16,000 youth before the project ends in 2020, with around 42 percent of youth who complete the course projected to obtain new jobs, or better ones than they had before the project. All youth participating in technical and soft skills trainings receive stipends ($5 to $7 per day) to be used for transportation, food, and child care. Activities for Objective 3: Improved workforce readiness demonstrated by targeted at￾risk youth. For this objective, Puentes conducted a participatory youth assessment that looked at barriers youth faced in accessing education; their opportunities to take advantage of training, employment, and entrepreneurship opportunities; their issues with social risk and discrimination; and so on. As noted, in addition to providing technical training that responds to the demands of the private sector and other labor market actors, Puentes offers life skills training to vulnerable youth. Puentes developed and implemented its own modules on life skills training according to the requirements and needs articulated by the private firms the project is working to serve. This new curriculum drew from content provided by the training program Elijo mi futuro (“I choose my future”), the life skills program developed by the Instituto de la Juventud (INJUVE), and the modules on creative conflict resolution developed by Fundación Pro Educación de El Salvador (FUNPRES). Puentes adapted some of those materials to its specific context and needs, and instructed training centers on how to adapt and impart the new life skills curriculum. For this objective, Puentes led and/or participated in more than 70 outreach and recruitment events where project staff shared information about the project’s training activities and screened more than 5,000 interested youth for eligibility (screening includes their completing an intake form and a psychometric test or a personal interview carried out by Puentes). Puentes also is offering career counseling and mentoring services for vulnerable youth to improve their chances of finding and performing well at a high quality job.5 Puentes is planning to offer select youth entrepreneurs seed capital and start-up mentoring services. Puentes is also planning to select and 4 The TVET-CAT tool is used to evaluate service providers; Puentes used the initial results to develop a customized capacity-building plan for each service provider. 5 Career counseling and mentoring services for beneficiaries include (1) providing a brochure on career opportunities, (2) developing “business cafes” for sharing knowledge and creating peer networks, and (3) holding job orientation sessions that will include counseling on labor market demand, mentoring with recruitment managers, and registering on online job databases. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 6 develop some service providers as “one-stop shop” activity hubs to coordinate WFD services, including additional training, information on job fairs, resume writing, interview preparation, and entrepreneurship guidance, among others. By the end of Year 5 of the project, Puentes expects to have benefitted approximately 18,400 youth through training and other services and to have directly helped 6,000 of them find new or better employment. C. Evaluation questions The evaluation is designed to describe and assess the performance of Puentes and generate knowledge on the progress Puentes made toward meeting its targets. The evaluation will assess which aspects of the program worked well, which could have been improved, what barriers or facilitators seem to have helped or hindered its implementation, and what are key stakeholders’ perceptions about the impact and benefits of the services provided by the project to prepare vulnerable youth to enter into and succeed in the labor market. USAID/El Salvador and USAID more broadly, as well as policymakers interested in developing similar programs, can benefit from the findings. The evaluation is expected to be finalized by June 2020. The objectives of the evaluation are: • To assess and generate learning on how the project met its performance targets. • To provide insights on what factors helped or hindered project results and implementation. • To assess the extent to which the project prepared vulnerable youth to face and succeed in the labor market. • To provide USAID with recommendations and lessons to inform the next generation of programs that will promote positive youth development and workforce development in El Salvador. In particular, the main research questions to be addressed include: 1. What were key barriers and facilitators to the achievement of project results? a) To what extent do these barriers and facilitators include focus on design and innovation, actors involved, program rollout, and/or external factors (environment)? b) How have the grants under contract (GUC) mechanism advanced, or detained, implementation and the strengthening of local solutions? c) How did the GUCs address sustainability? 2. To what extent has Puentes reached outcome targets in the following areas: a) Employment, improved skills, and educational outcomes among beneficiaries b) Strengthening of training centers c) New/revised certified programs d) Private sector inclusion of vulnerable youth in their staff e) Sustainability of private sector and training centers relationship to have updated curricula and trainings? 3. What positive and negative results, intended or unintended, has Puentes produced at the end of the project? a) What are stakeholders’ perceptions about those results? Which ones should be prioritized in future programs? Which ones seem to be the most sustainable? EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 7 4. What are the perceptions from employers, trainees and grantees regarding the training program and the support services provided by Puentes? a) What are employers’ perceptions about: i. Trainees regarding: Technical skills and knowledge? Were employers’ demands met with the current technical skills, or do they require more specific technical skills (formal education technical skills such as 2yr degrees, or non-formal training such as professional/vocational technical skills short term trainings)? Life/soft skills? On-the-job performance for current position or potential for upward mobility? ii. Training centers and their improved curricula regarding: Courses and curricula taught in response to private sector demand; does a sustainable demand-driven model exist between education service providers and private sector to inform relevant institutions responsible for training demand? iii. Private sector: Is the private sector more open now to hire youth from high-risk areas? b) What are Puentes’ trainees perceptions about: i. The training they received, including technical training and life/soft skills training, when applicable? ii. How useful the training/certificate has been in helping them get a job? iii. How useful the support services were in getting a job and succeeding at it? iv. How Puentes has improved their future prospects in the labor market? v. Is there more openness from the private sector to hire youth? vi. What are the perceived before and after effects given the project’s intervention? c) What are grantees’ perceptions about: i. Has engagement with private sector made a difference in the education approach? ii. In what areas was your institution most strengthened? iii. What approaches, technical assistance and activities have been most critical for vulnerable youth job placement success? 5. Additional questions to consider How were gender equality, female empowerment, and social inclusion integrated in the implementation of activity interventions? Were those efforts successful? Why or why not? In the next section, we describe the data collection and analysis plans designed to address these key research questions. This page has been left blank for double-sided copying. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 9 III. EVALUATION DESIGN: DATA AND ANALYSIS To answer the research questions laid out in the previous section, Mathematica will conduct a performance evaluation of Puentes. Performance evaluations are not designed to detect project impacts or attribute changes in participant outcomes to the project or specific components of it, due to the lack of a suitable comparison group or counterfactual for recipients of their services. However, a performance evaluation is a powerful way to generate and share valuable learning and insights about the implementation and potential effects of Puentes as a whole. In coordination with USAID/Washington, USAID/El Salvador, and Puentes, Mathematica will carry out this performance evaluation relying on mixed methods and using both qualitative data (obtained through a desk review, key informant interviews, and focus group discussions) and quantitative data (mainly M&E data and administrative records from Puentes). In the next two sections, we describe the qualitative and quantitative data we will collect. Table III.1 details the research questions to be addressed as well as the data source or type of respondent. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 10 Table III.1. Evaluation questions and data sources for final performance evaluation of Puentes Research questions Quantit ative Qualitative sources M&E data1 KII w/ firms KII Puentes staff KII TC Director staff Desk review2 FGDs w/ instructor s FGDs w/ trainees and graduates 1. What were key barriers/facilitators to achieve project results? a) Focus on design and innovation, actors involved, program rollout, and external factors (environment). b) How have the grants under contract (GUC) mechanism advanced, or detained, implementation and the strengthening of local solutions? c) How did the GUCs address sustainability? X X X X X X X X X 2. To what extent has Puentes reached outcome targets in these areas: a) Employment, improved skills, and educational outcomes among beneficiaries X X X X b) Strengthening of TCs X X X X c) Number of new/revised certified programs X X X X d) Private sector inclusion of vulnerable youth in their staff X X X e) Sustainability of private sector and training centers relationship to have updated curricula and trainings X X 3. What positive and negative results, intended or unintended, has Puentes produced at the end of the project? What are stakeholders’ perceptions about those results? Which ones should be prioritized in future programs? Which ones seem to be the most sustainable? X X X X X 4. Perceptions from employers, trainees and grantees: a) What are employers’ perceptions about: i. Trainees regarding technical skills and knowledge? Life or soft skills? On-the-job performance?; ii. Training centers and their improved curricula?; iii. Private sector’s own hiring practices? X X X b) What are Puentes’ trainees perceptions about: X X X X i. The training they received, including technical training and life skills training, when applicable?; ii. How useful has the training/certificate been in helping them get a job?; iii. How useful were the support services in getting a job and succeeding at it?; How has Puentes has improved their future prospects in the labor market?; Is there more openness from the private sector to hire youth?; What are the perceived before and after effects given the project’s intervention? c) What are grantees’ perceptions about: i. Has engagement with private sector made a difference in the education approach?; ii. In what areas was your institution most strengthened?; iii. What approaches, technical assistance and activities have been most critical for vulnerable youth job placement success X X X 5. Additional questions to consider X X X X X How were gender equality, female empowerment, and social inclusion integrated in the implementation of activity interventions? Were those efforts successful? Why or why not? 1 M&E data will include inputs provided by Puentes to Mathematica for the Workforce Development report. 2 Includes review of project documents, action plans, internal reports, assessments carried out during Year 1 of the project (Policy Assessment, Labor Market Assessment, Service Provider Assessment, and Participatory Youth Assessment), and other relevant sources. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 11 A. Qualitative data We will collect and use three types of qualitative data: (1) a comprehensive desk review of Puentes quarterly and annual reports and assessments or special studies; (2) semi-structured, in￾person interviews with key informants (KIIs) including staff from Puentes, some training centers, and some large employers of Puentes trainees; and (3) focus group discussions (FGDs) with instructors from TCs and with Puentes current and former trainees. These data will help us understand how the project has been implemented, which barriers and facilitators may have affected implementation, what stakeholders perceive as the effects of the intervention, what beneficiaries think about the services and training received, and what adjustments might improve the project in the future. 1. Desk review. This review will include Puentes’ annual and semiannual reports, work plans, M&E plans and reports, and the four early assessments, as well as other documents critical to understanding Puentes’ implementation. The reports will also be important in identifying facilitators and barriers to implementation. We will consider requesting access to the reports submitted to Puentes by the TCs who are reporting aggregate information on the beneficiaries, provision of services, challenges faced, and other information; this could provide valuable insights into challenges faced by the training centers and help shape our qualitative data collection protocols. 2. Key informant interviews. We will conduct open-ended, semi-structured, in-person interviews with three of respondents; the findings will help us answer research questions 1 to 5. First, we will interview staff from Puentes in charge of project activities linked to the provision of technical assistance to TCs, technical and soft skills training, and support services to vulnerable youth, as well as ones linked to partnerships between TCs and private firms. Second, we will interview either the director or a representative from a purposively selected sample of TCs participating in the Puentes project. We will seek interviewees’ perspectives on barriers and facilitators that directly affected their work and their capacity to partner with private businesses and adjust to a changing labor market. We will choose the final sample of TCs in coordination with Puentes, making an effort to cover training across the key economic sectors, the two types of training (technical and life skills training), and as many municipalities as possible. To capture perspectives on the work done by Puentes regarding institutional strengthening, awareness raising and enabling environment for youth employment, we may also interview representatives from a few key CSOs. To address research questions 2, 3 and 4, we plan conduct interviews with representatives of private firms that have hired Puentes trainees. We will ask respondents about Puentes outreach activities, partnerships between their firms and Puentes or training centers, and how they have benefited from the project. It will also ask them to assess the job performance, technical knowledge and skills, and life or soft skills of Puentes’ trainees, perhaps asking employers to compare them with non-Puentes workers. We will define the final list of private firms in coordination with Puentes, based on the firms that have hired most Puentes trainees, and work to include firms across key economic sectors and municipalities. Ideally, the point of contact within the private firms will be a person in a managerial position, either in the Human Resources department or someone having regular, direct interaction with the trainee. See Table IV.2 for details on the number of private firms that will be approached for the interviews. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 12 3. Focus group discussions. Findings from FGDs will help us answer research questions 2, 3, 4, and 5. We will seek three types of participants for the FGDs: first, we will visit training centers and talk with instructors at the TCs to discuss the training being provided, their perceptions on trainees’ skill improvement and potential, and challenges they face during the provision of training. Second, during the visits to the training centers we will hold FGDs with youth currently receiving training under Puentes, where they will share their experiences and overall perception and satisfaction with the training (including content and organization of the courses) and the support services being received from TCs. Third, we will conduct focus group discussions with former Puentes trainees or graduates where they will share their perceptions on how Puentes training have help them in their job search, current employment, and general job performance. B. Quantitative data M&E data from Puentes. The project has been collecting M&E data on project outputs and outcomes for reporting purposes, showing how activities are being rolled out, the types of services provided, the geographic reach and gender breakdown of beneficiaries, and whether implementation targets are being met, as well as indicating whether interventions are generating their intended effects. We will use Puentes M&E data to answer research question 2; to the extent possible, we will also look at performance indicators. For the full list of indicators, please refer to Bridges for Employment’s Monitoring and Evaluation Plan (2019). C. Analysis plan In this section, we discuss the main analytical tools and data sources we will use to address the key research questions. Table III.1 breaks down all the research questions and matches them with the data type and sources we propose using to address them. The desk review, KIIs, and FGDs will provide a wealth of information for qualitative analysis. We will analyze this information and identify patterns of consensus, instances of divergent or contradictory views, and variation across dimensions of program implementation as well as across program stakeholders. We will accomplish this through two primary methods: (1) thematic framing, and (2) data triangulation. Thematic framing. To uncover patterns, themes, and issues in the qualitative data, we will develop a coding framework to capture a hierarchy of conceptual categories and classifications that are linked to the research questions and the logic model proposed in Section II.1. We will update this coding framework as we systematically review and assess the qualitative data according to Puentes’ logic model. Assigning codes to the qualitative data will enable us to efficiently access and organize information obtained through the KIIs and FGDs in order to identify themes and compile supporting evidence. For instance, representatives from different TCs may describe their experience in various ways that reflect similar underlying themes, which can be captured in our coding framework. Conversely, Puentes staff may view preparedness for the job market differently than their former trainees do; our coding framework can classify those different perspectives. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 13 Data triangulation. Because our qualitative analysis will incorporate data from several different sources (and could even include some quantitative data, like M&E indicators), we will use triangulation to test for consistency in the findings from these data sources. This process will allow us to confirm patterns or findings and identify important discrepancies. For example, when we investigate employers’ perceptions of Puentes trainees and the training centers, we will triangulate their responses with the results from focus groups with trainees, interviews with staff from Puentes and the TCs. The goal of this exercise is to develop a comprehensive understanding of the project’s potential effects by identifying instances in which qualitative findings corroborate, contradict, or help contextualize quantitative findings. Findings from the qualitative analysis using thematic framing and data triangulation will allow us to answer specific questions. For example, for questions 4.a, 4.b, and 4.c, we will combine responses from Puentes staff and TC directors with findings from the desk review to obtain an in-depth understanding on the nature, magnitude, and perceived effectiveness of the support provided by Puentes to the TCs to improve their capacity to establish and maintain partnerships with the private sector, and whether—and to what extent—Puentes has been able to transfer some of that capacity and responsibilities to the centers. This type of analysis will also allow us to identify both barriers—factors that complicated the project’s implementation—and facilitators—factors that enhanced Puentes’ effectiveness. For research question 1, we will focus on specific areas including design and rollout of training courses and support services, characteristics of training centers and trainees, and external or environmental factors. We will conduct quantitative descriptive analysis to address research questions for which quantitative data are available. For research question 2, for example, we will use M&E data to assess the extent to which Puentes reached its target objectives. M&E data will allow us to describe trends across several variables or indicators, such as graduation and employment rates. We will also use the responses to the interviews to employers to capture their perceptions about Puentes’ trainees, the curricula provided to them, and the training centers. For example, for research question 4, we will ask for employers’ satisfaction with the performance of Puentes trainees who have been hired by their firms, including not just their overall performance and technical skills and knowledge, but also their life skills and their ability to follow rules and engage with co-workers in a respectful manner. We may ask employers to compare Puentes trainees with other workers performing tasks at the same level of responsibility or technical difficulty, regardless of whether or not they have received similar training. We will use mixed-methods analysis for questions for which we have both quantitative and qualitative data. For example, we will conduct a thematic analysis of the data from interviews and focus groups, sorted by research question and by Puentes’ three objectives, to identify key themes that will help contextualize the progress shown in the M&E data and project reports. For example, in interviews with Puentes and TC staff, we may learn that logistical and contracting delays, or weak capacity of TCs affected progress towards reaching the target number of courses and the number of people receiving training. We will also use data from the key informant interviews and focus group discussions to distill stakeholders’ perceptions on the quality of implementation and to complement project reports and M&E data on the quantity of activities implemented (research question 4). EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 14 Importantly, the accounts from Puentes’ staff or instructors on the nature and quality of the trainings and support services will be compared with the accounts from trainees, who will also be asked about their overall satisfaction with the project. For research question 2 (whether Puentes reached its target outcomes), we will triangulate qualitative findings with quantitative data, including administrative and M&E data, to conclude whether Puentes has reached its target outcomes, and which barriers or limitations might have played a role in the last phase of the project. Understanding challenges, the decisions made about what work to prioritize as Puentes draws to a close, the implications of any activities that were not completed, and the lessons learned through the implementation of the project will all be valuable inputs for this evaluation. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 15 IV. DATA COLLECTION PLAN A. Qualitative data collection We will conduct one round of qualitative data collection. Mathematica staff, assisted by a local consultant, will lead the interviews and focus groups. The interviews and focus groups will be recorded, and interviewers will carefully take notes that will be transcribed later. For the focus groups, a local consultant will help with logistical support and bring important knowledge of the local context and target participants to help guide the discussion. The first round of qualitative data collection is scheduled for October 2019 and will include (1) interviews with Puentes key staff in charge of activities linked to the three Objectives; (2) interviews with representatives of a selected group of TCs; and (3) focus groups with instructors and current trainees. For the focus groups, we will carefully schedule visits to training centers in order to recruit youth currently receiving the training and capture their perceptions about the training and services provided by Puentes. Puentes has agreed to give us the necessary contact information (see Table IV.2 for the estimated number of interviews and focus groups by type of respondent and round.) The second round of data collection is tentatively scheduled for January 2020 and will include interviews with (1) CSO and NGOS, (2) focus groups with Puentes graduates, (3) interviews with employers, and (4) interviews with Puentes key staff. B. Quantitative data collection We will review all indicators for monitoring and evaluation compiled by Puentes since the beginning of the project, according to its M&E Plan (therefore, we will not conduct any primary data collection for those indicators). We expect to have access to up-to-date indicators as soon as they are available. Table IV.1 shows the main outcomes or indicators that we will be targeting using M&E data. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 16 Table IV.1. Main outcomes or indicators from M&E data to be provided by Puentes Data source Outcome or indicator M&E data (indicators) Private sector • Number of firms implementing new practices for vulnerable youth • Number of private firms with awareness training about vulnerable youth • Number of industry-recognized certifications and/or accreditations Training centers • Number of training centers with technical assistance from Puentes • Number of training centers strengthened by Puentes • Number of training centers with partnerships with private sector Vulnerable youth6 • Number of eligible youth registered for training • Percentage of registered youth who are female • Number and percentage of eligible youth graduated from training • Number of individuals who benefitted from Puentes • Number of individuals who indirectly benefitted from Puentes • Number and percentage of trainees with new or better employment • Number and percentage of trainees with improved knowledge or skills • Number and percentage of trainees who report enrolling in formal school 6 An additional indicator may be salary or income, as reported by the trainees to the training centers. We will use this indicator depending on the data’s availability and quality and the speed with which the data are reported. For the specific case of trainees on Plastics, we will consider the fact that, after completion, trainees are expected to be hired for pasantías (internships) for a period of three to five months depending on the type of course received. This will restrict the number of trainees—especially from the plastics sector—whom we could use to track salary improvements. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 17 Table IV.2. Plans for qualitative and quantitative data collection Type of data Source of data Respondent Sample size Qualitative data Interviews Puentes staff 5 Interviews CSO and NGO representatives 3 Interviews Training center representatives 6 Interviews Employers 4-5 large employers Focus groups Instructors 2-3 groups, 4-5 instructors each Focus groups Current trainees 3-4 groups, 5-10 trainees each Focus groups Graduates 3-4 groups, 5-10 trainees each Quantitative data M&E data Puentes n.a. Note: Interviews and focus groups will take place in February 2020. n.a. = not applicable. * Final numbers for both rounds to be determined in coordination with Puentes. This page has been left blank for double-sided copying. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 19 V. TIMELINE, DELIVERABLES, AND PERSONNEL A. Timeline and deliverables Table V.1 shows the timeline for the evaluation, with key tasks for data collection, analysis, and reporting, by month and year. Table V.1. Evaluation timeline 2019 2020 S O N D J F M A M J Task 1: Instruments and data collection 1.1 Prepare qualitative instruments 1.2 Collect qualitative data 1.3 Document review Task 2: Analysis and reporting 2.1 Analysis 2.2 Draft report 2.3 Final report Task 1. Instrument development and data collection (September 2019 – January 2020). In September and October, we will prepare the instruments and protocols for the interviews and focus groups, which we will share with USAID/El Salvador and USAID/Washington, and Puentes. The first round of interviews and focus groups will be carried out in October 2019 and the second round is planned for January of 2020. Task 2. Analysis and reporting (December 2019 – March 2020). Analysis of quantitative data could begin as early as December 2019 and will continue with the qualitative analysis until March 2020. In March 2020, we will start writing the final report. We will submit a draft of the final report to USAID/El Salvador and USAID/Washington by May 2020 for comments and feedback, which we will address for the submission of the final report by June 2020. B. Personnel The performance evaluation of Puentes will require dedicated U.S.-based and in-country team members with subject matter expertise in workforce development projects, performance evaluation design and analysis, and management of data collection efforts. Our U.S.-based staff bring the experience and skills required to successfully lead this evaluation. We have strong management, communication, and coordination capabilities; deep experience in performance evaluation; subject expertise in workforce development, and a positive working relationship with Puentes and USAID/ES. The staff includes the following: • Dr. Nancy Murray will provide overall management advice and leadership, and serve as point of contact with USAID/Washington, USAID/El Salvador, and Puentes project director. EVALUATION PLAN FOR PUENTES PARA EL EMPLEO MATHEMATICA POLICY RESEARCH 20 • Dr. Larissa Campuzano will manage the project’s financial and staffing needs and provide technical advice to Ms. Costa on tasks related to design, data collection, analysis, and reporting. • Ms. Patricia Costa will lead the evaluation design’s implementation, analysis, and reporting tasks. She will closely coordinate with Puentes staff, and will manage the local consultant. • Mr. Camila Fernandez will lead the development of qualitative instruments and advise on qualitative analysis. • Other Mathematica staff may include a junior analyst who will support Ms. Costa in the development of instruments, help support analysis tasks, and work on final reports. • A local consultant will help coordinate and co-lead the interviews and focus groups with Ms. Costa. APPENDIX A LOGIC MODEL FOR THE PUENTES PROJECT This page has been left blank for double-sided copying. APPENDIX A MATHEMATICA POLICY RESEARCH Figure A.1. Logic model for the Puentes project This page has been left blank for double-sided copying. APPENDIX B PRESENTATIONS FROM 2016 This page has been left blank for double-sided copying. APPENDIX B MATHEMATICA POLICY RESEARCH 27 APPENDIX B MATHEMATICA POLICY RESEARCH 28 APPENDIX B MATHEMATICA POLICY RESEARCH 29 APPENDIX B MATHEMATICA POLICY RESEARCH 30 APPENDIX B MATHEMATICA POLICY RESEARCH 31 APPENDIX B MATHEMATICA POLICY RESEARCH 32 APPENDIX B MATHEMATICA POLICY RESEARCH 33 APPENDIX B MATHEMATICA POLICY RESEARCH 34 APPENDIX B MATHEMATICA POLICY RESEARCH 35 APPENDIX B MATHEMATICA POLICY RESEARCH 36 APPENDIX B MATHEMATICA POLICY RESEARCH 37 APPENDIX B MATHEMATICA POLICY RESEARCH 38 APPENDIX B MATHEMATICA POLICY RESEARCH 39 APPENDIX B MATHEMATICA POLICY RESEARCH 40 APPENDIX B MATHEMATICA POLICY RESEARCH 41 APPENDIX B MATHEMATICA POLICY RESEARCH 42 APPENDIX B MATHEMATICA POLICY RESEARCH 43 APPENDIX B MATHEMATICA POLICY RESEARCH 44 APPENDIX C IMPACT EVALUATION DESIGN, 2017 This page has been left blank for double-sided copying. APPENDIX C MATHEMATICA POLICY RESEARCH 47 APPENDIX C MATHEMATICA POLICY RESEARCH 48 APPENDIX C MATHEMATICA POLICY RESEARCH 49 This page has been left blank for double-sided copying. APPENDIX D SUMMARY OF PILOT OF RANDOM ASSIGNMENT This page has been left blank for double-sided copying. An Affirmative Action/Equal Opportunity Employer MEMORANDUM 1100 1st Street, NE, 12th Floor Washington, DC 20002-4221 Telephone (202) 484- 9220 Fax (202) 863-1763 www.mathematica￾mpr.com TO: Chris Moore, Orlando Hidalgo, Leesa Kaplan, Mario Rivas, Karen Towers, Barbara Knox-Seith, Yolanda Martinez, and Annie de Valencia FROM: Nancy Murray, Jorge Ugaz, and Larissa Campuzano DATE: 10/10/2017 SUBJECT: Summary of the Pilot of Random Assignment for the Puentes para el Empleo Evaluation I. INTRODUCTION In June 2017, Mathematica, Development Alternatives Incorporated (DAI), and the U.S. Agency for International Development (USAID) agreed on the importance of carrying out a pilot of a randomized controlled trial (RCT) for Puentes para el Empleo. The pilot’s main objectives were to: 4. Deepen our understanding of the Puentes program, with a special focus on outreach, eligibility screening, recruitment and selection processes, and the pace of program rollout— as well as the implications for the evaluation. 5. Design and test the random assignment process—including the approach to communicating the results of random assignment to Puentes, the training centers, and program applicants—and agree on final protocols, including those used to reach out to people in the control group. 6. Obtain the applicant data collected by Puentes and the training centers through intake forms, assess the quality and availability of data collected in the recruitment and application process, and make any necessary improvements or additions (whether needed for program implementation and monitoring or for the impact evaluation). 7. Use the findings from the pilot to inform the design of the impact evaluation, and give input on the feasibility of different options for the evaluation design. We are sharing this memo with Puentes para el Empleo, USAID/El Salvador, and USAID/Washington to continue our discussion on the potential evaluation of Puentes para el Empleo, the key research questions, and the feasibility of conducting a three-arm design, which would include a training component on soft skills only. MEMO TO: Chris Moore, Orlando Hidalgo, Leesa Kaplan, Mario Rivas, Karen Towers, Barbara Knox-Seith, Yolanda Martinez, and Annie de Valencia FROM: Nancy Murray, Jorge Ugaz, and Larissa Campuzano DATE: 10/10/2017 PAGE: 54 II. INITIAL PLAN FOR THE PILOT To provide context for the pilot implementation, we describe here the main features planned for the pilot study. Pilot sample and selection of courses. Puentes and Mathematica agreed on a pilot sample of about 300 people (encompassing both the treatment and control groups). The plan was to select a sample of courses from the estimated 40 courses that Puentes had planned to implement between July and September 2017 in partnership with INSAFORP.7 With an average duration of three months, the courses to be included in the pilot would be completed by the end of 2017. Outreach, screening, and recruiting. Outreach, screening, and recruiting of applicants interested in these courses are conducted by Puentes and the training centers the project enters into partnership with. Puentes can and will play a more active role in outreach and recruitment if a training center has trouble reaching the required number of eligible individuals. Eligibility was expected to be assessed by Puentes or the training centers through a three-step process: (1) the applicant completes an intake form (see Appendix C), (2) a psychometric test is given to the applicant, and (3) the applicant is interviewed. The two main prerequisites for eligibility were age (under 30) and residence in given municipalities. Puentes is responsible for giving final approval of the eligibility of each applicant. Random assignment groups. In coordination with INSAFORP and the training centers, Puentes determined that each class should have 25 students. Mathematica and Puentes then agreed that for any training course to be included in the random assignment pilot, a total of 40 interested and eligible applicants would have to be identified, so that 25 students would be assigned to the treatment group, 12 to the control group, and 3 to a waitlist. Thus, the final sample would have 320 students randomly assigned to one of those three groups. Mathematica developed a model and a protocol to conduct the random assignment for each class list in an efficient and transparent manner, stratifying the sample by the gender of the eligible applicants within each course list. Waitlists. These are helpful for implementing oversubscribed programs; they are also key tools in implementing evaluations with a random assignment design. For Puentes, staff had already estimated that out of 25 people assigned to the treatment group, a handful might not take up the program’s offer to participate—because they found a job, went back to school, chose a different training opportunity, or simply lost interest. A waitlist gave Puentes and the training center a small pool of interested and eligible people who could quickly be contacted and offered training in order to ensure all class spots were filled and the number of training (and study) participants maximized. This approach is preferable to having the training centers recruit people on their own after random assignment has taken place; the objective is to maintain the same 7 INSAFORP is the Instituto Salvadoreño de Formación Profesional, which partnered with Puentes for a round of courses implemented in 2017. MEMO TO: Chris Moore, Orlando Hidalgo, Leesa Kaplan, Mario Rivas, Karen Towers, Barbara Knox-Seith, Yolanda Martinez, and Annie de Valencia FROM: Nancy Murray, Jorge Ugaz, and Larissa Campuzano DATE: 10/10/2017 PAGE: 55 process for all participants and reduce the risk of introducing different incentives to training centers regarding recruitment. Random assignment of people to groups and communication of the results. Mathematica and Puentes agreed to share information on eligible applicants through Mathematica’s secure SharePoint platform in order to protect eligible applicants’ personal information and to nimbly organize information shared by Puentes and communicate results. Puentes was expected to work with the training centers to create the course lists of 40 eligible applicants and upload the lists to the SharePoint site so Mathematica could securely download the information, conduct random assignment, and quickly communicate the individual assignments back to Puentes within three days. Puentes was expected then to download the results and share the names of the individuals assigned to the treatment group and waitlist with the respective training centers. Mathematica was expected to communicate the results to the people assigned to control groups, give them information on other existing programs, and remind these people that they could not register for any other Puentes training course for the next four to five months.8 In order to make the process efficient, Puentes agreed to try to share the list of courses and participants with Mathematica in batches, ideally in two batches so Mathematica could conduct the random assignment process in two rounds. We agreed that this process would depend on the flow of applications, the time it took for eligibility screening, the starting dates for the courses, and other factors. Initially, we also agreed that Puentes would try to upload the lists for random assignment at least five business days before the first day of training so Mathematica would have enough time to communicate the results to the study participants. Applicants’ data. Training centers were responsible for (1) properly inputting into their own databases all information provided by applicants through the intake forms, and (2) sharing this information electronically with Puentes. Puentes agreed to share with Mathematica a database containing all applicants’ data (from all training centers) sometime after the applicants from the last course in the pilot had been randomly assigned. Integrity of the study design. Maintaining the integrity of the random assignment results— basically ensuring that the people assigned to the control group do not receive training offered to the treatment group—is essential for the success of the evaluation. Puentes agreed to reinforce the importance of the integrity of random assignment both within its own team and with the training centers—reminding them how important it was to ensure that no participants assigned to the control group would receive training. It was also agreed that Puentes would flag any deviations from the agreed-upon protocols with Mathematica so we could identify solutions, prioritizing both the integrity of the study design and Puentes’ implementation. During the pilot, 8 In the evaluation of Puentes, individuals assigned to the control group will not be able to apply to or register for other courses provided by Puentes for a period of 15 months after the random assignment. MEMO TO: Chris Moore, Orlando Hidalgo, Leesa Kaplan, Mario Rivas, Karen Towers, Barbara Knox-Seith, Yolanda Martinez, and Annie de Valencia FROM: Nancy Murray, Jorge Ugaz, and Larissa Campuzano DATE: 10/10/2017 PAGE: 56 Puentes demonstrated strong commitment to the different protocols and processes, which played a large role in the pilot’s overall success. III. PILOT IMPLEMENTATION Selecting courses for the pilot. A total of eight courses were included in the pilot; this number guaranteed a pilot sample of more than 300 individuals. Table 1 shows the final list of courses that were included, the training centers, the focus of training, and the training start dates. The initial plan was to select the pilot courses so as to maximize the diversity of topics, municipalities, and training centers. But we ultimately opted to include the first eight courses to be rolled out under the INSAFORP agreement in the pilot once we learned that Puentes would implement only half the number of courses it initially planned to. Table 1. Courses and enrollment by study groups Training Center Municipality Course Starting date Enrollment by group T C W CF Total OEF Zacatecoluca Procesamiento artesanal de productos lácteos 21-Aug-17 20 0 0 5 25 OEF Zacatecoluca Producción de hortalizas 21-Aug-17 20 0 2 3 25 OEF Zacatecoluca Mantenimiento y Reparación de Computadoras 21-Aug-17 19 0 0 6 25 Agape Santa Tecla Recamarero/a o Camarista 21-Aug-17 19 0 1 5 25 OEF Ciudad Delgado Mantenimiento y Reparación de Computadoras 21-Aug-17 10 1 1 13 25 OEF Ciudad Arce Producción de hortalizas 21-Aug-17 10 0 0 15 25 Agape Zacatecoluca Recamarero￾Zacatecoluca 28-Aug-17 19 0 1 5 25 OEF Ciudad Arce Producción de hortalizas 18-Sep-17 12 0 2 7 21 OEF – Asociación para la Organización y Educación Empresarial Femenina de El Salvador. T = treatment; C = control; W = waitlist; CF = identified by the training center outside of the pilot process. Working with INSAFORP’s intake procedures resulted in some challenges for Puentes. In some cases, INSAFORP screened and did not approve a few training centers; in other cases, the additional procedures and logistical steps required by INSAFORP discouraged some training centers from participating. These challenges reduced the total number of training centers working with Puentes, and therefore also affected the number of courses that Puentes was able to roll out immediately. Puentes has told us this is not a reflection of its future capacity to roll out courses, given that the next rounds of training courses will not involve INSAFORP. Nevertheless, these or similar issues may emerge during later phases of implementation with other training centers, and we may want to anticipate them. MEMO TO: Chris Moore, Orlando Hidalgo, Leesa Kaplan, Mario Rivas, Karen Towers, Barbara Knox-Seith, Yolanda Martinez, and Annie de Valencia FROM: Nancy Murray, Jorge Ugaz, and Larissa Campuzano DATE: 10/10/2017 PAGE: 57 Recruiting and screening participants. Recruitment took place without major impasses, as the training centers, sometimes in collaboration with Puentes, were able to recruit, screen for eligibility, and register for application the required number of eligible individuals (40) for the respective courses. Oversubscription was not a major hurdle for the training centers. The processes of screening, interviews, psychometric tests, and final applications and their approval by Puentes were carried out according to plan. Applicant data from intake forms.9 A few training centers shared this information with Puentes as they entered the information, but Puentes identified some data quality issues with the information, which it is still trying to sort out. To date, Mathematica has not been able to review the data from the intake forms for the pilot participants. In order to streamline the data entry process, Puentes is working on an interactive, web-based system (called Sistema de Informacion, Monitoreo y Evaluacion-SIME) where training centers can directly enter the applicants’ data so it can be reviewed in real time by Puentes, and the data quality of key indicators can be ensured. It will be key to have this system functional for the impact evaluation of Puentes so this information can be easily uploaded to SharePoint for the random assignment (and for later analysis of baseline information across study group participants). Random assignment and communication of results. Puentes put together the lists of 40 eligible applicants for the eight courses and uploaded them through the SharePoint platform between August 12 (first group of courses) and September 14 (the last course). For each round, Mathematica took between one and two days to carry out the random assignment every time lists were uploaded. The random assignment process worked as planned: Mathematica stratified each of the eight lists by gender, and assigned the 40 individuals in each list to either a treatment group, a control group, or a waitlist. Random assignment for the eight courses was implemented in four batches, or rounds. The first batch included the first five courses, and the next three batches included one course each. After carrying out random assignment, Mathematica shared the results with Puentes through SharePoint so Puentes in turn could share those results with the respective training centers. This process took place without major incident, and Mathematica wrote brief memos to summarize the results and shared them with Puentes and USAID. Table 2 shows the consolidated results from the random assignment. With similar ratios for the assignment groups for women and men, overall, women made up 64 percent of all eligible applicants before random assignment; the high proportion of female applicants is even more pronounced for courses not related to repairing computers (about 70 percent of women). 9 The intake form currently being used by Puentes (Appendix C) is an adaptation of the RUP (Registro Único de Participantes) form used by the Secretaría Técnica de la Presidencia (STP). Puentes seems open to the option of Mathematica suggesting a couple of additional questions to the form if necessary. MEMO TO: Chris Moore, Orlando Hidalgo, Leesa Kaplan, Mario Rivas, Karen Towers, Barbara Knox-Seith, Yolanda Martinez, and Annie de Valencia FROM: Nancy Murray, Jorge Ugaz, and Larissa Campuzano DATE: 10/10/2017 PAGE: 58 Table 2. Consolidated results of random assignment, by group and gender Courses 1 to 8 Assignment groups Total by T C W Total gender (%) Women 131 59 17 207 64 Men 69 37 9 115 36 Total 200 96 26 322 100 Total by group (%) 62 30 8 100 T = treatment; C = control; W = waitlist. Use of waitlists. Across the eight courses, some participants declined the training offer when they were contacted by the training center. In a few cases, the training centers reached out to Puentes, asking for replacements from the waitlist. However, because the training centers were required by INSAFORP to have at least 20 students registered and attending by the end of the second day of training, some centers did not wait to receive the waitlists and searched for eligible candidates on their own (following the protocol already established by Puentes). Thus, for most courses, people assigned to the waitlists were not contacted at all. In order to avoid this issue in the future, we recommend that Puentes share both lists (the individuals assigned to the treatment group and those assigned to the waitlist) simultaneously with training centers so the centers can contact individuals on the waitlist right away as necessary to ensure there are enough participants in the treatment group. However, it needs to be made clear that the training centers should reach for the waitlist only if they cannot find or reach people assigned to the treatment group. Importance of location for participation. The location for the course called Produccion de hortalizas in Ciudad Arce (by OEF) changed a few days before the start of training. When people assigned to the treatment group were contacted and informed about the change of location, 15 out of 25 declined the invitation, citing safety concerns or long traveling distances. Because the training center had to have at least 20 people by the second day of the course—following INSAFORP requirements—the center quickly identified and screened 15 people independently (without using the waitlist). A similar thing happened with the course “Mantenimiento y reparacion de computadoras” in Ciudad Delgado, where the location for the training was “to be determined” at the time of application; once the location was identified, again 15 people out of 25 declined the invitation due to similar concerns. Clearly, participation is sensitive to location, and it will be extremely important that similar last-minute changes to the training location do not take place, or that they take place before random assignment has been carried out and the applicants are contacted again to reassess their eligibility and interest. We should be mindful of the fact that the evaluation will focus on an intent-to-treat impact, so that people assigned to the treatment group will be analyzed as such regardless of whether they actually participate in training or not. Therefore, if a large number of people in the treatment group refuse training, it could attenuate any program impact. MEMO TO: Chris Moore, Orlando Hidalgo, Leesa Kaplan, Mario Rivas, Karen Towers, Barbara Knox-Seith, Yolanda Martinez, and Annie de Valencia FROM: Nancy Murray, Jorge Ugaz, and Larissa Campuzano DATE: 10/10/2017 PAGE: 59 Puentes shares lists for random assignment. It was challenging for Puentes to compile and share with Mathematica the list of 40 eligible applicants more than five days in advance of the beginning of the training and to share those lists in two or three batches. Although that issue did not represent a problem for the pilot, in an impact evaluation there will be more course participants in the pipeline, and this could potentially affect the prompt sharing of the results of random assignment and disrupt the course calendar. Final enrollment rates. Table 1 shows key enrollment information taken after the second day of training in each course. On average, and excluding the two courses mentioned above, 19 people out of 25 assigned to the treatment groups were attending the training by the end of the second day. This rate of 80 percent enrollment by Day 2 of training is within Puentes’ expected range, and will be taken into consideration for the power calculations in the impact evaluation. All training centers were able to register 25 students (although they did resort to recruiting participants on their own to fulfill the required total enrollment), as seen in the column “CF” in Table 1. Table 1 also confirms that training centers barely included any people from the waitlists. However, one individual assigned to the control group was receiving training, and the circumstances of this situation are being investigated by Puentes. Communicating with people assigned to the control groups. Mathematica hired a local consultant, Rafael Rendon, to contact the people assigned to the control groups, inform them of the results, and share information on other courses from INJUVE and INSAFORP that were available to them in their municipalities. The consultant was able to contact—by cell phone or text—77 percent (74 out of 96) of the people assigned to the control group in the eight courses. Some of the people who were not reached had changed their phone numbers, or the phone numbers they had provided were wrong or not in service. Our consultant perceived that these people had little interest in receiving information about the other courses offered by INJUVE and INSAFORP. IV. IMPLICATIONS FOR THE IMPACT EVALUATION Overall, the pilot of random assignment was implemented successfully. Our communication processes with Puentes were tested and they worked as planned. Our random assignment method was also tested, and worked without major impasse. Puentes used a successful messaging strategy to bring the training centers on board in such a way as to ensure they would participate in safeguarding the integrity of the random assignment design. Finally, we also obtained important information on oversubscription, course uptake, and other factors that will be key to the evaluation. Reflecting on our original objectives for the pilot, we summarize here the important lessons we learned and their implications for the impact evaluation of Puentes. 8. Clear understanding of Puentes program rollout - Mathematica clearly understands Puentes’ processes for outreach, recruitment, screening for eligibility, and the application and its approval. During the pilot, the involvement of INSAFORP restricted the number of courses that could be rolled out and the number of MEMO TO: Chris Moore, Orlando Hidalgo, Leesa Kaplan, Mario Rivas, Karen Towers, Barbara Knox-Seith, Yolanda Martinez, and Annie de Valencia FROM: Nancy Murray, Jorge Ugaz, and Larissa Campuzano DATE: 10/10/2017 PAGE: 60 training centers that were involved. It will be helpful to discuss in detail any new barriers or challenges that may emerge when Puentes works with new partners to offer courses to be included in the evaluation. Working along with Puentes, we must scope out any potential factors that may affect course rollout in later phases of program implementation. - In Appendix A, we present a timeline for the evaluation, considering that both the Puentes contract and the LAC Reads Evaluation contract end in October and September of 2020, respectively. Considering the time needed for data collection, analysis, report writing, and sharing of results, and assuming an average length of three months for each training course, we estimate that we will need a period of eight (8) months between March and October of 2018 to roll out the necessary courses to achieve our desired sample size. - It is important to discuss the number of courses to incorporate in the evaluation, and what that may imply for Puentes. Based on preliminary power calculations, we estimate that the most simple evaluation design option currently being considered, which is a two-arm design with only one treatment and one control group, would require a total sample size of about 2,550 participants.10 Assigning two-thirds of that sample to the treatment group and one-third to the control group, around 1,710 people (67 percent) would be assigned to the treatment group, and 840 assigned to the control group (33 percent).11 Under the assumption that every class will train 25 individuals, Puentes will need to roll out about 70 courses over eight months, most likely between March and October of 2018. - Overall, we recommend that Puentes and the training centers follow a similar process for recruitment, screening, and application as they did with the pilot.12 9. Random assignment process - Regardless of the design option chosen for the evaluation of Puentes, we would like to suggest that random assignment be carried out in batches, and not on a rolling basis as we did during the pilot. Considering that we have programmed the random assignment, that stratification is fairly straightforward, and that fewer rounds of random assignment are preferable (from the communication, waitlist, and sample management perspectives), we propose to carry out random assignment every two or three weeks with the course lists that are available; the final timing to be agreed upon with Puentes, depending on the rate of recruitment. This approach should allow Puentes to keep moving forward without 10 To detect an impact of 3.2 percentage points in the likelihood of being employed. 11 A separate subgroup of 544 eligible applicants will also have to be included for the waitlists (which are usually not considered part of the “evaluation sample”). 12 See Appendix B for a proposed flowchart on random assignment. MEMO TO: Chris Moore, Orlando Hidalgo, Leesa Kaplan, Mario Rivas, Karen Towers, Barbara Knox-Seith, Yolanda Martinez, and Annie de Valencia FROM: Nancy Murray, Jorge Ugaz, and Larissa Campuzano DATE: 10/10/2017 PAGE: 61 having to wait too long for the results of the random assignment processes, and allow us to work efficiently with Puentes to manage the communication with study participants. - We recommend that waitlists include seven or eight eligible individuals, considering the estimated proportion of individuals assigned to the treatment groups who will ultimately decline participation is around 20 percent, based on our pilot experiment. The main objective for increasing the size of the waitlist is to prevent training centers from recruiting participants independently after random assignment has already taken place. The final size of the waitlist should be confirmed by Puentes, because it will have implications on their team’s efforts on recruitment. It is also important to note that individuals from the waitlist will not be included in the final analysis sample, regardless of whether they accept or refuse to attend the trainings. - We estimate that Mathematica will need three business days to complete its random assignment tasks. Considering that it would be ideal for people assigned to the treatment or control groups to be contacted at least three business days before the beginning of training, we propose that Puentes uploads to SharePoint the lists of eligible applicants at least six (6) business days before the beginning of training. This new process will be crucial to guarantee that any unexpected problem can be solved quickly enough to avoid the risk of having to drop the course from the evaluation. It will also be crucial to make sure people in the treatment and control groups are promptly informed of the results so they can plan accordingly. 10. Applicants’ data and follow-up data collection - Mathematica will carefully evaluate if there is a need to include more questions in the current intake form that collects applicants’ data (see Appendix C), either for assessment of baseline equivalence of groups or for follow-up of study participants. - We recommend that all contact information provided by the applicants in the intake forms is included in the online platform SIME. This particularly applies to mobile phone and landline numbers and Facebook and social media account information provided by the applicants in the intake forms. This will increase the likelihood of contacting all individuals assigned to the control groups after the random assignment. - It will be crucial for the impact evaluation of Puentes that the “Sistema de Informacion, Monitoreo y Evaluacion–SIME” is working efficiently by the time Puentes and the training centers begin their outreach and recruitment tasks for those courses. This platform would considerably increase the quality and consistency of the data being reported by the training centers before the random assignment is carried out. - According to the timeline in Appendix A, Puentes would have a maximum of eight months to roll out the training courses for the evaluation. We propose two rounds of quantitative follow-up (endline) data collection; the first round would collect data from the courses rolled out during the first four months, and the second round from the MEMO TO: Chris Moore, Orlando Hidalgo, Leesa Kaplan, Mario Rivas, Karen Towers, Barbara Knox-Seith, Yolanda Martinez, and Annie de Valencia FROM: Nancy Murray, Jorge Ugaz, and Larissa Campuzano DATE: 10/10/2017 PAGE: 62 courses rolled out during the next four months. If we assume a follow-up period of 12 months after training, then the first and second rounds of follow-up data collection would take place in August 2019 and January 2020, respectively. - For the implementation evaluation study, we propose two rounds of qualitative data collection that can be implemented concurrently with the quantitative data collection. 11. Evaluation design for Puentes - We need to decide whether to test the hypothesis that training in soft skills only can improve young Salvadorans’ employability and labor market outcomes. If so, then we will be able to move forward with the three-arm design discussed earlier this year. We note the following implications of this:  Puentes should confirm it is open to offering and providing training in soft skills and/or technical skills to a similar pool of eligible applicants, regardless of whether they already have technical skills or not. In other words, the sample of participants randomly assigned to receive soft skills training only needs to come from the same eligibility pool of applicants; the eligibility criteria must be identical.  During the pilot, we learned that the training centers strongly believe that both type of skills should be taught at the same time, instead of exhausting one type of skills training first and following it with the other. The latter practice apparently increases the chances that students will drop the course or simply not show up on the days that focus only on soft skills.  The timeline implications from Appendix A are the same regardless of whether the impact evaluation of Puentes considers a two-arm or a three-arm design. However, while the two-arm design will require around 2,550 participants and 70 courses, a three-arm design will require around 5,600 participants and 168 courses. If the timeline stays the same, then this will require a larger level of effort from Puentes and the training centers.  We also learned that geographic distance and convenience matter significantly for participation and attendance. Therefore, soft-skills courses will have to be provided at the respective training centers, and not in a centralized location within each municipality. - We confirmed the existence of some imbalance in the distribution of the participants by gender. Yet, that imbalance may still allow for the possibility of exploring potential impact heterogeneity under Puentes. The other potential variable to explore for impact heterogeneity is economic sector of the training. Yet, we cannot confirm whether that will be possible until we know the possible breakdown of the training courses by economic sector. APPENDIX D MATHEMATICA POLICY RESEARCH APPENDIX A (OF SUMMARY OF THE PILOT) POTENTIAL TIMELINE FOR EVALUATION OF PUENTES APPENDIX A (OF SUMMARY OF THE PILOT) MATHEMATICA POLICY RESEARCH Potential timeline for evaluation of Puentes APPENDIX A (OF SUMMARY OF THE PILOT) MATHEMATICA POLICY RESEARCH APPENDIX D MATHEMATICA POLICY RESEARCH APPENDIX B (OF SUMMARY OF THE PILOT) PROCESS FOR RANDOM ASSIGNMENT APPENDIX B (OF SUMMARY OF THE PILOT) MATHEMATICA POLICY RESEARCH APPENDIX E UPDATED POWER CALCULATIONS FOR IMPACT EVALUATION This page has been left blank for double-sided copying. APPENDIX E MATHEMATICA POLICY RESEARCH 73 APPENDIX E MATHEMATICA POLICY RESEARCH 74 APPENDIX E MATHEMATICA POLICY RESEARCH 75 This page has been left blank for double-sided copying.