Monitoring & Evaluation Support for Collaborative Learning and Adapting (MESCLA) 1 MONITORING AND EVALUATION SUPPORT FOR COLLABORATIVE LEARNING AND ADAPTING (MESCLA) ACTIVITY HONDURAS EMPLEANDO FUTUROS PILOT IMPACT EVALUATION PHASE 1: FINDINGS AND RECOMMENDATIONS July 25, 2018 Contract Number: AID-OAA-I-15-00019 – Task Order Number: AID-522-TO-16-00002 This publication was produced for review by the United States Agency for International Development. It was prepared by Dexis Consulting Group. Monitoring & Evaluation Support for Collaborative Learning and Adapting (MESCLA) i MONITORING AND EVALUATION SUPPORT FOR COLLABORATIVE LEARNING AND ADAPTING (MESCLA) ACTIVITY HONDURAS EMPLEANDO FUTUROS PROGRAM PILOT IMPACT EVALUATION PHASE 1: FINDINGS AND RECOMMENDATIONS VERSIONS: • Draft 1 – July, 25, 2018 SUBMITTED TO: USAID/Honduras Prepared by: Dexis Consulting Group DISCLAIMER: The authors’ views expressed in this document do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Monitoring & Evaluation Support for Collaborative Learning and Adapting (MESCLA) 1 TABLE OF CONTENTS Table of Contents.............................................................................................................................................................. 1 Acronym List ..................................................................................................................................................................... 2 1 Executive Summary.................................................................................................................................................. 3 2 Introduction ............................................................................................................................................................... 5 2.1 Background ....................................................................................................................................................... 5 2.2 Workforce Development “Empleando Futuros” Activity..................................................................... 5 3 Evaluation Purpose and Evaluation Questions................................................................................................... 8 3.1 Evaluation Purpose.......................................................................................................................................... 8 3.2 Evaluation Questions...................................................................................................................................... 8 4 Evaluation Methods and Limitations .................................................................................................................... 8 4.1 Data Collection Tool: Violence-Involved Persons Risk Assessment.................................................. 8 4.2 Data Collection Methods.............................................................................................................................. 9 4.3 Issues and Limitations...................................................................................................................................10 5 Findings, Conclusions and Next Steps...............................................................................................................11 5.1 Feasibility of Conducting a Full-Scale Impact Evaluation......................................................................11 5.2 Findings: Do Youth in the Treatment and Control Groups Show Different Post-Activity Employability?...............................................................................................................................................................11 5.3 Findings: Do Youth in the Treatment and Control Groups Show Different Post-Activity Risk of Violence? .......................................................................................................................................................................12 6 Recommendations..................................................................................................................................................12 7 References................................................................................................................................................................13 8 Annex ........................................................................................................................................................................14 Monitoring & Evaluation Support for Collaborative Learning and Adapting (MESCLA) 2 Acronym List CBT Cognitive Behavioral Therapy CDCS Country Development Cooperation Strategy DO1 Development Objective 1 IE Impact Evaluation MESCLA Monitoring and Evaluation Support for Collaborative Learning and Adapting NDIGD Notre Dame Initiative for Global Development PE Performance Evaluation RCT Randomized Controlled Trial USAID United States Agency for International Development VIP-RA Violence-Involved Persons Risk Assessment WFD Workforce Development Monitoring & Evaluation Support for Collaborative Learning and Adapting (MESCLA) 3 1 EXECUTIVE SUMMARY “Empleando Futuros”, the United States Agency for International Development (USAID) Honduras’ Workforce Development Activity (WFD), aims to contribute to the Mission’s Country Development Cooperation Strategy (CDCS)1’s Development Objective 1 (DO1), Sub-Intermediate Result 1.1.2, “Quality services that protect against violence increased” (USAID, 2015). Given the dearth of literature on the link between workforce development and violence in Honduran youth, USAID/Honduras commissioned the Monitoring and Evaluation Support for Collaborative Learning and Adapting (MESCLA) Activity to perform an impact evaluation (IE) of the Activity and clarify the link between WFD, risk of perpetuating violence, employment, and employability. MESCLA subcontracted Notre Dame Institute for Global Development (NDIGD) to lead the IE of the WFD activity. Because the literature suggested mixed results (and even in studies with positive results, these were small to moderate) in terms of WFD effect of the key indicators, USAID/Honduras and MESCLA decided to implement pilot IE on a small subset of the beneficiary population. The findings of the pilot IE were to be used to calculate the effect size and inform the decision of whether USAID/Honduras would invest in a full IE of the WFD Activity, The team devised the pilot IE as a Randomized Controlled Trial (RCT) with a phase-in design in three neighborhoods in or near Tegucigalpa2. A target sample size of 200 participants in the treatment group and 100 participants in the control group was established. These numbers were determined to be sufficient to calculate effect size for any future IE (as opposed to determination by power calculations) and considered an anticipated drop-out rate of 37.5 percent. MESCLA randomly assigned WFD beneficiaries who completed the WFD Activity’s “expression of interest” form, met minimum qualifications, and who returned for the administration of data collection as program intake, to treatment or control groups. Beneficiaries in the treatment groups were set to start the WFD training immediately, whereas those in the control groups would enter during a subsequent round of training approximately one year later. The team measured risk of violence involvement, employment, and employability in a baseline data collection in September and October of 2017. A subsequent data collection exercise, post Phase 13, took place in March of 2018. To measure employment and employability, MESCLA and NDIGD used tools developed by USAID/Honduras METAS Activity4; and to measure the risk of violence involvement NDIGD developed a Violence-Involved Persons Risk Assessment (VIP-RA) tool. The final sample size during the baseline was 170 beneficiaries assigned to the treatment group and 102 to the control group. Of those assigned to treatment, much less than the anticipated number, only 15 of 170, completed Phase 1. During post Phase I, MESCLA was able to collect data for 73 youth from the control group and 62 youth from the treatment group. Among those in the treatment group who were found and answered the questionnaires for Phase 1, only 15 completed the activities as expected (this pool of beneficiaries is referred to as “Complete Phase 1”). The remaining 47 individuals dropped out before finishing Phase 1 (this pool of beneficiaries is referred to as “Incomplete Phase 1”). The small number of Complete Phase 1 cases precludes reliable statistical comparisons between treatment and control groups and the changes between the baseline and post Phase I. Descriptive comparisons of the baseline and post Phase I data of those in the control, Complete Phase 1 and Incomplete Phase 1 show: 1 https://www.usaid.gov/honduras/cdcs 2 Villa Franca, Campo Cielo and Los Pinos 3 After beneficiaries completed Life Skills, Basic Labor Competencies and Cognitive Behavior Therapy 4 https://www.edc.org/proyecto-metas Monitoring & Evaluation Support for Collaborative Learning and Adapting (MESCLA) 4 • A larger percentage increase in beneficiaries studying in the Complete Phase 1 pool (23.2 percent change) as opposed to those in the Incomplete Phase 1 pool (17.6 percent change) and in the control group (12.0 percent change). • A larger increase in the percentage who were employed among those in Incomplete Phase 1 (7.5 percent change) pool, as compared to those in the control group (3.0 percent change) and in the Complete Phase 1 (0 percent change). • A small decrease in risk of violence involvement (-0.007) among participants with Incomplete Phase 1, and an increase among those in the control and Complete Phase 1 groups (0.073 and 0.075, respectively). It is important to note that the mean increase in the Complete Phase 1 group was largely driven by two outliers with very high scores at the end of Phase 1 –a result exacerbated by the small number of cases. These results may hint at positive desertion among those from Incomplete Phase 1 groups, but this hypothesis cannot be tested with the current data. The final sample sizes do not permit conclusions about the statistical significance of these changes and the reasons of desertion can only be inferred upon direct questioning. The implication of this study for the implementation of a full-scale IE of the WFD Activity is that, given current desertion rates and heterogeneity in program implementation, a full IE is not advisable in 2018. The Performance Evaluation (PE) planned for the second half of 2018 should inform of any changes in WFD, including those in retention rates and standardization of the activities. It is advisable to revisit the decision to conduct a full-scale IE in 2019, once the 2018 leg of the PE is finished. Further follow-up with all participants for the purposes of this pilot IE should cease, and the control group beneficiaries should be allowed to enroll in the WFD Activity. Monitoring & Evaluation Support for Collaborative Learning and Adapting (MESCLA) 5 2 INTRODUCTION 2.1 BACKGROUND Dexis Consulting Group (Dexis) has been contracted by USAID/Honduras to provide management and technical services for the implementation of its MESCLA activity aimed at supporting and strengthening monitoring and evaluation of USAID financed development assistance in Honduras. USAID/Honduras has outlined a CDCS that focuses on creating economic opportunities, reducing violence and insecurity, strengthening weak institutions, and reducing impunity. More specifically, USAID/Honduras DO1 aims to improve citizen security for vulnerable populations in high-density urban areas with notably high-crime rates. These are communities inside the municipalities of Tegucigalpa, San Pedro Sula, Choloma, Tela, and La Ceiba. Under DO1, USAID/Honduras programs provide services such as: • In-school and out-of-school educational and off-street social opportunities for at-risk children and youth • Community infrastructure that mitigates crime • School and classroom infrastructure to increase access to education beyond grade six • Social services to address household and gender-based violence • Community policing • Public financial management in the Government of Honduras (GOH) security and justice agencies USAID/Honduras engaged MESCLA to assess the possibility of conducting an IE of the WFD Activity. MESCLA subcontracted NDIGD to lead the exercise. Together with USAID, MESCLA agreed on conducting a pilot IE on a subset of the beneficiary population, with the objective to use its findings to inform whether USAID should invest in a full IE of the WFD Activity. 2.2 WORKFORCE DEVELOPMENT “EMPLEANDO FUTUROS” ACTIVITY Since 2016, Banyan Global implements the WFD Activity, named “Empleando Futuros”.5 The five-year activity runs from 2016 - 2021 and targets at-risk youth living in the five Honduran municipalities that are most affected by violence and crime and aims to increase citizen security for vulnerable youth (ages 16-30). The activity works to support workforce development that will increase income-generating opportunities and decreases the risk of youth being perpetrators of violence. Among other activities, it strengthens comprehensive workforce readiness services, including for job linkage and self-employment. (USAID, 2016) WFD Activity beneficiaries are drawn from the following population, including those who qualify for secondary and tertiary violence prevention services: • Males between the ages of 16 and 30; • Living in specifically identified communities within the following high-crime municipalities: Tegucigalpa, San Pedro Sula, Choloma, La Ceiba, or Tela; • Currently out-of-school, with no less than a sixth-grade education and no more than a high school degree; and • Currently unemployed, underemployed, or with inconsistent or low-skill employment. 5 https://banyanglobal.com/project/honduras-workforce-development-activity/ Monitoring & Evaluation Support for Collaborative Learning and Adapting (MESCLA) 6 The WFD Activity in Honduras is comprised of three main phases (see Figure 1): • Phase I (approximately ten weeks): focuses on life skills, cognitive behavioral therapy (CBT), and basic labor competencies • Phase 2 (up to nine months): focuses on professional training (job skills in a specific trade) • Phase 3 (approximately 3 days): focuses on job placement support and general observation and mentoring (approximately six months) All communities for the pilot IE share RETE as the community-level implementing partner responsible for recruitment, selection and mentoring of youth in year one in all the pilot communities. This was to ensure some amount of homogeneity in implementation (though variations existed in scheduling and program implementation even within RETE). Monitoring & Evaluation Support for Collaborative Learning and Adapting (MESCLA) 7 Figure 1: WFD Activity Phases and Data Collection Time Source: WFD Pilot IE: Baseline data collection for the treatment and control groups took place during the activity’s enrollment in September and October of 2017 Pilot IE: Post Phase 1 data collection for the treatment and control groups took place 5 months after the baseline on March of 2018 Post Phase 1 Baseline Phase 1 Phase 2 Phase 3 Monitoring & Evaluation Support for Collaborative Learning and Adapting (MESCLA) 8 3 EVALUATION PURPOSE AND EVALUATION QUESTIONS 3.1 EVALUATION PURPOSE The pilot IE aimed to answer the following question, “what is the potential effect of receiving a comprehensive WFD services on the risk levels related to violence in at risk youth”? To the extent possible the pilot attempted to address secondary research questions related to whether the WFD Activity’s effect on risk varies by employment status, socioeconomic variables, initial risk levels etc. The size of a sample needed for significance depends partially on effect size. If one expects the effect of an intervention to be a small change (in employability, employment, and violence risk factors in this case), a bigger sample size is needed in order to detect the effect. The literature on the link between WFD and violence shows mixed results among WFD projects that aim to impact violence. Several studies fail to demonstrate a link between employment and gang violence among youth in the United States (Millensky, Bloom, Muller-Ravett and Broadus, 2011; Uggen, 2000). Some suggest job training may increase violence risk in youth (Bloom, Orr, Bell, Cave, Doolittle, Lin and Bos, 1997), yet others demonstrate a positive impact but at a great investment (Schoschet, Burghardt and McConnell, 2008; Cave, Bos, Doolittle and Toussaint, 1993). In Honduras specifically, the METAS basic labor competencies activity (2010–20176) increased confidence among youth, but did not ultimately impact employment (USAID, 2014). Similarly, among young adults there have been mixed results in the United States (Raphael, 2011), but the team could not find any previous studies in that area among young adults in Honduras. The need to determine the “effect size” stems from the lack of studies on the effect of WFD activities on violence in Honduras. 3.2 EVALUATION QUESTIONS The questions that informed the pilot IE are as follows: 1. Do youth in the treatment group and the control group, selected on same basis and assigned randomly, show different post-program employability? 2. Do youth in the treatment group and the control group, selected on same basis and assigned randomly, show different post-program risk of violence? 2.a. Do the final treatment group’s risk levels vary by employment status and type of employment, referral status, sex, age, education, location, initial risk level, duration and composition of the intervention? 4 EVALUATION METHODS AND LIMITATIONS 4.1 DATA COLLECTION TOOL: VIOLENCE-INVOLVED PERSONS RISK ASSESSMENT NDIGD and faculty in the Notre Dame Department of Psychology developed and validated the VIP-RA tool for the WFD Activity in order to distinguish between primary risk individuals (persons who, despite living in areas with high levels of violence, exhibit few signs of violent behavior in their thoughts or actions, or have many protective assets7) and secondary risk individuals (persons that exhibit signs of violent behavior in either their thoughts or actions or do not have protective assets). The VIP-RA will be used in the impact and performance evaluations, and will also serve for the following purposes: (1) establish empirically-defensible risk of violence among young adults ages 16-30, (2) help target 6 https://www.edc.org/proyecto-metas 7 Have a supportive family or are part of a positive peer group Monitoring & Evaluation Support for Collaborative Learning and Adapting (MESCLA) 9 interventions for the secondary risk population, and (3) measure change in risk during and after program participation via an impact and/or performance evaluation (Hare, Guzman, and Miller-Graff, 2018). The VIP-RA seeks to determine which potential program beneficiaries are “the most at risk of being perpetrators of violence” as the mandated target population of the WFD activity. This allows the Activity to focus on the specific needs of high-risk individuals. The VIP-RA must reliably distinguish between “primary” and “secondary” risk among young adults. This categorization of risk corresponds to the USAID theory of change that, “providing a combination of primary (community-based infrastructure), secondary (focused on individuals at risk of becoming perpetrators), and tertiary (focused on those who have already committed crimes) prevention programs at the individual, family, relational, and community levels will lead to a reduction in violence” (USAID, 2015; p. 12). The VIP-RA was required to help the WFD activity tailor its recruitment and programmatic activities to better reach those at secondary risk; and to measure changes in risk levels pre-post participation in the Activity. The tool does not rely on static risks (age, ethnicity, local demographics, personal history) that could lead to social profiling. Rather, it focuses on many of the “dynamic characteristics, attitudes, circumstances, and behaviors” that are related to violence and that can be modified (Hennigan, Maxon, Sloane, Kolnick, and Vindel, 2014). The VIP-RA covers domains – personal history, emotional dimensions, relationship/community dimensions, and past deviance – similar to past tools used to determine primary/secondary risk. The tool places these domains in the social-ecological framework of violence prevention. The VIP-RA was validated using a methodology similar to that used in the validation of past risk assessment tools (Hennigan, et al., 2014). 4.2 DATA COLLECTION METHODS The pilot IE was an RCT with a phase-in design. For the Pilot IE, the team created a treatment group for the initial participants in the WFD training, and a control group for those that would enter during a subsequent round of training one year later. Randomization was performed at the individual level among all young adults who completed the WFD Activity “expression of interest” form, met minimum qualifications, and returned for the administration of the employability and VIP-RA tools as program intake. Three neighborhoods in or near Tegucigalpa were ultimately chosen for the pilot (Villafranca, Campo Cielo, and Los Pinos). Community selection was based on the experience of the implementing organization (RETE) working in the community, availability of facilitators and mentors, sufficient young adult population from which to draw the sample, and similar socio-demographic features. Given the exploratory nature of the pilot IE, a target sample size of 200 participants in the treatment groups and 100 participants in the control group was established. This number was determined to be sufficient to calculate effect size for any future IE (as opposed to determination by power calculations). In addition, according to the WFD Monitoring, Evaluation and Learning plan, the implementer anticipated a drop-out rate of 37.5 percent, and the MESCLA and NDIGD team also anticipated an attrition rate (people who cannot be re-contacted during the Pilot) of less than 10 percent. In situ random assignment was conducted in Villafranca and Los Pinos into treatment and control via a public lottery, and remote random assignment was conducted for Campo Cielo. Public lotteries were conducted to increase confidence in selection, where possible, and were attended by approximately one-third to half of those who completed intake in both locations. The pilot IE anticipated three waves of data collection: baseline, post Phase 1 and post Phase 2. Baseline data collection was performed in September and October 2017, by volunteers from RETE, under the direction of the MESCLA/NDIGD team. The survey was conducted using tablets that used the Monitoring & Evaluation Support for Collaborative Learning and Adapting (MESCLA) 10 Open Data Kit software8. As described in Table 1, the achieved sample size of the baseline was 272 individuals: 170 in the treatment group and 102 in the control group (see baseline report for details on this sample.) Table 1: Baseline Treatment and Control Observations by Community Community Treatment Control Total Villafranca 100 55 155 Campo Cielo 30 25 55 Los Pinos 40 22 62 Total 170 102 272 Source: MESCLA WFD Pilot IE Baseline Study Post Phase 1 data collection was completed in March 2018. Table 2 shows the achieved sample size of this data collection effort and how it relates to the number of observations during the baseline data collection. Table 2: Baseline and Post Phase 1 Treatment and Control Observations Exposure to intervention Baseline Post Phase 1 Control 102 73 Incomplete Phase 1 172 47 Complete Phase 1 15 Total 272 135 Source: MESCLA WFD Pilot IE Study Of the 102 baseline participants in the Control group, 73 (72 percent) were contacted and interviewed during data collection for post Phase 1. The percentage of participants lost to follow-up was much larger in the treatment group: Of the 172 participants, 47 (27 percent) dropped out before completion of Phase 1 and were reached for a follow-up interview (Incomplete Phase 1). Only 15 (9 percent) completed Phase 1 and were available for an interview (Complete Phase 1). The remaining 110 had dropped out but were not reachable for a follow-up interview. 4.3 ISSUES AND LIMITATIONS The sample size of the pilot IE was determined to obtain parameters for a sample size for a larger IE and not to produce statistically significant estimates of the impact of WFD on employability, employment and risk to violence. Its purpose was to determine the effect size and feasibility for a full IE. As with any IE, using an experimental design, it is important to note that this pilot IE has limited generalizability, or low external validity. The pilot IE was conducted on three communities and is not representative of the whole Activity. In addition, the limited sample size was reduced even further by desertion and attrition. Consequently, comparisons between the control and treatment groups, including those where the treatment are disaggregated by Complete Phase 1 and Incomplete Phase 1, should be taken with caution. 8 https://opendatakit.org/ Monitoring & Evaluation Support for Collaborative Learning and Adapting (MESCLA) 11 5 FINDINGS, CONCLUSIONS AND NEXT STEPS 5.1 FEASIBILITY OF CONDUCTING A FULL-SCALE IMPACT EVALUATION The objectives of the pilot included determining the feasibility for a large-scale IE. Post Phase 1 data collection confirmed that conducting a full-scale IE under current activity conditions would be difficult. The two primary challenges encountered during this data collection phase were (1) inconsistent implementation and (2) attrition. 1. When the activities of a program are inconsistently implemented it is difficult to estimate its true impact, as activities with different characteristics and potentially varying impacts are grouped and considered homogeneous. This is the case of the WFD Activity. The application of the complete CBT component, the element most related to risk reduction, varied between the communities in the pilot IE. For example, youth in Villafranca received group CBT only, while those in Campo Cielo received both individual and group CBT. 2. Drop-out was much larger than expected, particularly among those in the treatment group. Based on Banyan’s estimation, MESCLA anticipated a 37.5 percent dropout plus approximately 10% attrition from the study. It was deemed that the pilot IE would be of some value down to 50 observations in treatment and 50 in the control group (The rule of thumb for the minimum treatment sample size for average effect is 30). However, only 15 individuals completed Phase 1. 5.2 FINDINGS: DO YOUTH IN THE TREATMENT AND CONTROL GROUPS SHOW DIFFERENT POST-ACTIVITY EMPLOYABILITY? With the low count of beneficiaries that received CBT, there is no reliable statistical comparison between the control, Incomplete Phase 1, and Complete Phase 1 groups or differences between baseline and post Phase 1. Tables 3 to 5 are shown for illustrative purposes and no statistical conclusions should be made from them. Table 3 shows that the percentage of those that began studying during this period was similar among groups. The 17.6 percent of Incomplete Phase 1 who started studying could be positive desertion to continue studies (though the exact reason for desertion can only be determined by directly asking the individuals about their reasons for desertion). Table 3: Post Phase 1 Findings - Studying n Baseline End Phase 1 Change Control 73 14.9% 26.9% 12.0% Incomplete Phase 1 47 13.2% 30.8% 17.6% Complete Phase 1 15 7.6% 30.8% 23.2% Source: MESCLA WFD Pilot Study As expected, table 4 shows that among those that Complete Phase 1 no more were employed at post Phase 1 than at baseline, compared to 7.5 percent more among incomplete Phase 1, and 3 percent more of the control group. Table 4: Post Phase 1 Findings - Employment n Baseline Post Phase 1 Change Control 73 10.4% 13.4% 3.0% Incomplete Phase 1 47 7.9% 15.4% 7.5% Complete Phase 1 15 0% 0% 0% Source: MESCLA WFD Pilot Study Monitoring & Evaluation Support for Collaborative Learning and Adapting (MESCLA) 12 5.3 FINDINGS: DO YOUTH IN THE TREATMENT AND CONTROL GROUPS SHOW DIFFERENT POST-ACTIVITY RISK OF VIOLENCE? Table 5 shows that the average risk doubled for the control group, decreased slightly among the Incomplete Phase 1 respondents, and increased notably among those that Completed Phase 1. (Note: The small “n” means we cannot rely on this last change. See Annex to visualize changes by case for the Complete Phase 1 group, where two large increases in risk scores skew the results. Overall, approximately half of the scores increased, and half decreased in the Complete Phase 1 group.) Table 5: Post Phase 1 Findings – Risk Baseline Post Phase 1 Change n Mean Min Max Mean Min Max Mean Control 73 0.078 0.0002 0.6162 0.152 0.0002 0.9964 0.073 Incomplete Phase 1 47 0.141 0.0002 0.9544 0.134 0.0001 0.9758 -0.007 Complete Phase 1 15 0.104 0.0027 0.5972 0.179 0.0001 0.9541 0.075 Source: MESCLA WFD Pilot Study 6 RECOMMENDATIONS Given the limitations described above, the MESCLA team made the following recommendations: 1. Given current desertion rates and heterogeneity in program implementation, a full IE is not advisable in 2018; the Pilot IE should be discontinued, and former control group participants should be allowed to enroll. Very few youths among the treatment group completed Phase 1 of the training, even when counting for those that only completed partial (individual or group) CBT. As a result, the pilot IE would continue to suffer from lack of statistical significance. USAID/Honduras may wish to reconsider a fuller impact evaluation after the WFD Activity improves retention rates and standardizes its program implementation. 2. The planned PE should focus on reasons for high drop rates from the Activity and inform strategies to improve retention. There is immense learning possible through the planned PE, including further follow-ups (qualitative and quantitative) with those who dropped out of the program as well as those who remained. Monitoring & Evaluation Support for Collaborative Learning and Adapting (MESCLA) 13 7 REFERENCES Abdalla, M., Barth, A., Dunn, A., Holter, A., Ortega, A., & Tinta, P. (2013). Youth Employability Evaluation Tool Validation. Education Development Center and Proyecto METAS Honduras: Washington, DC. Bloom, H., Orr, L., Bell, S., Cave, G., Doolittle, F., Lin, W., Bos, J. (1997). The benefits and costs of JTPA Title II-A programs: Key findings from the National Job Training Partnership Act study. Journal of Human Resources 32, 549–576. Cave, G., Bos, H., Doolittle, F., & Toussaint, C. (1993). JOBSTART. Final Report on a Program for School Dropouts. Manpower Demonstration Research Corporation: New York. Hare, T., Guzman, J.C., & Miller-Graff, L. (2018). Identifying high-risk young adults for violence prevention: A validation of psychometric and social scales in Honduras. Journal of Crime and Justice 41 (2). DOI: 10.1080/0735648X.2018.1446184 Hennigan, K. H., Maxson, C. L., Sloane, D. C., Kolnick, K. A., & Vindel, F. (2014). Identifying high-risk youth for secondary gang prevention. Journal of Crime and Justice, 37(1), 104-128. Millensky, M., Bloom, D., Muller-Ravett, S., & Broadus, J. (2011). Staying on Course: Three-Year Results of the National Guard Youth ChalleNGe Evaluation. Manpower Demonstration Research Corporation: New York. Raphael, S. (2011). Controlling Crime: Strategies and Tradeoffs, P. J. Cook, J. Ludwig, J. McCrary, Eds. Chicago: University of Chicago Press. 521–565. Schochet, P., Burghardt, J., & McConnell S. (2008). Does Job Corps work? Impact findings from the National Job Corps Study. American Economist Review 98, 1864–1886. Uggen, C. (2000). Work as a turning point in the life course of criminals: A duration model of age, employment, and recidivism. American Sociological Review 65, 529–546. UNAH-IUDPAS (2016). Boletín Nacional Observatorio de la Violencia. Tegucigalpa. http://www.iudpas.org/pdf/Boletines/Nacional/NEd43EneSep2016.pdf UNODC (2016). UNODC Statistics. https://data.unodc.org. USAID Honduras. (2014). Employability Study: An Evaluation of METAS’ Basic Labor Competences training and Certification Program [Draft]. Washington: Education Development Center, Inc. USAID Honduras. (2015). Country Development Cooperation Strategy. USAID Honduras. (2016). Workforce Development Activity – Statement of Work. Monitoring & Evaluation Support for Collaborative Learning and Adapting (MESCLA) 14 8 ANNEX Case Study Baseline Study Phase 1 Study Change Work Baseline Work Phase 1 Work Change Risk Baseline Risk Case 1 0 0 0 0 0 0 0.5971754 0.4287095 -0.1684659 2 0 1 1 0 0 0 0.0058586 0.005015 -0.0008436 3* 0 0 0 0 0 0 0.3375514 0.9540969 0.6165455 4 0 0 0 0 0 0 0.0782218 0.014697 -0.0635248 5 0 0 0 0 0 0 0.0223736 0.0210818 -0.0012918 6 1 1 0 0 0 0 0.0026976 0.0055912 0.0028936 7 0 0 0 0 0 0 0.0603554 0.0065211 -0.0538343 8* 0 1 1 0 0 0 0.037109 0.3525435 0.3154345 9 0 . . 0 0.011793 . 10 1 . . 0 0.1369134 . 11 0 0 0 0 0 0 0.0058306 0.0582445 0.0524139 12 0 0 0 0 0 0 0.0040581 0.0001403 -0.0039178 13 0 0 0 0 0 0 0.0293971 0.0912197 0.0618226 14 0 0 0 0 0 0 0.009776 0.1513806 0.1416046 15 0 1 1 0 0 0 0.1580141 0.2436143 0.0856002 Notes: “*” denotes outlier Monitoring & Evaluation Support for Collaborative Learning and Adapting (MESCLA) 1 U.S. Agency for International Development/Honduras Avenida La Paz, P.O. Box 3453 Frente a la Embajada Americana Tegucigalpa, Honduras www.usaid.gov