Photo credit: WEARERESTLESS.WORDPRESS.COM MIDTERM EVALUATION REPORT PERFORMANCE EVALUATION OF THE AT-RISK YOUTH ACTIVITY April 2017 This publication was produced for review by the United States Agency for International Development/ Dominican Republic. It was prepared independently by Social Impact Inc. I. Photo credit: ii MIDTERM EVALUATION REPORT PERFORMANCE EVALUATION OF THE AT-RISK YOUTH ACTIVITY April 2017 This publication was produced for review by the United States Agency for International Development/ Dominican Republic. It was prepared independently by Social Impact Inc. USAID Dominican Republic AID-51-TO-16-00002; AID-517-I-12-00001 This publication was produced at the request of the United States Agency for International Development (USAID). It was prepared independently by Social Impact, Inc. The evaluation team was comprised of Team Leader, Olga Nazario, Evaluation Specialist, Basab Dasgupta, Local Evaluation Specialist, Julio Canario, Qualitative Research Analyst, Jacqueline Lyon of MCI, Senior Youth Development Specialist, Christy Olenik of MCI, Project Director, Danielle De Garcia, Project Manager/ Quantitative Research Analyst, Catherine Villada, and Project Assistant, Sierra Frischknecht. DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government iii TABLE OF CONTENTS EXECUTIVE SUMMARY..................................................................................................................viii I. BACKGROUND .................................................................................................................... 1 INTRODUCTION ....................................................................................................................................................... 1 PROJECT CONTEXT ................................................................................................................................................. 1 AT-RISK YOUTH ACTIVITY COMPONENTS AND EVOLUTION .............................................................. 2 EVALUATION PURPOSE AND AUDIENCE ........................................................................................................ 6 EVALUATION QUESTIONS ..................................................................................................................................... 6 II. METHODOLOGY ................................................................................................................. 8 EVALUATION DESIGN .............................................................................................................................................. 8 DATA COLLECTION ................................................................................................................................................. 9 DATA PREPARATION AND ANALYSIS ............................................................................................................. 12 LIMITATIONS ............................................................................................................................................................. 13 III. EVALUATION QUESTION 1: YOUTH OUTCOMES ................................................... 16 FINDINGS .................................................................................................................................................................... 16 CONCLUSIONS ........................................................................................................................................................ 23 IV. EVALUATION QUESTION 2: SUCCESS FACTORS .................................................... 25 FINDINGS .................................................................................................................................................................... 25 CONCLUSIONS ........................................................................................................................................................ 32 V. EVALUATION QUESTION 3: SUSTAINABILITY ......................................................... 34 FINDINGS .................................................................................................................................................................... 34 CONCLUSIONS ........................................................................................................................................................ 38 VI. EVALUATION QUESTION 4: CAPACITY BUILDING ................................................. 40 FINDINGS .................................................................................................................................................................... 40 CONCLUSIONS ........................................................................................................................................................ 44 VII. EVALUATION QUESTION 5: PERCEPTIONS .............................................................. 45 FINDINGS .................................................................................................................................................................... 45 CONCLUSIONS ........................................................................................................................................................ 50 VIII. RECOMMENDATIONS ...................................................................................................... 51 FOR USAID AND ENTRENA ................................................................................................................................. 51 FOR THE DOMINICAN NATIONAL AND LOCAL AUTHORITIES .......................................................... 52 THEMATIC RECOMMENDATIONS .................................................................................................................... 52 IX. DISSEMINATION AND UTILIZATION PLAN............................................................... 55 iv ANNEXES .................................................................................................................................... 57 ANNEX A. EVALUATION SCOPE OF WORK ............................................................................................................. 57 ANNEX B. SCHEDULE OF ACTIVITIES AND INTERVIEWS.......................................................................................... 64 ANNEX C. DATA COLLECTION INSTRUMENTS (ENGLISH) ................................................................................. 75 ANNEX D. DATA COLLECTION INSTRUMENTS (SPANISH) ............................................................................... 132 ANNEX E: QUALITATIVE ANALYSIS CODEBOOK.................................................................................................. 192 ANNEX F: DECOMPOSITION ANALYSES ................................................................................................................. 197 ANNEX G: TEAM COMPOSITION AND DISCLOSURE OF COI ............................................................................. 200 ANNEX H. DESK REVIEW OF DOCUMENTS REVIEWED......................................................................................... 207 ANNEX I. PRESENTATION FOR DISSEMINATION ................................................................................................... 210 TABLES AND FIGURES Figure 1: At-Risk Youth Activity Theory of Change Diagram ................................................................................ 2 Figure 2: Duarte Corridor ............................................................................................................................................... 6 Figure 3: Enrollment Rate by Age Group .................................................................................................................. 18 Figure 4: Respondent Use of Birth Control during Last Sexual Encounter ....................................................... 19 Figure 5: Youth involvement in crime and violence ................................................................................................ 21 Figure 6: Distribution of School Enrollment and Employment by Age ............................................................ 199 Figure 7: Distribution of Drinking and Sexual Habits by Age ............................................................................ 199 Table 1: At-Risk Youth Activity Results and Objectives…………………………………………………... 3 Table 2: Program Description………………………………………………………………………………4 Table 3: Evaluation Questions, Data Needs, Methods, and Respondent Groups………………………… 8 Table 4: Summary of KII and FGD respondents………………………………………………………….. 11 Table 5: Enrollment Rates………………………………………………………………………………… 16 Table 6: Profiles of sub-awardees interviewed during evaluation………………………………………... 40 Table 7: Fairlie Decomposition Results…………………………………………………………………. 198 v ACRONYMS ARY CBO CBSI CESDEM CSA CSO DR EpC ET FGD FOSIS FPUM GoDR IDDI IP JADOM KII MCI MEPyD PEPFAR Profamilia SI SOW STI USAID USD USG At-Risk Youth Community-Based Organization Caribbean Basin Security Initiative Centro de Estudios Sociales y Demográficos Alianza para el Servicio Comunitario Civil Society Organization Dominican Republic Espacios para Crecer Evaluation Team Focus Group Discussion Fondo de Solidaridad e Inversión Social de Chile Fundación Pediátrica por un Mañana Government of the Dominican Republic Instituto Dominicano de Desarrollo Implementing Partner Junior Achievement Dominican Republic Key Informant Interview Making Cents International Ministry of Economic Planning and Development President’s Emergency Plan for AIDS Relief Pro-Bienestar de la Familia Social Impact, Inc. Scope of Work Sexually Transmitted Infection United States Agency for International Development U.S. Dollar U.S. Government vi ACKNOWLEDGEMENTS The At-Risk Youth Activity Evaluation team would like to express our appreciation to USAID, Entrena SRL and the implementing partners for their support throughout this evaluation. Ms. Maricela Ramirez, USAID/Dominican Republic, provided consistent support, guidance, and comments to the team. Her contributions and acumen are much appreciated. We also appreciate the generous assistance received from Ipsos for logistics, collecting data, and supporting the Social Impact team throughout the evaluation process. We are grateful to all not only for the contributions to the evaluation, but for your work with and commitment to improving the situation of at-risk youth in the Dominican Republic. Finally, special thanks to the many youth who spoke with us and provided us with vital insights and answers to our many questions. Your candor, time, and trust in our team were invaluable, and we hope this report accurately reflects your experiences and recommendations. vii ABSTRACT The Dominican Republic is home to a large population of youth considered at risk of becoming perpetrators and victims of crime, violence, and other risky behaviors. The At-Risk Youth (ARY) Activity was designed to respond to the needs of at-risk youth with the long-term goal of implementing a sustainable, multi-disciplinary safety net for at-risk youth that also prevents crime. This mixed-method evaluation incorporated the use of interviews, focus groups, and a youth survey. The key findings and conclusions for each of the key research questions are: 1- Regarding youth outcomes, the evaluation found that, though the quantitative data were unable to definitively measure impacts, beneficiaries and stakeholders cited strong benefits from the program; 2- Regarding the key factors influencing success in achieving outcomes, the evaluation found that challenges with the complexity of implementation and community factors such as violence were the key limiters to success; 3- Regarding sustainability, the evaluation found that, despite the program’s positive reviews, a sustainable source of funding and management have yet to be identified; 4- Regarding sub-awardee capacity building, though some challenges were encountered, organizations benefited from their participation in the ARY Activity; 5- Regarding stakeholder perceptions of the program, the ARY Activity received very positive reviews from stakeholders. These findings and conclusions resulted in recommendations for USAID regarding project implementation and project monitoring systems, a recommendation for the GoDR to continue making investments to reduce violence in communities, and several thematic recommendations that would help improve the design and implementation of future programs like the ARY Activity. viii EXECUTIVE SUMMARY BACKGROUND The Dominican Republic is home to a large population of youth considered at risk of becoming perpetrators and victims of crime, violence, and other risky behaviors. Particular risk factors that present challenges for these youth include living in vulnerable neighborhoods and/or violent homes, low levels of schooling, early sexual debut, and a lack of skills for employment and for coping with everyday situations. USAID/Dominican Republic (USAID/DR) has responded to these challenges through the At-Risk Youth (ARY) Activity, designed to reduce the likelihood of youth becoming involved in crime, violence and other risky behaviors. At a cost of approximately USD $20 million, the project is funded by the Caribbean Basin Security Initiative (CBSI) and the President’s Emergency Plan for AIDS Relief (PEPFAR). This mid-term performance evaluation, conducted by Social Impact, Inc. (SI) and its subcontractors, Making Cents International (MCI) and Ipsos, seeks to determine the extent to which the ARY Activity is on track to achieve its stated goals, as well as to identify lessons learned for future youth interventions in the Dominican Republic and the region. Fieldwork for this evaluation was carried out in October 2016. AT-RISK YOUTH INITIATIVE ACTIVITIES AND EVOLUTION In July 2012, USAID/DR awarded a cooperative agreement to Entrena SRL, a private sector Dominican firm, to implement the multidisciplinary project for at-risk youth, in partnership with RTI International, Instituto Dominicano de Desarrollo (IDDI), and Asociación Dominicana Pro-Bienestar de la Familia (Profamilia). Project activities and services are provided by 23 sub-grantees across 15 types of interventions ranging from health and education to life skills and entrepreneurship, all under Entrena’s guidance and supervision. The five-year project will end in July 2017. However, several of the sub-awardees with contracts shorter than the full life of the project (18, 24, 36 months) had either completed activities, or were in the process of closing in September 2016, when the evaluation took place. As of December 2016, the ARY Activity has reached more than 130,000 at-risk youth, who were identified using a matrix of risk factors, throughout the northern Duarte Corridor, a geographical area spreading from Santo Domingo to Puerto Plata that includes nine provinces and more than 500 neighborhoods. The project has provided at-risk youth between 11-24 years of age health and education interventions, opportunities for youth to become productive citizens, obtain identity documentation for children and families, and youth advocacy and participation. USAID/DR identified three objectives for this evaluation: Improved understanding of what works best to help ensure successful ARY Activity programming1; assessment of the effectiveness of the ARY Activity on crime prevention; and, guidance on how to improve programming in the future, not only for the ARY Activity, but for all programming that falls under CBSI. EVALUATION QUESTIONS USAID/DR defined five primary evaluation questions to be addressed through this evaluation: 1. What is the likelihood that At-Risk Youth Activity youth beneficiaries will: return to and stay in school; reduce risky behaviors; obtain and maintain jobs; and, start their own business? 2. (a) Which major factors have contributed, so far, to the achievement (or failure to achieve) of the results through each one of the project interventions by the end of the third year of implementation? (b) And, to what extent have public-private partnerships contributed or not to 1 Due to delays in the evaluation, less time remains in the ARY Activity than originally envisioned to apply lessons learned. The team hopes that this evaluation will inform the remaining months of the intervention; and any time thereafter should programming continue further. ix the results achieved? (c) What are the essential components for the workforce development program under the At-Risk Youth Activity to be successful? 3. Are the services offered to youth by Entrena and sub-awardees sustainable, in terms of financial resources and institutional capacity built under the project and potential local uptake? 4. To what extent has the capacity building training offered by Entrena helped sub-awardees to improve their technical and financial management capabilities which could help them access other sources of funding once the At-Risk Youth Activity is over? 5. How is the At-Risk Youth Activity perceived and valued by the beneficiaries, the community and other related parties including pertinent national authorities? METHODOLOGY EVALUATION DESIGN AND DATA COLLECTION To answer the evaluation questions, the SI Evaluation Team (ET) carried out a mixed-methods performance evaluation that included both quantitative and qualitative methods. For the qualitative analyses, the ET conducted 58 interviews and 17 focus group discussions between October 13 and 28, 2016 with project stakeholders including sub-awardees, beneficiaries, and others. The ET conducted random sampling for project participants, while using purposive sampling for other respondent groups, with input from USAID. For the quantitative analyses, the team conducted 606 youth outcome surveys with randomly selected project participants between October 19 and November 21, 2016. These data were combined with intake data collected by the implementer to conduct descriptive, pre/post, and logit model regression analyses. Quantitative data analysis relied on Stata software and included an initial round of data cleaning, followed by full statistical analysis. The qualitative data analysis was completed using Dedoose software. A codebook was developed to ensure consistent coding, highlight common themes, and provide robust analysis. EVALUATION QUESTION 1: YOUTH OUTCOMES FINDINGS Across all four key project outcomes—returning to and staying in school, reducing risky behaviors, obtaining employment, and starting one’s own business—interviewees and focus group participants often cited strong benefits of the ARY Activity. Respondents recognized the limitations of the program in affecting larger dynamics in the communities, such as the employer hesitation to hire youth, and the lack of access to credit for starting one’s own businesses. Beyond the qualitative feedback, survey data conveyed a more nuanced picture of the findings. Though school enrollment rates for participants were higher than the national averages, both at intake and at follow-up, statistical analyses with baseline conditions were confounded by the effects of natural age progression, a problem that also affected analyses of sexual behavior and employment. Despite these challenges, however, the data were able to show some interesting trends. First, among those who are sexually active, the data found a significantly higher likelihood to be using an effective form of birth control than at intake. Second, with respect to alcohol use, though the data found that significantly fewer respondents were drinking at all, among those that were drinking, they were more likely to be drinking heavily. Third, in regards to criminal and violent behaviors, the results were mixed, with some behaviors (such as theft and violent attack) increasing since intake, while others decreased (such as getting in fights and injuring others). x CONCLUSIONS The available evidence generally suggests that the ARY Activity had a positive effect on youth returning to and staying in school, as well as on the use of contraception during sexual activity. For alcohol use, the evaluation finds that although the number of individuals consuming alcohol has decreased, the extent of drinking among those who do consume alcohol is rising. The evidence is more ambiguous or mixed regarding the program’s effect on involvement in criminal and violent behaviors, drug use, employment, and likelihood of starting one’s own business. EVALUATION QUESTION 2: SUCCESS FACTORS FINDINGS Interviews and focus groups revealed that although the ARY Activity was highly rated by all, there remained some concerns, primarily from local implementers, about the underlying structure of the program. Concerns voiced included that the ARY Activity was implemented across too many areas and too many implementing partners with varying levels of capacity, which led to inconsistencies in implementation. Local implementers requested additional support, particularly for implementing program components with which they had no prior experience. Additional challenges to implementation (and to the evaluation) were posed by difficulties in tracking outputs and outcomes using the program’s M&E and tracking system. Respondents also suggested extending the length of program interventions to align more with the Espacios para Crecer (EPC) component, which lasted ten months (other programs ranged widely – from just three sessions upwards – and several were in the 40-80 hour range, while others were not time bound). The EPC program was cited by the largest number of stakeholders as having a positive impact both on school retention, as well as imparting softer skills, like interpersonal relations and psychological well-being. Its longer duration (ten months) was also well regarded and linked with its positive effects. Other well￾regarded programs within the ARY Activity were the Advantages of Staying in School program, which utilized the “Quantum Learning2” methodology, and programs about sexual and psychological health, and job training and insertion programs. Though the health interventions were highly rated, difficulties were encountered recruiting sex workers for participation, which had been a targeted group for the activity. The entrepreneurship and documentation programs were seen as least effective. Though respondents indicated that the entrepreneurship program had a positive effect on motivation for starting a business, the lack of access to credit and continuing support services often prevented them from doing so. Though the documentation efforts exceeded their initial targets, respondents indicated that significant vulnerabilities still existed, particularly for those who had not yet been able to access proper documentation paperwork. In addition to the internal (project-specific) factors influencing success that are noted above, respondents also indicated that there are several external factors outside the program that were limiting the success of the ARY Activity interventions. Some respondents indicated that safety and security were deteriorating within communities and that crime and violence were becoming more of an issue, which produced a negative effect on many anticipated outcomes – from education to criminal activity and violence, and drug and alcohol use. School overcrowding was also cited as a constraint for academic achievement. Resistance 2 Alerta Joven 2015-2016 Annual Report explains that Quantum Learning (QL) methodology “is a recognized Methodology for its brain-based, holistic approach to learning.” The methodology is used by Alerta Joven “to turn potentially boring or ineffective teaching/learning experiences into exactly the opposite: an environment where teachers and students are engaged.” Entrena SRL, “Proyecto Alerta Joven: Annual Report 2015-2016”, p. 15. USAID and Entrena, “Proyecto Alerta Joven: “Comunicado de Prensa,” Santo Domingo, DR: October 11, 2016. Entrena SRL, “Proyecto Alerta Jóven: Annual Report 2015-2016,” p. 3. xi by employers to hire at-risk youth was cited as a concern for obtaining and maintaining employment. Accessing credit was also very difficult for most youth, which limited the ability to start their own business. CONCLUSIONS A variety of internal and external program factors affected project success. Internal factors affecting success included the use of implementers with an established record working in the local area, clear project methodologies, quality staff, and programs that maintained contact with youth. The constraints to success were typically external factors beyond the program’s control, such as access to financing for entrepreneurs, and the reported increased prevalence of violence within communities. EVALUATION QUESTION 3: SUSTAINABILITY FINDINGS The ARY Activity’s interventions were intended to continue beyond USAID funding, particularly through the support of the Government of the Dominican Republic (GoDR) and/or the private sector. To date, however, program sustainability presents a real challenge for the continuation of the ARY Activity. Sub￾awardees now have the institutional capacity, experience, and willingness to continue offering at-risk youth activities and services, but are constrained by lack of financial resources. On the positive side, sub￾awardees were asked to develop sustainability plans, and a contract was signed for the sustainability strategy. However, sub-awardees stated that the sustainability plans that they were asked to develop arrived too late in the process (in year three), when several of them were already closing down. The key to sustainability would be financing through public-private partnerships, and yet, the GoDR has yet to make a clear commitment and assign a budget to the services for at-risk youth, and the support of the private sector – while significant – is not enough to carry on the project after USAID’s support ends. Entrena indicated that one of the project’s weaknesses was a lack of engagement with the private sector, particularly in partnering with large business associations from the start. CONCLUSIONS Not infrequently, experiences with internationally funded projects suggest that, despite their success, interventions are not continued by national authorities, nor do local organizations have the capacity or finances to sustain them once funding ends. This has been true in the DR in the past, and it would seem unrealistic to think that the ARY Activity would be the exception, particularly given the late start to the process of developing a sustainability strategy, and its current failure to find a committed source of funding, external to USAID, to continue the programs. EVALUATION QUESTION 4: CAPACITY BUILDING FINDINGS All representatives from sub-awardee organizations that were interviewed positively reviewed the ARY Activity and the support and training provided by Entrena. Entrena staff recognize that project implementation requirements were a challenge for most organizations, but still believe these requirements strengthened the organizations, particularly regarding service delivery. Considering the complexity of this multidisciplinary project and the delays encountered at start-up, activities such as grant writing training, building absorptive capacity, and developing links with the private sector could not receive the attention or effort they deserved, and were perhaps initiated too late into the intervention schedule. xii CONCLUSIONS All At-Risk Youth Activity sub-awardees benefitted from the experience by acquiring new technical and administrative skills. Their institutional capacity was strengthened, allowing them to operate more strategically and efficiently in terms of time and resources. EVALUATION QUESTION 5: PERCEPTIONS FINDINGS Respondents expressed very favorable opinions of the ARY Activity. Nearly 90% of survey participants found the programs in which they participated to be very useful, and, from interviews and focus groups, youth highlighted that the ARY Activity helped them increase their self-esteem and feel more confident in their ability to move forward and stay out of trouble. Particularly significant was the perception among youth in focus groups that they now had hope for their future, having acquired skills that would help them get ahead. Dominican government officials who were interviewed perceived the concept and results of the ARY Activity very positively. INFOTEP, which has provided vocational and technical training to the at-risk youth, perceived the project as a game-changer for vulnerable youth, offering them skills that could lead to better opportunities in life. The Ministry of Economic Planning and Development (MEPyD), Ministry of Youth, as well as the Office of the Attorney General, have been supportive of the project, and attempted to learn from its experiences to implement their own initiatives. CONCLUSIONS The ARY Activity, which built on existing programs and efforts of the GoDR, is a relevant project that is positively viewed across stakeholders, responds to the needs of Dominican at-risk youth, and fills a void left by a lack of government services, which also made significant advances in reaching vulnerable youth during this time. RECOMMENDATIONS FOR USAID AND ENTRENA ▪ CONSIDER A MORE FOCUSED GEOGRAPHICAL AREA FOR THE AT-RISK YOUTH ACTIVITY PROGRAM. Evaluation results suggest that the wide coverage area and the large number of sub-awardees (of varying levels of capacity) introduced challenges in coordination and consistency in implementation, which could be alleviated through improved targeting of scope and partners. ▪ IMPROVE THE MONITORING SYSTEM. The evaluation had difficulties using the monitoring and tracking system information, which USAID could improve in future iterations of the program. FOR THE DOMINICAN NATIONAL AND LOCAL AUTHORITIES ▪ CONTINUE TO MAKE IMPROVEMENTS, INCLUDING INFRASTRUCTURE, TO REDUCE VIOLENCE IN COMMUNITIES. Continuing problems of violence and criminal activity were voiced by many respondents, yet are largely beyond the purview of the project. Thus, it is recommended that the government take action to help reduce violence in communities. xiii THEMATIC RECOMMENDATIONS ▪ PUBLIC POLICY Continue to push for legal changes. The At-Risk Youth Activity was unable to break through congressional impasse to advance the practice of youth protection policy, but it is recommended that this effort continue so that the Dominican Government can be held responsible for at-risk youth. ▪ EDUCATION Expand on the gains made in at-risk youth education, and continue the training and incorporation of EpC methods in schools. Continue supporting family engagement in children’s education, and increase support for youth extracurricular activities. ▪ HEALTH Continue the ARY Activity interventions and expand activities to include sex workers, who could particularly benefit from the health interventions. The health interventions were well received, and there is some evidence that they have produced positive results. However, though some attempts by sub-awardees were made to reach sex workers, more work is needed. ▪ EMPLOYMENT OPPORTUNITIES Include higher-level training and additional internship opportunities. Though the interventions were thought to have increased work opportunities, recommendations for improvement include providing training in higher-level skills, such as management and technical areas, to increase job prospects. It is also recommended that opportunities for more internships, particularly paid internships, be explored. ▪ ENTREPRENEURSHIP Expand funding opportunities and create networks. The key constraints facing potential entrepreneurs was access to funding and access to continued support services (training, capacity building, etc.). Thus, future programming should provide added attention to helping participants to access credit and assisting them to access ongoing support services. ▪ SUSTAINABILITY Planning for sustainability must start at the beginning. Currently, the ARY Activity programs are unlikely to continue after the end of USAID funding. The ARY Activity defined its model for long-term sustainability as a co-financing mechanism with financial support from the government and private sector but with operational autonomy. The model continues to be valid but substantial efforts made by Entrena have not yet yielded expected results. To better assure sustainability, establish a clear plan, with appropriate funding commitments from all parties, from the very beginning of the project. 1 I. BACKGROUND INTRODUCTION This report presents the findings, conclusions, and recommendations of the mid-term performance evaluation of the USAID Dominican Republic (USAID/DR) At-Risk Youth (ARY) Activity. The fieldwork was conducted by Social Impact, Inc. (SI) and its subcontractors, Making Cents International (MCI) and Ipsos, in October 2016. The youth survey data collection was completed in November 2016. The task order (No. AID-517-TO-16-00002) was awarded through the Monitoring and Evaluation Services for the Caribbean Basin Security Initiative Indefinite Quantity Contract, and the contract Scope of Work (SOW) can be found in Annex A. The purpose of the performance evaluation is to inform and help the USAID/Dominican Republic Mission determine which intended results have been met and, specifically, the effectiveness of implementation, lessons learned, and best practices. Because the evaluation timeline falls late in the last year of programming, USAID/DR seeks action-oriented recommendations to improve future programming or similar programming targeting at-risk and vulnerable youth populations. PROJECT CONTEXT The Dominican Republic (DR) has an estimated population of 10.6 million people, more than 40 percent of which live in poverty, despite the country’s middle-income economic status.3 Though the DR showed the fastest economic growth in Latin America during 2014-15, income inequality remains high. Poor-quality education and health systems, inefficiencies in the energy sector, and increasing violence pose considerable challenges to sustaining growth over the long term and reducing poverty.4 The population of the DR is rather young. Approximately 20% is between the ages of 10 and 19, and over 1/3 is between the ages of 10 and 24.5 Many low-income youth live in vulnerable and challenging environments. According to the Encuesta Demográfica y de Salud (ENDESA) 2013 survey, only 48% of minors (under 15 years) live with both parents, and 40% of households are headed by a woman, increasing from 35% in 2007. While student drop-out rates declined for the majority of primary school students (2nd￾6th grades) from between 4-5% in 2007 to 1-3% in 2013, drop-out rates increased during this time period from 9% to 11% for 8th grade students. Additionally, while male students had higher desertion rates than women in 2nd-7th grades, women’s desertion rates jumped from 3.5% among 7th graders to 12.8% among 8th graders, higher than the rate for men. In Santo Domingo District, school desertion for 8th graders reached 14.5% overall in 2013, and was as high as 21% among the poorest students nationwide.6 One of the primary causes of this desertion is that 8th grade is the highest grade level a student is able to attend school without proper documentation. After that point, students are able to continue only informally and receive no credit, which is a major disincentive to attendance.7 While only 4% of the total population lacks a birth certificate, this affects 12% of children 5-9 years old and 7% of 10-14 year-olds. Among the poorest segments of the population, 15% lack birth certificates.8 Birth certificates are required to obtain identity cards, which are a prerequisite for obtaining formal employment. In 2013, 27% of youth 3 CIA World Factbook, “Dominican Republic,” January 12, 2017: https://www.cia.gov/library/publications/the-world￾factbook/geos/dr.html 4 US Department of State, “US Relations with the Dominican Republic,” Bureau of Western Hemispheric Affairs, Fact Sheet, June 30, 2016: http://www.state.gov/r/pa/ei/bgn/35639.htm 5 Ministerio de Salud Pública, “Encuesta Demográfica y de Salud: Republica Dominicana,” Centro de Estudios Sociales y Demográficos (CESDEM), Santo Domingo, Dominican Republic and ICF International, Rockville, MD, October 2014; (countryoffice.UNEPA.org/dominicanrepublic/drive/drdhs2013-final02-10-2013.pdf ) 6 Ministerio de Salúd Pública, op. cit. 7 Entrena, “Alerta Joven Annual Report 2014-2015.” pg 13. 8 Ministerio de Salúd Pública, op. cit. 2 aged 18-19 years and 11% of 20-24 year-olds did not have an identification card,9 forcing them to rely on informal work opportunities. Early pregnancy also creates a challenge for youth development in the DR. The median age at which youth become sexually active is between 17-1810 and around seventeen percent (16.7%) of women between the ages of 15-19 have at least one child.11 According to 2007 data in the 2012 DHS Comparative Reports, 14% of women and 21.1% of men in the Dominican Republic made their sexual debut before the age of 15, indicating an increase across the sexes (2002 data showed 12.8% for women and 18.4% for men).12 Pregnancy and maternity, according to the ENDESA Report, are directly related to poverty and low levels of education. In fact, the risk of an adolescent woman getting pregnant is six times higher for women with only elementary schooling than women with higher levels of education.13 The obstacles facing at-risk youth in the DR represent a challenge for the country’s development. Residing in vulnerable neighborhoods, low levels of education, early engagement in sexual activities, exposure to violence, and a lack of marketable skills for employment all result in youth less likely to be economically and socially productive, and more likely to be harmed by crime and violence, either as victims or participants. Encouraging youth productivity is critical in the Dominican Republic, where high unemployment (15% overall and 31.4% among youth ages 15-24 in 2014) is a contributing factor to criminal activities, including homicides, drug trafficking, and kidnapping.14 AT-RISK YOUTH ACTIVITY COMPONENTS AND EVOLUTION The At-Risk Youth Activity is better known as Alerta Jóven (Alert Youth), the Spanish name selected from a youth competition, which is derived from a message to Dominican youth to remain alert and not give in to the pressures and temptations of crime and violence in their communities. The goal of the ARY Activity is to create a sustainable, multidisciplinary safety net for at-risk youth with the intended result of preventing crime. Table 1 shows the intermediate results and objectives that support the ARY Activity’s steps toward this ultimate goal. 9 Ministerio de Salúd Pública, op. cit. 10 Centro de Estudios Sociales y Demográficos (CESDEM) (2014). Encuesta Demográfica y de Salud República Dominicana 2013. 11 Ministerio de Salúd Pública, op. cit. Pg. 93 12 USAID, DHS Comparative Reports 29, http://dhsprogram.com/pubs/pdf/CR29/CR29.pdf (2012) 13 Ministerio de Salúd Pública, op. cit. 14 World Bank, “World Development Indicators,” retrieved December 21, 2016: http://databank.worldbank.org/data/reports.aspx?source=world-development-indicators Figure 1: At-Risk Youth Activity Theory of Change Diagram 3 Table 1: At-Risk Youth Activity Results and Objectives15 FINAL RESULT IMPLEMENT A SUSTAINABLE, MULTI-DISCIPLINARY SAFETY NET FOR AT-RISK YOUTH THAT ALSO PREVENTS CRIME Intermediate Result 1 IR.1 Youth protection policy is improved in practice Obj. 1.1 Youth stakeholders empowered to influence policy decisions Obj. 1.2 Implementing agencies understand and comply with youth legal networks Obj. 1.3 Documented youth and children increased Intermediate Result 2 IR.2 Youth opportunities to be productive citizens improved Obj. 2.1 Formal and alternative education activities increased Obj. 2.2 Access to quality health prevention and health care services improved Obj. 2.3 Employment for youth increased Intermediate Result 3 IR.3 Integrated youth system developed Obj. 3.1 A comprehensive information system implemented Obj. 3.2 Multidisciplinary working groups coordinate and implement innovative solutions to reduce crime and gang activities Obj. 3.3 Public-private partnerships enhanced to support and advocate for vulnerable children and youth The ARY Activity is a five-year, approximately USD $20 million project, funded by the Caribbean Basin Security Initiative (CBSI) and the President’s Emergency Plan for AIDS Relief (PEPFAR) and overseen by USAID/DR. The project has been implemented since July 2012 by Entrena SRL, a private sector Dominican firm, in partnership with RTI International, Instituto Dominicano de Desarrollo (IDDI), and Asociación Dominicana Pro-Bienestar de la Familia (Profamilia). The award, given through cooperative agreement (AID-517-A-12-00007), will end on July 4, 2017. Sub-grants managed by Entrena were awarded to 23 civil society organizations for the implementation of the initiatives at the community level. As of December 2016, the project reported reaching more than 130,000 at-risk youth throughout the Duarte Corridor, spreading from Santo Domingo to Puerto Plata, covering 9 provinces and 500 neighborhoods.16 The ARY activity is a complex program with many different types of interventions; for ease of reference, Table 2 below presents an outline of all 15 types of programs included in the ARY Activity, the ages of youth involved in programming, when the program began, the implementing partners who conducted them, and the intended outcomes of the programs. 15 Entrena, “Annual Work Plan for Year 1: July 5, 2012 – July 31, 2013.” USAID 16 Proyecto Alerta Joven, “Alerta Joven En Numeros,” Alerta Joven: http://alertajoven.com/es/impacto/numeros/numeros 4 Table 2: Program Description PROGRAM TYPE AGES OF YOUTH INVOLVED WHEN PROGRAM BEGAN LENGTH OF PROGRAM IPS THAT IMPLEMENTED IT INTENDED OUTCOMES Entrepreneurship 15-24 7-2013 CEFE: 80 hours La Compania: 40 hours IDDI, Sur Futuro, ACOPRO, CSA, Children International, Caminante, MOSTCHA, JADOM, Project Hope, FPUM, FURJUG. Youth trained and certified to develop micro- business skills Vo-Ed Training 15-24 4-2013 Infotep: 80 hours minimum. Courses vary up to 240 hours. IDDI, Profamilia, Sur Futuro, CSA ACOPRO, Children International, Caminante, MOSTCHA, JADOM, Project Hope, FPUM and FURJUG Youth trained and certified to access jobs according to market demand. Life Skills 11-24 6-2013 40 hours IDDI,Profamilia, Sur Futuro, ACOPRO, CSA, Children International, Caminante, MOSTCHA, Project Hope, FPUM, and FURJUG. Youth trained to complement vocational and educational training in soft skills. School Insertion 11-24 9-2013 Depends on specific case and is not time bound. IDDI, Dream, CSA, Sur Futuro, ACOPRO, Children International, Caminante, MOSTCHA, Project Hope and FPUM. Youth identified and reinserted back in the formal school system through Minerd. School Retention 11-17 9-2013 LVPE: 25 hours QL: 72 hours AL: 48 hours IDDI, CSA, Sur Futuro, ACOPRO, Children International, Caminante, MOSTCHA and Project Hope. Youth identified, trained and retained in the formal school system reducing desertion risk. Public Policy Advocacy 11-24 9-2013 Initial Training: 40 hours with ongoing implementation through the youth networks formed in each organizations. IDDI, CSA, Sur Futuro, ACOPRO, Children International, Caminante, MOSTCHA, Project Hope and FURJUG. Implementing partners and youth trained to understand and put in practice youth legal framework. HIV Prevention 11-24 1-2013 7 Sessions Profamilia, IDDI, CSA, Sur Futuro, ACOPRO, Children International, Caminante, MOSTCHA, Project Hope, and FURJUG. Youth trained that complete PEPFAR program, increase the use of modern contraceptives. Reach juvenile most at-risk population. 5 HIV Testing 18-24 7-2013 3 Sessions (Pre/Test/Post) Profamiia, IDDI, CSA, Sur Futuro, ACOPRO, Children International, Caminante, MOSTCHA, Project Hope and FPUM. Promotion, application and result sharing with sexually active youth. Health Services 2-2013 Depends on specific case and is not time bound. Profamiia, IDDI, CSA, ACOPRO, Children International, Project Hope and FPUM. Youth receive youth friendly health services for at-risk youth. Psychological Therapy 11-24 1-2013 Depends on specific case and is not time bound. Profamilia, IDDI, Children International, FPUM, ACOPRO and Caminante. Youth receive specialized counseling on behavioral change towards violence, (including GBV and domestic violence), Also addresses specific learning disabilities. Crime/ Violence Reduction 11-24 4-2013 IDDI, Profamilia, Sur Futuro, ACOPRO, CSA, Children International, FURJUG and MOSTCHA. Youth trained through activities that promote community based solutions to address conflict, crime and violence reduction. Legal Documentation 11-24 10-2013 Depends on specific case and is not time bound. CSA, Sur Futuro, ACOPRO, Children International, Caminante, MOSTCHA and Project Hope. Youth without documentation identified and followed up to obtain documentation. Espacios para Crecer 11-14 5-2013 450 hours IDDI, CSA, Sur Futuro, ACOPRO, Children International, Caminante, MOSTCHA and Project Hope. Youth at risk of deserting school identified and enrolled in parallel program to promote and increase academic achievement and personal growth. Microcredit 18-24 7-2013 80 hours IDDI, CSA, Sur Futuro, ACOPRO, Children International, Caminante, MOSTCHA, JADOM, Project Hope, FPUM and FURJUG. Youth trained to manage their business, including monetary funds. Youth Mediation 14-22 3-2013 Initial Training: 40 hours. Program implementation is ongoing through peer to peer networks. IDDI, CSA and Children International Youth trained to become part of a community based peer to peer network to manage conflict and promote culture of peace. 6 The ARY Activity has provided interventions in school retention, insertion, and completion, technical-vocational and entrepreneurship skills training, HIV/AIDS prevention and attention services, identity documentation for youth (both for those who are Haitian immigrants themselves and those who are children of Haitians in the DR), and youth policy and civic participation. The ARY Activity focuses on vulnerable young people ages 10-24 who are at risk of leaving (or have left) school, committing criminal offenses, or becoming involved in gangs. It also targets teenage mothers, HIV-positive youth, and commercial sex workers. Youth ages 11-17 were prioritized for education and health interventions, while activities for youth ages 18-24 focused on workforce development. Sustainability was a major focus of the program’s original design, which called for co-financing from donors apart from USAID, the private sector, and other groups. The project aimed to implement activities that improved the ability of youth service providers to carry on operations after the end of the program, particularly through the ongoing participation of the GoDR. However, as discussed in section V, success in achieving sustainability has varied widely, in part due to difficulty coordinating across IPs and GoDR entities. Entrena indicated that while sustainability was a focus of the project as a whole, not every organization they worked with made this a priority. Entrena added that while the project itself emphasized sustainability of activities, the true sustainability of each local partner can only be determined as the project is implemented and evolves. EVALUATION PURPOSE AND AUDIENCE In May 2016, USAID/DR contracted Social Impact to carry out a mid-term performance evaluation of the initiative. USAID/DR identified three objectives for this evaluation: 1. Improved understanding of what works best to help ensure successful ARY Activity programming; 2. An assessment of the effectiveness of the ARY Activity on crime prevention; and, 3. Guidance on how to improve programming in the future, not only for the ARY Activity, but for all programming that falls under CBSI. The purpose of achieving these objectives is for the Youth Programming Office at USAID/Dominican Republic (USAID/DR) and Entrena to use the evidence to improve their programming for the final year of the project, and to consider changes that can be incorporated into future youth-oriented projects. EVALUATION QUESTIONS To meet their overarching objectives, USAID/DR defined five primary evaluation questions to be addressed through this evaluation:  EVALUATION QUESTION ONE (EQ1) What is the likelihood that the ARY Activity youth beneficiaries will: return to and stay in school; reduce risky behaviors; obtain and maintain jobs; and, start their own business? Figure 2: Duarte Corridor 7  EVALUATION QUESTION TWO (EQ2) (a) Which major factors have contributed, so far, to the achievement (or failure to achieve) of the results through each one of the project interventions by the end of the third year of implementation? (b) And, to what extent have public-private partnerships contributed or not to the results achieved? (c) What are the essential components for the workforce development program under the ARY Activity to be successful?  EVALUATION QUESTION THREE (EQ3) Are the services offered to youth by Entrena and sub- awardees sustainable, in terms of financial resources and institutional capacity built under the project and potential local uptake?  EVALUATION QUESTION FOUR (EQ4) To what extent has the capacity building training offered by Entrena helped sub-awardees to improve their technical and financial management capabilities which could help them access other sources of funding once the ARY Activity is over?  EVALUATION QUESTION FIVE (EQ5) How is the ARY Activity perceived and valued by the beneficiaries, the community and other related parties including pertinent national authorities? 8 II. METHODOLOGY EVALUATION DESIGN During the Best and Final Offer stage of the proposal for this evaluation, USAID/DR worked with the ET to modify the overall scope to center the study around the activities taking place in Santo Domingo, as nearly half of all beneficiaries are located in Santo Domingo and the surrounding areas. Due to this modification and focus on Santo Domingo, findings are not generalizable to areas outside of this geographic scope, as the team recognizes that there are likely significant differences between the rural and urban contexts and targeted populations. To answer the evaluation questions, the SI ET (ET) carried out a mixed￾methods performance evaluation that brought together both a quantitative survey with youth program participants as well as key informant interviews (KIIs) and focus group discussions (FGDs) that centered on the twelve activities under the ARY Activity in the Santo Domingo area. In Table 3 below, we summarize the evaluation questions and the specific methodology the ET implemented to answer them. Table 3: Evaluation Questions, Data Needs, Methods, and Respondent Groups EVALUATION QUESTION DATA NEEDS METHODS RESPONDENT GROUPS EQ1: What is the likelihood that the ARY Activity youth beneficiaries will: return to and stay in school; reduce risky behaviors; obtain and maintain jobs; and, start their own business? Outcome measures related to education persistence and attainment, sexual behavior, drug and alcohol use, employment, and entrepreneurship Youth Outcomes Survey Perceptions data gathered through KIIs and FGDs regarding successes and challenges of youth attaining desired outcomes Two groups of youth: 11- 17 and 18-24 Entrena, sub-awardees, school representatives, and employers EQ2: (a) Which major factors have contributed, so far, to the achievement (or failure to achieve) of the results through each one of the project interventions by the end of the third year of implementation? Qualitative information on the successes and challenges encountered by partners Quantitative outcome data on youth to determine correlations Youth Outcomes Survey Perceptions data gathered through KIIs and FGDs regarding successes and challenges of implementation Two groups of youth: 11- 17 and 18-24 Entrena and sub￾awardees EQ2: (b) And, to what extent have public-private partnerships contributed or not to the results achieved? Qualitative information on successes and challenges of public￾private partnerships Perceptions data gathered through KIIs and FGDs regarding successes and challenges of public and private partnerships Entrena, sub-awardees; public and private partners including GoDR and employers EQ2: (c) What are the essential components for the workforce development program under the ARY Activity to be successful? Qualitative information on successes and challenges of workforce development program, key components for success, and most useful elements of the program Perceptions data gathered through KIIs and FGDs regarding successes, challenges, and most useful aspects of workforce development program Entrena, sub-awardees implementing the workforce development program, and employers Youth participants of workforce development program 9 EQ3: Are the services offered to youth by Entrena and sub- awardees sustainable, in terms of financial resources and institutional capacity built under the project and potential local uptake? Qualitative information on sustainability of programming and measure put in place to ensure sustainability Qualitative interviews with IP and subs about sustainability related to financial resources and technical and management capacity Entrena, sub-awardees, other key stakeholders, such as private partners and GoDR EQ4: To what extent has the capacity building training offered by Entrena helped sub-awardees to improve their technical and financial management capabilities which could help them access other sources of funding once Alerta Jóven is over? Scoring of financial resources, institutional capacity, and local uptake Qualitative information regarding pursuit of additional funds Mini survey to take quantitative measures of capacity Qualitative data gathered through KIIs with IP and subs on capacity and activities undertaken on capacity building. Mini survey to a diverse group of sub-awardees Qualitative information from Entrena, sub￾awardees, and counterparts regarding capacity EQ5: How is the ARY Activity perceived and valued by the beneficiaries, the community and other related parties including pertinent national authorities? Quant perceptions score for youth Qualitative perceptions of value by community and other parties Youth Outcomes Survey Qualitative data gathered through KIIs and FGDs with youth, principals, community leaders, and GoDR. Two groups of youth: 11- 17 and 18-24 FGD-Entrena, diverse range of sub-awardees, reps from GoDR, employers, school reps, and other relevant community partners. DATA COLLECTION Three sets of data were used in this evaluation: 1) The ARY Activity’s Intake Form, which was used as a soft baseline measure; 2) a quantitative Youth Outcomes survey; and, 3) qualitative interviews and group discussions. The intake form captured youth’s demographic information, as well as their responses to questions on educational and work experience, personal security, participation in the justice system, drug and alcohol use, and sexual activity. Some of the sections in the intake form were adapted and expanded upon in the Youth Outcomes survey, allowing the ET to form a pre-post comparison for questions that were relevant to the evaluation. The team had to use the exact intake wording on the outcomes survey to draw accurate pre-post comparisons, even though many of the questions were worded in a way which the team would consider to be too direct to elicit fully unbiased data. Similarly, the information included in the intake forms did not cover all variables needed to answer the evaluation questions. As such, the ET also drafted recall questions in the Youth Outcomes survey that detailed youth’s behaviors and perceptions at the beginning of the program and at present to allow for further pre-post comparisons. Data collection was completed in two segments, with the ET conducting 75 qualitative interviews and group discussions between October 14 and 31, 2016, and the Youth Outcomes survey conducted with 606 youth between October 19 and November 21, 2016. 10 YOUTH OUTCOMES SURVEY The survey was administered one-on-one via tablet to selected youth by enumerators from the team’s data collection firm, Ipsos. Prior to administering the survey, enumerators underwent a three-day training overseen by SI, during which they became familiar with the survey, relevant protocols, and use of tablets. Enumerators began survey administration by reading a consent script verbatim to a participant, ensuring that the participant clearly consented to participation. For youth under the age of 18, enumerators first made contact with the parent / guardian / in loco parentis to gain parental consent for the youth to be interviewed, and then proceeded to gaining youth assent to be interviewed. A verbal or visual consent from the participant was logged as “participant consented” in the enumerator’s tablet. As an enumerator moved through the survey, programmed skip logic ensured that only relevant questions were posed to participants. For sections of the survey that addressed sensitive topics such as sexual, drug, and violence related behavior, the enumerator was prompted to pass the tablet to the youth to answer the questions, which offered an additional level of confidentiality to the respondent. Enumerators uploaded data from all surveys daily. SI and the data collection firm had access to uploaded survey data via a secure, cloud-based SurveyCTO server. All final data submitted was anonymized, with all potentially identifying information digitally removed from the data. While the original sample size intended for the youth outcomes survey was 800, various challenges in contacting youth from the sample frame of 1600 led to a slight delay in quantitative data collection and a reduction in the target sample to 600. This reduction, however, still allowed for disaggregation by age. Challenges encountered by Ipsos and the sub awardees included 1) incorrect tracking information for the selected youth 2) the closure of the majority of programs before data collection commenced 3) difficulty in tracking older youth through the schools and 4) a delayed start to using a door to door method for contacting youth. KEY INFORMANT INTERVIEWS AND SUB-AWARDEE SURVEY The team conducted key informant interviews (KIIs) with implementing partners and sub-partners, as well as other community stakeholders, including relevant community-based organizations (CBOs) and civil society organizations (CSOs), school representatives, and employers on October 14-24, 27, and 30, 2016 in Santo Domingo and Boca Chica. (Please see Annex B for the complete schedule of data collection activities.) For implementing partners and sub-partners, questions focused on the implementation of the ARY Activity and its integrated nature, the successes and challenges encountered, the level of collaboration among partners, the role of public private partnerships, and sustainability of the program. Additionally, sub-awardees were asked to complete a brief survey that examined their financial and management capacity. This was used both as an assessment tool, and, in the future, could serve as a baseline measure to see how the partners self-assess progress in this area. Additionally, sub-awardees were asked to comment on the value of the capacity building training they have received. Employers were also asked about workforce development and skills priorities for youth. All key informant interviews lasted between 45-60 minutes and were audio-recorded and transcribed. (Please see Annex C for the complete list of questions covered in the key informant interviews. (Annex D for Spanish version)) FOCUS GROUP DISCUSSIONS There were two groups of stakeholders with whom the ET conducted focus groups: youth (male and female groups), and implementing partners. Focus group discussions with youth participants took place in Santo Domingo and Boca Chica on October 20-21, 25-28, and 31, 2016. (Please see Annex B for the complete schedule of data collection activities.) Discussion groups engaged participants in age-appropriate 11 activities that helped to answer questions about how they perceived themselves and their environment prior to the ARY Activity and now. Facilitators prompted youth to gather as much information as possible on how the ARY Activity has impacted their ideas and behaviors. Focus group questions also captured information on what they liked and did not like about the program, as well as additional services or programs they felt youth in their community needed. (Please see Annex C for the complete list of questions covered in the focus group discussions. (Annex D for Spanish version)) Focus group facilitators worked separately with youth in the two age cohorts (11-17 and 18-24), with discussion lasting approximately 75 minutes. Focus groups with key implementing partners asked the participants to score the ARY Activity overall and then asked about specific features including the various topical areas, the cross-sectoral nature of the project, and the collaborative processes. The intent was not to render an “effectiveness” score, but rather to stimulate discussion amongst the participants regarding the various features of the ARY Activity and their relative values. Table 4 below summarizes the number and range of respondents sought for qualitative data. Table 4: Summary of KII and FGD respondents RESPONDENT ORGANIZATION FOCUS GROUP RESPONDENT TYPES DIRECTOR SPECIALIST SCHOOL ADMINISTRATOR COORDINATOR OR STAFF BUSINESSMAN COMMUNITY LEADER SUB-AWARDEES Children International – Boca Chica 2 1 1 1 1 Acción Comunitaria para el Progreso, Inc (ACOPRO) 2 1 1 1 1 Caminante 2 1 1 1 1 1 Children International – Santo Domingo 2 1 1 1 1 1 Community Service Alliance (CSA) 2 2 1 1 1 Movimiento Socio-Cultural para los Trabajadores Haitianos Incorporados (MOSTCHA) 2 1 1 2 1 1 Project Hope 2 1 1 1 2 Instituto Dominicano de Desarrollo Integral (IDDI) 1 12 Profamilia 3 1 4 1 Fundación Pediátrica por un Mañana (FPUM) 1 Fundación Red de Jóvenes Unidos de Guachupita (FURJUG) Junior Achievement Dominican Republic (JADOM) 1 Sur Futuro 1 Other stakeholders Entrena 1 8 RTI 1 INFOTEP 1 EDUCA 1 Ministry of Youth 1 Ministry of Economy Planning and Development 1 TOTAL FGDs 18 TOTAL KIIs 57 TOTAL FGDs and KIIs 75 DATA PREPARATION AND ANALYSIS Data preparation and analysis for the quantitative youth outcomes survey and the qualitative KIIs and FGDs began upon completion of their respective data collection periods, and concluded with the submission of the draft report on December 23, 2016. YOUTH OUTCOMES SURVEY DATA Upon completion of data collection, Ipsos submitted the fully cleaned raw dataset on November 23, 2016, on which the ET performed an additional round of data cleaning. Using the statistical software Stata, the ET looked to identify unexpected outliers and data inconsistencies. SI worked alongside the data collection firm to help explain and inform any errors or anomalies found in the data. When the intermediate round of data cleaning was completed, the team began its initial analysis:  DESCRIPTION OF THE SAMPLE This section set out the basic descriptive and summary statistics of the full sample, disaggregated by gender, age, and other interesting and relevant indicators. The team employed tabulations, cross-tabulations, and means to generate this information.  ANALYSIS OF THE EVALUATION QUESTIONS This section focused on the three evaluation questions that the survey sought to answer, and consisted of comparisons between results from the intake-data based soft baseline and current data, interpreting and explaining 13 observed changes and linking to project activities wherever possible. Results were disaggregated by sex, age, and other characteristics. The team used a range of analytical techniques including correlations, regression, and t-tests to measure and interpret statistical change.  FURTHER INTERESTING FINDINGS AND ADVANCED ANALYSIS To the extent possible, SI delved into more nuanced data analysis, employing bivariate and multivariate regression analysis to help add context to outcome findings. It is important to note that the midline evaluation of the ARY Activity is considered a performance evaluation (as opposed to an impact evaluation) due to the absence of a counterfactual that would demonstrate desired outcomes within a similar population that had not received the intervention. In practice, this means that SI was not able to definitively attribute the ARY Activity’s activities to any observed changes in outcomes of interest. This is particularly important given the wording of the second and fourth evaluation questions.17 Nonetheless, SI used rigorous quantitative data analysis, combined and triangulated with informative qualitative analysis, to explore and comment on potential associations between project activities and outcomes of interest within the target population. QUALITATIVE DATA FROM KIIS AND FGDS All qualitative notes and recordings were transcribed in Spanish by the data collection firm in Santo Domingo, then translated and back translated by a translation firm they contracted. The ET received the Spanish and English transcripts of KIIS and FGDs on November 4, 2016, and then employed several qualitative research assistants to code and analyze the resulting data using Dedoose. The team developed a codebook as the first step in analyzing the data which can be found in Annex E. The codebook consisted of major sections corresponding to the five evaluation questions to capture findings on the ARY Activity’s outcomes, contributing factors to those outcomes, sustainability, capacity building, and perceptions of the ARY Activity. Additionally, the codebook included several sub-sections corresponding to activity sectors (education, health, livelihoods, etc.) and cross-cutting themes like gender which allowed for more nuanced analysis. When possible, the team identified and compared feedback on specific program interventions to highlight the most or least effective approaches. The team utilized Dedoose qualitative analysis software, a program that allows for robust analysis of qualitative data, and proceeded to use the qualitative data to inform, triangulate, and complement quantitative data, as well as informing answers to evaluation questions on its own. Data was compared across respondent groups (employers, Entrena staff, sub-awardee staff, youth, etc.) to identify differences in perspective. LIMITATIONS The ET is confident that the methodology detailed above is appropriate for the stated purpose, objectives, and questions and that this evaluation report provides valuable information that can guide future programming. However, there were a few challenges and limitations associated with this approach. PROJECT COMPLEXITY The ARY Activity is a complex project that crosses multiple sectors. While this approach is true to a Positive Youth Development approach, it also makes it more complicated to evaluate, as different partners 17 Should USAID/DR determine that they would like to determine the contribution/attribution effects for EQ2 and EQ4 with statistical significance, an experimental or quasi-experimental design for future similar interventions could be explored. 14 and implementers worked with different youth and utilized different implementation approaches. To overcome this challenge, the ET examined both the successes and challenges encountered by single sector partners, as well as success and challenges implementing multi-sector integrated approaches. Furthermore, the team’s sampling strategy targeted stakeholders from each category: youth participants, IP, sub￾awardees, CSOs, private sector, CBOs and government representatives. LOW REPORTING OF NEGATIVE BEHAVIORS Due to the nature of the project, many of the questions included in the Youth Outcomes survey18 pertained to participation in sensitive activities. These questions received a lower than expected number of affirmative responses, despite taking actions to reduce negative response bias, such as having the enumerator hand the tablet over to the youth to record his/her answers privately. The topics in question include drug use, pregnancy, prostitution, and violent behaviors. In addition to casting some doubt on the accuracy of respondent’s self-reporting, the very small number of people who admitted to engaging in the above activities made it difficult to perform in-depth analyses beyond descriptive analyses. The evaluation reports how many respondents indicate participating in these activities; however, due to the low number of people who admitted to using drugs, being party to a pregnancy, or engaging in prostitution, further analyses like comparisons with intake data were not useful to the ultimate findings of the evaluation. Though the overall number of reported participants is low, a sufficient number of respondents reported violent behaviors (such as being involved in fights and attacking others), which allowed for some additional analyses, explored under Evaluation Question 1. VALIDITY AND RELIABILITY Finally, it is important to note that the data collected are not generalizable to the larger population of project participants. This is due to two primary reasons. First, the sample was drawn from one geographic region as mentioned above, as per agreement with USAID; therefore, the team cannot draw conclusions for participants located outside of that context. Further, the challenges experienced during data collection led to a different group of youth being interviewed than originally intended. Difficulties locating and gaining agreement for participation, especially for a potentially vulnerable group like at-risk youth, are not infrequent. Thus, the evaluation team over sampled, and randomly selected 1600 youth as potential participants in the survey. Though the sample was randomly selected from the ARY Activity’s database of youth participants within the area, many could not be located or interviewed. Many of the location and contact information details obtained from the implementing partner were inaccurate, making it difficult to find survey respondents. Because the interventions had already ended by the time data collection was underway, it was not possible to use the ARY Activity implementation sites as potential data collection sites. Though the survey team did not stray from the randomly selected sample of 1600 youth, the team is aware that those respondents unable to be located may differ in key characteristics from those the surveyors did locate. This phenomenon is known as non-response bias. For instance, the ET visited schools and youth centers, and relied on the support of sub-awardees for youth outreach to better locate some of the respondents. However, it is likely that youth who were still in school and actively visited the youth centers might be different from those who do not attend school or frequent youth centers. As a result, those included in our sample may have different characteristics than those not included, and results are likely skewed toward higher school enrollment than the full sample would be, for example. 18 It should be noted that many of the sensitive questions in the youth outcomes survey were taken from the intake data collection form, in order to ensure consistency in approaches between baseline and endline. 15 The team gathered surveys from a sample of 600 youth in Santo Domingo, stratified by age group and drawn from a sample frame of 1600. This sample allowed the team to conduct a number of analyses, including comparisons between age groups and, in some cases, sex. However, these analyses became more difficult with questions that only pertained to a smaller portion of the respondents (such as questions about sexual activity, which only applied to the 44% of respondents who reported being sexually active). In conducting statistical tests, statistical significance is typically a key indicator of whether a finding can be claimed with any certitude. However, statistical significance can be highly dependent on sample size (the larger the sample size, the smaller amount of change that can be detected). In some of the analyses, no statistical significance was found. However, this may be due to the small sample size, as opposed to a lack of change. These cases have been noted in the text. The ARY Activity was not implemented in a way that allowed the estimation of what would have happened to targeted youth had the project not intervened (the counterfactual). This limited the ability of the quantitative analyses to estimate the impact of the project on anticipated outcomes (research question 1). The lack of counterfactual estimate is particularly challenging when youth are involved, as natural age progression has a strong influence on life choices. For instance, as children get older and become young adults, they are more likely to have ever had sex and they are less likely to be enrolled in school – particularly after the age of 18 – even if they have not formally completed their studies. This effect of age progression is supported by decomposition analyses conducted by the evaluation team. The details of these analyses are reported in Annex F. Age was the primary contributing factor for changes observed in enrollment status, engagement in sexual activity, consuming alcohol, and current employment. Given this effect, estimation of what has happened to at-risk youth who did not benefit from the program would have helped isolate the effect of the program from the effects of age progression. To mitigate this limitation to the extent possible, the ET has combined the quantitative findings with the qualitative findings, and has sought secondary data on national averages for youth outcomes, which were used to contextualize the findings. DATA ON INTERVENTION INTENSITY The evaluation team would have liked to conduct analyses examining the effect of the frequency or intensity of participation. However, given the type of data that was available from the implementer regarding the intervention intensity, these analyses were not possible. The data necessary for this type of analysis would have represented the frequency of individual participation. However, though the team was able to obtain data regarding the number of intervention types (health, education, etc) each person participated in (regardless of how frequently they had participated) as well as the total length of their enrollment (from start to end, regardless of how frequently they had participated during that time), neither was a good correlate to the frequency of participation. Though the evaluation team attempted some analyses using the available data, the results were not meaningful, and therefore have not been presented here. 16 III. EVALUATION QUESTION 1: YOUTH OUTCOMES WHAT IS THE LIKELIHOOD THAT THE AT-RISK YOUTH ACTIVITY’S YOUTH BENEFICIARIES WILL: RETURN TO AND STAY IN SCHOOL; REDUCE RISKY BEHAVIORS; OBTAIN AND MAINTAIN JOBS; AND, START THEIR OWN BUSINESS? FINDINGS RETURN TO AND STAY IN SCHOOL The majority of survey respondents indicated that they were currently enrolled in school (86%), and the majority reported having good records: an average of 1.5 days of unexcused absence from school in the last year, 81% of students were not suspended in the last year, and 75% had never in their life had to repeat a grade. Table 5, below, provides an overview of the changes that occurred between intake and follow up in enrollment. Table 5: Enrollment Rates ENROLLMENT STATUS BOYS GIRLS TOTAL Returned to school (not enrolled at intake/enrolled at follow up) 8 20 28 Stayed in school (enrolled at both intake and follow up) 175 305 480 Dropped out of school (enrolled at intake/not enrolled at follow up) 28 30 58 Remained unenrolled (not enrolled at both intake and follow up) 3 13 16 Further, 88% of respondents who were in school indicated that they felt the ARY Activity had a strong influence on their likelihood of staying in school until they completed their education. Among survey respondents who were not in school and had not yet completed their studies, the majority indicated that they were at least somewhat likely to return to school. Overall, the survey responses regarding educational outcomes were largely positive. However, the ET recognizes that a large number of youth were contacted through schools. Thus, the youth contacted may be different in these characteristics than those who were not able to be contacted. These largely positive averages are supported by responses from interviewees and focus groups, which indicated that the ARY Activity had a positive effect on youth’s enrollment in school and/or youth’s regular attendance. Similarly, a majority of interviewees and one focus group noted that the ARY Activity had had a positive impact on school retention and reducing drop outs. As one school administrator said: “I think those benefits will remain throughout their whole life. Because, if they had difficulties reading and, with the tutoring and the personalized education, they can read, that will stay with them forever, if they write. Because, you know what happens when those kids are over aged in those grades. They tend to dropout, because they … feel embarrassed if they’re given homework and have to do a 17 presentation on it. …by having that personalized attention, they managed to cross that barrier.” ~School administrator Despite these positive impacts, however, respondents did note the limitations of the program and the reality that some obstacles remained. Among the remaining obstacles was the influence of family, as well as neighborhood/street environment. Unfortunately, given currently available data, it is not possible to control for these potential influencing factors through the use of a comparison group. Thus, the ET relied on a pre/post assessment of respondent’s enrollment status and level of academic achievement. In terms of enrollment, the ET was able to use two different estimates of the respondents’ enrollment status at the time they started the program to assess changes before and after the program. In the first case, a retrospective question was added to the follow-up survey to ask respondents whether they had been enrolled in school when they started the program. The data collected by the implementing partners through the intake survey provided the second estimate of enrollment status at the time respondents entered the ARY Activity (e.g. an approximation of the baseline). Having access to two separate estimates of enrollment status when participants entered the ARY Activity allowed the ET the opportunity to evaluate the reliability of the retrospective recall data collected during the follow up survey. In comparing the two estimates of the participants’ enrollment status at the time they entered the program, the team found that respondents were over-reporting their baseline enrollment (96% said at follow up that they had been enrolled when they joined the program, whereas intake data indicates that only 92% had actually been enrolled when they joined the program). This difference is likely due to recall bias, a known challenge to using retrospective recall data. Given this bias, the preferred analyses use the intake data rather than the recall data (though the results are similar). For the respondent pool as a whole, pre/post analyses find that respondents were significantly less likely to be enrolled in school at follow up than they had been at baseline. However, this is likely due, at least in part, to natural age progression; as years pass, students are more likely to complete their studies, leave school to find a job, etc. When age groups are compared (for example, those who were 11-17 at intake to those who were 11-17 at follow up), the differences between the intake and follow up groups are not statistically significant. But, given the short span of time between the start of the ARY Activity and when the follow up survey was conducted, and given the national dropout rate of 3.3% per year19, without the program, one would assume that fewer of the respondents, even within the same age groups, would be enrolled at follow-up. The likely effect of natural age progression on student enrollment is supported by the decomposition analysis described in Annex F. This analysis shows that age is the number one explanatory factor impacting the change over time seen in the data. Given the strong role of age progression in the quantitative findings (though not conclusive evidence), the quantitative and qualitative findings are aligned in suggesting that the ARY Activity had a positive impact on enrollment. Another way to examine enrollment data is to compare national averages for each age group. Figure 3 provides a comparison of the national enrollment rate alongside the survey respondent’s enrollment rates at intake and follow up. As noted above, the differences between intake and follow up among respondents are not statistically significant. 19 ENTRENA, 2013 18 In terms of educational achievement, the pre/post analyses found similar findings to that as enrollment. Respondents were significantly more likely to have achieved additional levels of schooling at follow-up than intake. Again, this would follow the natural progression. This finding was consistent across age groups and sex. REDUCE RISKY BEHAVIORS Sexual Activity According to the survey results, 44% of respondents indicated ever having been sexually active (39% of girls and 54% of boys). The median age at which they first became sexually active was 15, which is younger than the national average of 17-18 years old (depending on age group, which were different than in this survey)20. Among those who were sexually active, 76% of boys and 77% of girls reported that they had used an effective form of birth control during their last encounter (defined as condoms, birth control pill, IUD, implants, injections, or sterilization). Eighty-eight (88) percent indicate that they believe they could convince their partner to use a condom, even if the partner was reluctant (this finding was consistent across genders, with 89% of boys and 88% of girls responding “yes” to the question). Only four people reported having ever had sex for compensation, and only one person reported having contracted an STI. In considering the potential for differences in sexual behavior between boys and girls, the ET asked the implementer about ways in which the training on sexual and reproductive health may have been tailored for the specific needs of different sexes. However, given that the PEFPAR guidelines do not require this type of tailoring, the implementer reports that the same content was used for all sexes. A number of interviewees (24) and focus groups (8) indicated that they thought the program had had a positive effect on the sexual activity of youth, particularly in terms of contraception use, as well as in HIV and STD prevention. As one young man phrased it: “One used to think things… I would think about having children early on, so that when I’m with them, I’m not too much older than them; but the project taught me 20 Centro de Estudios Sociales y Demográficos (CESDEM) (2014). Encuesta Demográfica y de Salud República Dominicana 2013. Figure 3: Enrollment Rate by Age Group 90% 47% 94% 65% 96% 66% Under 17 18 and over Enrollment rate by age group Entrena ARYI intake ARYI follow-up 19 another way and to think differently. You can’t have intercourse without protection, a pregnancy at an early age either because I was getting involved with minors like myself.” ~At-Risk Youth Activity participant Pre/post analyses on birth control use support the finding that the program increased the use of birth control. The analyses found that respondents were significantly more likely to have used an effective form of birth control during their last encounter at follow-up than at intake. This finding is consistent across age groups and sexes, as depicted in Figure 4 below. Figure 4: Respondent Use of Birth Control during Last Sexual Encounter 76% 76% 76% 24% 24% 24% 17 and under 18 and over All men Men's use of birth control during last sexual encounter Effective None used 79% 76% 77% 18% 22% 21% 17 and under 18 and over All women Women's use of birth control during last sexual encounter Effective None used No response Not effective 78% 76% 77% 21% 23% 22% 17 and under 18 and over All men and women All respondents' use of birth control during last sexual encounter Effective None used No response Not effective 20 At the national level, 38.1% of women (data were only available for women)21 reported using effective forms of birth control compared with 61.7% of women in the ARY Activity survey. When disaggregated by sex, the team found statistically significant differences for girls, though not statistically significant for boys. In a similar vein, respondents were significantly more likely at follow-up to believe they could convince their partner to use a condom (of those with both intake and follow-up data, 96% believed this at follow-up, compared to 80% at intake). In examining whether youth were sexually active at all, however, the data were less conclusive. The project may have hoped to see a reduction in youth sexual activity; however, because the intake form question was phrased as, “have you ever…,” as opposed to, “In the last 12 months…” the number of respondents reporting “yes” could only increase over time. Thus, as might be expected, the team finds that respondents are statistically significantly more likely to have ever engaged in sex at follow-up than at intake (which is consistent across age groups and sex). This finding is also consistent with natural age progression; as youth age, they are more likely to have sex, a likelihood that is supported by correlation analyses showing a positive and statistically significant relationship between a youth’s age and ever having sex. Crime and Violence Crime and violence are complex issues that are experienced both at the individual and the community level. Though interviewees and focus groups often reported that ongoing crime and violence in their communities were exogenous factors that might limit the ARY Activity’s impact, there was also a sense among these respondents that, at the individual level and among the specific participants, the program was able to impact crime and violence. In particular, the qualitative data collected from interviews and focus groups find that professionals involved in the project believed that youth seemed to be developing more responsible and less aggressive behaviors. For example, one headmaster noted that “Before [the ARY Activity] they fought a lot, everything was a quarrel, an argument, this has diminished a lot this year, in fact, we are surprised… the youths are getting on a lot better, they seem calmer, they seem more serene, more focused." Further, some community members indicated witnessing change in the surrounding communities. Surveys of the participant youth themselves revealed a mixed picture. On average, using a 10-point scale where 10 is “high impact”, respondents felt that crime (5.1/10) and delinquency (rating 6.4/10) had a medium impact on their lives. Very few respondents indicated having engaged in the crime and violence scenarios asked about in the survey during the last 12 months (such as stealing, attacking someone, carrying a weapon, etc.). However, exceptions were those having participated in a fight (18% of respondents) and having been attacked or robbed (25% of respondents). Comparisons to intake data show some, albeit mixed, changes before and after the project. 21 Ibid. 21 Whereas self-reported involvement in theft or violent behavior had decreased from intake data, involvement in fights and injuring others had increased. The surveys also showed an increase in the percentage of people who reported having been attacked or robbed, which could be contributing to the sense from some interviews that overall violence within the communities was increasing. Figure 5 depicts the changes over time in those participating in theft, robbery, and fights, as well as those who were a victim of an attack or robbery. Figure 5: Youth involvement in crime and violence Alcohol and Drugs Twelve interviews and three focus groups noted that they had seen a positive effect of the ARY Activity on the use of drugs and alcohol. In particular, some interviewees noted how youth previously tended to hang out and drink on the street. But, with jobs to go to and lectures to attend, interviewees reported less of this activity, which benefits the whole community. The number of youth reportedly consuming alcohol is supported by survey data. Fifty percent of respondents report having consumed alcohol in the last year, and comparison with intake data show that significantly fewer youth reported consuming alcohol; among those with data at both intake and follow-up, those who had consumed alcohol reduced to 64% from 76% at intake. When separated by age groups, the difference between intake and follow-up was statistically significant for younger youth, but not for older youth (however, this may be due to the small sample size, with p=0.09). When analyzed by sex, boys were significantly less likely to drink at follow up than at baseline, but for girls, the change was not significant (again, this may be due to the small sample size, with p=0.08). As with school enrollment rates, the team’s decomposition analysis found that age progression played a significant factor in the change over time that was observed (see Annex F for details). The findings were more nuanced, however, when it came to questions about how much youth were drinking. Among those who did drink alcohol, a comparison to intake data for those who reported consuming five or more drinks at a time showed that more respondents were drinking more than five at 2% 3% 9% 12% 5% 0% 19% 24% Participating in theft Participating in robbery Participating in fights Victim of attack or robbery Youth involvement in crime and violence ARYI intake ARYI follow-up 22 follow-up than at intake (50% of drinkers at intake vs 87% of drinkers at follow-up). This finding was consistent across sex and age group. As for drug use, only three respondents admitted to using drugs in the last year, and 69% of all respondents reported that having participated in the ARY Activity programs had a strong positive influence on their likelihood of using drugs in the future. Fifty percent of respondents said they had consumed alcohol in the last year, and, of those that drank, 27% reported drinking five or more servings of alcohol at least once a week. Due to the small sample size of respondents at intake and follow-up that had used drugs (five and three, respectively), comparisons to baseline data for drug use were not possible. OBTAIN AND MAINTAIN JOBS Thirty interviews and ten focus groups mentioned that the ARY Activity had had a positive impact on youth obtaining jobs such as clerks at local hotels and stores. Ten of those interviews and two of the focus groups specifically mentioned the positive impact of the internship program for youth to be able to gain experience, enhance their work ethic, and motivate them to find a job, all of which helps increase their likelihood of finding a job. Eleven individuals also specifically noted that the program had had a positive effect on youth’s ability to maintain a job, particularly given the focus on soft skills like emotional control, work ethic, and relationship management. One employer noted, in response to a question on the difference between youth who had been in the program vs. other youth: [There was] “an enormous difference. You receive someone completely rough, without any training about anything. But from here [the ARY Activity] people already take away a concept, a discipline, a dynamic, orientation and it becomes easier for them.” ~Employer According to the surveys, nearly a quarter of respondents indicated that they currently have a job (144 youth). Of these, 77 were for full-time jobs. Sixty-eight percent of those who had jobs said that they were either “very satisfied” or “somewhat satisfied” with their job, a finding that was consistent across sexes, with 69% of boys and 67% of girls reporting satisfaction with their job. Among girls, the biggest positive factors affecting satisfaction were the job responsibilities and the wages, followed by boss or supervisor, while the biggest negative factors for girls were wages and the hours worked. For boys, the most common positive factors were the job responsibilities, their boss or supervisor, and their prospects in the future, while the biggest negative factors were their wage and the hours worked. For both boys and girls, it appears that wages had the ability to be both a positive factor as well as a negative factor, depending on the situation of the particular respondent. In comparison with baseline, respondents were significantly more likely to have ever had a job than at intake, which was consistent across age groups and sexes. However, as with education, there is a likely link with age progression; as youth become older, they are more likely to hold a job. However, when comparisons were made regarding whether the respondents had worked in the last week, there was no statistically significant difference (though it nearly was at p=0.05). This was consistent across age groups and sexes. As described in Annex F, the effect of natural age progression was confirmed by the decomposition analysis, which, again, found that the primary explanatory variable impacting change over time was age. 23 Though positive sentiments regarding the ARY Activity’s impact on youth’s ability to find and maintain jobs were more prevalent, a notable number of interviewees and focus groups (seven and one, respectively), noted that the potential impact of the program did have its limitations. In one case, the implementing partner did not provide the promised internship opportunities. Others noted that employers were still reluctant to hire youth, particularly those with legal records. In other cases in which internships were obtained, they did not necessarily lead to a paid a position, which was a limitation. START THEIR OWN BUSINESS Twenty-six individuals and four focus groups mentioned that the ARY Activity had had a positive impact on entrepreneurship and the preparation and skills (both hard and soft skills) one would need to start their own business. In spite of this, respondents did note that only a few participants were actually successful in starting their own business. One staff member noted that only about 5% of the youth they trained moved forward with presenting a business plan. Others noted that finding financing is still a significant constraint, particularly for youth. Many do not have credit histories or track records to qualify for loans, and many do not have the necessary collateral. These findings show the eagerness of participants to start businesses, and survey data confirms qualitative evidence of constraints. Only 55 respondents (25 boys and 30 girls) reported ever having started their own business, and only 24 (12 boys and 12 girls) reported currently running one. Forty-seven percent of respondents said that they were either very likely or somewhat likely to start a business in the next year, which was consistent across sexes (46% of boys and 48% of girls). However, respondents did have some concerns about starting a new business, including having access to funding, as well as access to support services. No data on business ownership was collected during intake, so no pre/post analysis is possible. CONCLUSIONS On all four key outcomes—returning to and staying in school, reducing risky behaviors, obtaining and maintaining jobs, and starting a business—the evaluation finds at least some evidence to support a positive effect of the ARY Activity. However, the limitations of the quantitative evidence temper the evaluation’s ability to definitively determine the extent of these effects. First, the quantitative evidence is not fully representative of the larger population, which limits the ET’s ability to generalize any findings from the data to all participants. Second, without a comparison group of at-risk youth that are similar to those who were targeted in the ARY Activity with which to compare the changes over time among program participants, it is not possible to determine what part of the results is actually due to the program vs. other, outside factors. The ET’s ability to assess the impact of the project on outcomes was also challenged by the limitations noted regarding intervention intensity. Ideally, the team would have been able to assess the impact of participating in more interventions or participating more frequently in interventions. However, as described above in the limitations section, the available data on intervention intensity was not adequate to support these analyses. This is true for both “positive” results such as the evidence showing fewer respondents are drinking as well as for “negative” results, such as the finding that more respondents were participating in fights and attacking others. To demonstrate the usefulness of a comparison group, one could take the example of youth participation in fights, where the survey data showed an increase in youth participation in fights from 8.9% at intake to 18.6% at follow-up. In the interviews, there was a perception among respondents that communities were deteriorating and violence was increasing. It could be possible that other at-risk youth, who had not participated in the program, may have experienced an even greater increase in violence given the general deterioration noted by interviewees. If, for example, 30% of these comparable, at-risk youth were involved 24 in fights at the time of follow up compared with the 18.6% of the ARY Activity respondents, the data would actually show that the ARY Activity was able to decrease the likelihood of youth participating in fights, as compared to what would have happened had the program not existed. As this hypothetical example illustrates, comparison data can help contextualize data and provide a reference point for gauging impact. Similar comparisons to non-participant youth of the same age groups would have also been helpful for mitigating the age progression issues noted in the findings. Given that no data are available for a comparable group of at-risk youth, however, such firm conclusions about the size and extent of impact cannot be made in this evaluation. However, the ET has made every effort to triangulate data sources to arrive at sufficiently robust conclusions about the effect of the project on outcomes, given the circumstances. The outcome for which the evaluation can most confidently show an impact of the ARY Activity is on participants’ use of effective birth control methods, which is supported both by the qualitative and quantitative evidence. As noted above, however, it is not possible to determine the size of this impact. The ET is also reasonably confident that the ARY Activity has had a positive impact on school enrollment, where the team believes that the decrease in enrollment from baseline to follow up is largely explained by age progression. On alcohol use, the ET is reasonably confident that the program has had a positive effect on whether youth are drinking. However, the evidence showing that those who are consuming alcohol are doing so more heavily is concerning. Further, without comparison data with which to compare these data, it is not possible to determine whether this change is better or worse that it would have been without the program. On obtaining and maintaining jobs, the ET is confident that the program has had a positive effect on the underlying soft skills necessary for employability (self-confidence, maintaining interpersonal relations, etc.) However, the team is not able to determine with confidence whether the program has had an impact (positive or negative) on the likelihood that participants would be currently employed. The mixed evidence regarding participation in criminal and violent behavior, as well as drug use, prevent the ET from confidently claiming any impact (positive or negative). For these two outcomes, the lack of comparison data is also compounded by the particularly low numbers of respondents who admitted participating in these activities. Though the ET is confident that the program has had a positive effect on the underlying soft skills for starting a business (interest, self-confidence, planning skills, etc), the ET does not believe there is substantial evidence that participants have been able start businesses as a result of the ARY Activity. The latter point cannot be definitively concluded given the lack of intake data on business ownership (in addition to the other quantitative limitations noted), but the small number of respondents reporting that they are currently operating their own business, and the high percentage of respondents who noted problems accessing financing and ongoing support, all of whom are reflected in the qualitative data, support this conclusion. 25 IV. EVALUATION QUESTION 2: SUCCESS FACTORS (A) WHICH MAJOR FACTORS HAVE CONTRIBUTED, SO FAR, TO THE ACHIEVEMENT (OR FAILURE TO ACHIEVE) OF THE RESULTS THROUGH EACH ONE OF THE PROJECT INTERVENTIONS BY THE END OF THE THIRD YEAR OF IMPLEMENTATION? (B) AND, TO WHAT EXTENT HAVE PUBLIC-PRIVATE PARTNERSHIPS CONTRIBUTED OR NOT TO THE RESULTS ACHIEVED? (C) WHAT ARE THE ESSENTIAL COMPONENTS FOR THE WORKFORCE DEVELOPMENT PROGRAM UNDER THE AT-RISK YOUTH ACTIVITY TO BE SUCCESSFUL? FINDINGS ARY ACTIVITY METHODOLOGY AND INFRASTRUCTURE Local implementers and Entrena staff reported that the program’s integrated approach reflects the complexity of the problems faced by youth and is key to affecting long term change. However, some felt that this integrative methodology is best suited to a long-term program22 with a focused area of intervention such as in healthcare or education alone. Representatives of five implementing partners desired a consortium of organizations to support their entrance into new intervention areas. Entrena staff found that not all organizations were equally equipped to succeed in each intervention sector and those more well-established programs benefitted the most. In addition to the founding partnership with IDDI and Profamilia, local implementers were evaluated by Entrena through an open call for proposals. Criteria established by USAID requires that sub-awardees “have the ability to perform fully under the terms and conditions of a proposed award” considering the organization’s “integrity, record of past performance, financial and technical resources, and accessibility to other necessary resources.”23 However, during the program, sub-awardees were provided support for improving technical, financial, and M&E capabilities. Entrena noted difficulties early on in “attracting organizations with established local presence and the technical qualifications and background we are requesting.”24 RTI’s failure to implement the monitoring and evaluation (M&E) system within the first two years made it difficult to track the quality of the intervention and the outputs. Even when the M&E system was in place, many organizations had difficulty maintaining an accurate and up-to-date database. Entrena conducted its own evaluation of implementing partners in 2015. However, youth beneficiaries rated more positively the teachers and staff from programs that received unsatisfactory ratings from Entrena. The main public￾private relationship in the ARY Activity was between community organizations and Entrena. This partnership was viewed as extremely successful in the technical support Entrena and RTI provided to organizations, specifically the management of records and the introduction of new programs and methodologies such as the Espacios Para Crecer, FOSIS, and creating the M&E system. However, five of the implementing partners interviewed requested more support in establishing formal agreements with public and private institutions. 22 AJ programs were implemented from 2012-2017. 23 “Subawards” AID-517-A-12-00002 USAID, Pg. 72. 24 Entrena SRL, “Dominican Republic At-Risk Youth Activity: Alerta Joven,” ANNUAL Report 2012-2013, p. 21. 26 EDUCATION INTERVENTION Key factors for achievement Espacios para Crecer (EPC) was identified more often than any other program across the intervention sectors as a key factor in the success of the ARY Activity. EPC was cited by over half of the 11 sub￾awardees participating in the evaluation (54%) and by key Entrena staff members as having a positive impact not just on school retention, but improving student’s interpersonal skills, psychological well-being, and sense of citizenship. Entrena staff, youth beneficiaries, and community organizations highlighted its pedagogical approach, which utilizes a dynamic, interactive, lesson plan with games, songs, and other activities, as among its strengths. The Youth Outcomes survey showed a close relationship between positive feelings about school and school insertion and retention. EPC also benefited from being a well-developed program trialed in similar contexts. As such, training methods and materials were clear and easy to use, contributing to what many found to be well-trained instructors. Of all the educational programming, including Abilities for Life (HPLV), and the Advantages of Staying in School (LVPE), EPC offered the longest period of continuous contact with participants: three hours a day for five days a week throughout the entire school year (10 months), for two program years. With 150 EPCs in ten municipalities, it is one of the most widely-implemented programs. Its work with the 11-14 age population was also seen by both youth and local implementing partners as an advantage of the ARY Activity, as it reached youth early. “[Espacios para Crecer] is a total success because it is a program that is implemented, also because of the time, it´s 10 months and is based on a schedule based on modules that are established, intervention modules and the children themselves really love the “Espacios para Crecer.” I really think it's one of the best initiatives. ~Entrena staff member Outside of the ARY Activity, yet enabling a positive impact on this intervention’s success, was the concurrent establishment of the “tanda extendida” (extended school day), which presented the opportunity to integrate EPC within the regular school day. The extension of the school day also reduced the time period that youth would be otherwise unoccupied and potentially drawn to illicit activities. The program “Quantum Learning” was also identified in several interviews as successful in promoting the retention of students returning to school or repeating grades. The accelerated pace of the program provides students the opportunity to advance through school more quickly, allowing them to rejoin their peers in the appropriate grade. The Youth Outcomes survey confirmed that youth who repeat grades tend to fall farther behind their peers in educational achievement, an issue Quantum Learning is designed to address. Particularly in the batey communities, a public graduation was viewed as a powerful motivation factor for staying in school, as they are rarely seen by community members. Access to spaces to offer the ARY Activity programming and for schools to accept undocumented students was facilitated by partnerships with local schools. The Ministry of Education was key to establishing some of these agreements. Local government entities, such as the mayor or Junta de Vecinos (Neighbor Board) also facilitated access to spaces. 27 Key factors in failure to achieve Many of the factors limiting the impact of education sector interventions were external to program design and implementation. School overcrowding is a nationwide problem that disproportionately affects socially and geographically marginal youth. Local implementing partners have worked with parents to ensure early enrollment as schools reach capacity and turn away students not enrolled on time. For many living outside the city center, transportation to an alternative school is financially and practically prohibitive. One leader of a community organization spoke about the impact of her organization’s work in this area: “After the, which included the children going to school, that they had to be enrolled on time to project, there was more awareness at the family level about what their responsibilities are avoid having the problem of not having room to be in the closest one and then having to send them elsewhere. This way they could even save in transportation expenses. This is what I mean about school insertion.” ~Community organization leader Moreover, as schools become overcrowded, school officials more frequently turn away students lacking documentation and those with a history of behavioral problems. Although the right to education is guaranteed by law and in 2011, the Ministry of Education issued a declaration that all children be registered in school regardless of their documentation, many do not comply.25 The elimination of the dual school session through the creation of the extended day exacerbated overcrowding. Organizations faced obstacles reaching youth who had already abandoned school as much of the educational programming revolved around the school. As of 2014, new regulations for youth 15 years and older who wished to return to school, require them to enroll in the adult literacy program Quisqueya Aprende Contigo without the possibility of re-enrolling with their peers.26 A public-private partnership was formed with private educational institutions UNIBE and INTEC to produce data on perceptions of causes of delinquency among youth in the country.27 Other partnerships were not evident in the data collected. HEALTH INTERVENTION Key factors for achievement The follow-up survey shows that the ARY Activity had a positive impact on the use of contraceptives. Among those who were sexually active, use of effective contraceptives among youth ages 11-17 increased from 57% before the program to 79% after the program. Youth ages 18-24 also increased contraceptive use by over 30% (from 58% before the program to 77% after the program). Rates of use of effective contraceptive were consistent for both sexes. Of those who had been taught about pregnancy prevention, 36% cited the ARY Activity as the source of the information. Three components of the health initiatives 25 “Illegal People’: Haitians and Dominico-Haitianos in the Dominican Republic” Human Rights Watch, 2002 Pg. 27. https://www.hrw.org/reports/2002/domrep/domrep0402.pdf. “Dejado Atrás: Como la Apatridia en la República Dominicana Limita el Acceso a la Educación de los Niños.” Instituto de Derechos Humanos, Georgetown Law 2014, Pg. 23. 26 Quisqueya Aprende Contigo Manual de Organización.” Ministerio de la Presidencia. 27 “Dominican At-Risk Youth Activity Alerta Joven Annual Report 2014-2015.” Entrena SRL, Pg. 45. 28 were deemed essential to the success of these health interventions. First, information about services and safe practices was provided in a variety of activities and settings not necessarily related to healthcare where both information and vouchers for free services were provided. This method reduced the stigma around seeking information and services, particularly HIV and STI testing. Healthcare interventions also benefitted from the participation of organizations that have extensive experience working in sexual and reproductive health, particularly Profamilia and IDDI. The second key factor was the use of peer multipliers to distribute information within their communities. As one community coordinator said, this approach made a difference because “[it] is not the same for a kid that I talk to them from an adult perspective than to be approached by a young person like them that can speak knowledgably about such difficult subjects for them like HIV, sexually transmitted diseases, pregnancies and the like, to get advice from someone young just like you that not only knows, but lives in the same conditions as you…the success of the program was in them being approached by other young people.” ~Community coordinator Six thousand of these multipliers worked to provide accurate information about healthy practices in ten municipalities.28 Additionally, addressing psychological issues, such as increasing self-esteem in educational interventions, was key to encouraging youth to value their health so that would reduce risky behavior. Key factors in failure to achieve Stigma and discrimination remain important obstacles in recruiting the most at-risk youth including commercial sex workers, intravenous drug users, and those having same-sex relations. The program reported only reaching 144 of these youth by the third year of implementation representing less than 7% of the cumulative target.29 This failure can also be attributed to youth’s reluctance to identify as a member of this group on intake documents. A report by the Consejo Nacional para el VIH y Sida found that 50.3% of sex workers in Santo Domingo reported hiding their status as sex workers from healthcare professionals.30 General stigma around issues of sexual and reproductive health in some cases was a barrier to distributing HIV test results and providing information to younger program participants, although many youth become sexually active early on. In year three, the ARY Activity did not reach its goal of providing HIV prevention interventions to youth ages 10-14 of either gender.31 One health coordinator highlighted this issue: “The [HIV] tests demand that a person younger than 15 years old has to have a consent signed by their parents, so what happens, we have to adhere to the health protocol, I mean, we couldn’t allow a child of 11, 12 years old to take the test without the parents’ consent, and that affected a lot the goals because for example 28 “Dominican At-Risk Youth Activity Alerta Joven Annual Report 2014-2015.” Entrena SRL, Pg. 10. 29 Dominican At-Risk Youth Activity Alerta Joven Annual Report 2014-2015.” Entrena SRL, Pg. 31. 30 http://countryoffice.unfpa.org/dominicanrepublic/drive/CONAVIHSIDASegundaEncuestaVigiliancia.pdf Consejo Nacional para el VIH y Sida, Pg. 76 31 “Dominican At-Risk Youth Activity Alerta Joven Annual Report 2014-2015.” Entrena SRL, Pg. 29. 29 we were in the community, the youngster that already taken the test, and sometimes the parents were not present and it [distributing the test results] couldn’t be done, which was affected greatly because of the population.” ~Health coordinator The Youth Outcomes survey found that a large proportion of the ARY Activity youth begin having sexual relations between 13 and 18, with a mean age of first sexual encounter of 14.5 for boys and 16 for girls. Implementers faced difficulties in presenting safe-sex practice information in some religiously affiliated schools due to stigma. In addition, historically, youth under age 15 could not access sexual health information in public schools, making it difficult for the ARY Activity to target a part of the population in need.32 Ability to reach this population was also affected by the project eliminating funding for health programs in 2014. These funds were targeted specifically at HIV prevention and testing. KIs insist that teenage pregnancy is a key risk factor that deserved more support. Beyond sexual and reproductive health, there was limited funding to address issues of alcohol and drug consumption and psychological health. Despite this limitation, over two-thirds (69.1%) of youth stated that the ARY Activity had a “strong positive influence” on their future likelihood to reduce their use of drugs. Rather than being treated directly via the ARY Activity health interventions, the improvement of self-esteem seems to have been addressed more fully by the educational programs. The education programs also kept the students occupied after school, which is a time many of them are left unsupervised and are likely to participate in risky behaviors. Although fewer reported consuming alcohol, those that did were more likely to drink more heavily (50% baseline vs. 87% follow-up). DOCUMENTATION INTERVENTION Key factors for achievement The documentation intervention included support to youth in obtaining documents such as birth certificates, notary corrections, school certification, and identity corrections. By year three, organizations were able to reach 83% of their goal of identifying undocumented youth and surpass their goal of obtaining documentation for participants.33 Partnerships with organizations experienced in this area, working with parents to resolve the issue, and the accompaniment through the process by community workers were key to reaching intervention goals of obtaining documentation for 2,000 youth. Key factors in failure to achieve However successful, documentation was identified by six of the partner organizations that participated in the ARY Activity and one government representative as one of the primary areas of failure. At the start of the program, documentation was identified as a “major challenge” given the extreme vulnerability of those affected.34 That is, while they met program goals for documentation (10% of those identified), those that could not secure documentation remained significantly at risk as their status prevented them from participating in other intervention programs. Organizations were not equipped with the financial or 32 “Dominican At-Risk Youth Activity Alerta Joven Annual Report 2014-2015.” Entrena SRL, Pg. 30. 33 “Dominican At-Risk Youth Activity Alerta Joven Annual Report 2014-2015.” Entrena SRL, Pg. 21-22. 34 “Dominican At-Risk Youth Activity Annual Work Plan for Year 1 July 5, 2012-July 31, 2013.” USAID-Entrena SRL, Pg. 8. 30 practical skills needed to navigate the complex legal process of obtaining documentation and sought partnerships with UNHCR and more established organizations for support. Addressing this issue put stress on organizations as lack of documentation is often a multi-generational issue that requires a significant amount of time to resolve. Lack of documentation affected the ability of youth to enroll is school, technical courses, find employment, and/or start a business. ENTREPRENEURSHIP INTERVENTION Key factors for achievement The self-employment fairs were often cited as the most successful entrepreneurship-related activities. The ARY Activity sponsored six regional fairs where 169 youth presented business start-up proposals. The strongest proposals in these fairs advanced to the national level and 60 youth were awarded seed money to start a business. The integrated method of the FOSIS program Entrena implemented was viewed as successful since it not only included entrepreneurship education, but also labor insertion and life skills. Many ARY Activity youth expressed a desire to start their own businesses (47%). Key factors in failure to achieve The first entrepreneurship training program that was introduced called “The Company” was deemed to be too short and limited to produce the desired results. One major weakness was that it did not integrate a life skills component. The follow-on entrepreneurship program was most frequently identified as the weakest part of the ARY Activity (ten KIs representing Entrena and five organizations), due mostly to the lack of access to microcredit. No current banking product exists for youth to establish businesses. The population of at-risk youth that the ARY Activity serves often does not have the credit history, access to guarantors, or collateral to take advantage of existing private sector products. Public-private partnerships did not result in the wide availability of a loan product for entrepreneurial youth in the ARY Activity. The program was also not able to offer loans. Entrena, JADOM, and its partner organizations were largely unsuccessful in working with banks to provide a loan product for entrepreneurs due to the high risk associated with these loans. EDUCA secured co-sponsoring for a 2014 conference on youth education and employment from Banco Popular. Banking institutions including Banreservas and Banco ADEMI were the most frequent collaborators. These collaborations were largely established with the cooperation of the Ministry of Planning and Development. Future programs may consider alternatives to private banking institutions when working with vulnerable populations. In addition, there was little accountability for fund management by fair winners. Monitoring and support for those who established business was also limited. A representative of IDDI, who has 14 years’35 experience providing micro-credit, suggested that the ARY Activity adopt their model of solidarity groups to address the issue of low accountability in the program. External factors include the conditions of the youth who have limited access to markets in the batey and often face difficult circumstances that compel them to use their business funds for personal reasons. Access to funding and business support services were the main concerns for youth planning to establish their own businesses. 35 http://iddi.org/es/microcreditos/ 31 JOB TRAINING AND INSERTION INTERVENTION Key factors for achievement The technical courses36 offered by the ARY Activity in partnership with INFOTEP are viewed by community implementers and Entrena as one of the most successful components of the program. The job training program benefitted from being based on a study of the areas of future employment needs. The top three areas of predicted growth were service and hospitality, information technology, call centers, and cinematography. The quality of the courses offered were well-regarded by youth beneficiaries, community coordinators, and Entrena staff. Contributions to training by employers experienced in the job requirements was also an essential part of these relationships. Opportunities for business people to meet youth during job fairs organized by Entrena, community implementers, and employers is a key factor in securing commitments to hire them. This direct contact resulted in more success with locally-based small companies according to community implementers. Key factors in failure to achieve Although the technical courses offered were based on a study of areas of employment need, they were largely limited to a few technical areas and the informal sector (e.g., grocery, pharmacy and haircare) and did not correspond closely with all the areas identified in the study. Courses in three of the areas of projected growth, information technology, call centers, and cinematography require greater technical and time resources that, according to Entrena, were not suitable for at-risk youth. Further, undocumented students were not eligible for study with INFOTEP, preventing one organization from partnering with the institute. Future programs may base career projections on specific communities and populations to guide workforce development. While the ARY Activity had surpassed its goal of providing job-training for youth by its third year, it failed to reach its goal of securing new or better employment for youth.37 The project found that it was difficult to place youth without previous work experience. In addition, youth are often not incentivized to work for unpaid internships given high transportation costs and companies that offer unpaid internships often lack the funds to hire the youth afterwards. Youth residing in bateyes may be disadvantaged in employment as there is heightened stigma towards youth from these areas and the increased cost and unreliability of transportation to businesses outside of the community.38 There are concerns about the sustainability of employment in the tourism industry. Service positions are frequently eliminated during the low season, making them unstable jobs. While most employed ARY Activity youth are satisfied with their jobs according to the follow-up survey, low wages are a key source of dissatisfaction with their employment. Moreover, despite the fact that females completed the job￾training programs at a higher rate (4640 females vs 2663 males), they were half as likely as their male counterparts to experience positive employment outcomes (928 females received new or better employment vs 1042 males). This figure in consistent with lower female participation in the workforce in the DR as a whole (59.0% of females aged 20-24 and 24.5% of females aged 15-19 were employed in either 36 The Entrena labor insertion specialist reported that most technical courses averaged 80 hours. 37 “Dominican At-Risk Youth Activity Alerta Joven Annual Report 2014-2015.” Entrena SRL, Pg. 35-36. 38 Interview with community leader CSA October 2016. 32 the informal or formal sector compared to 87.8% and 42.7% respectively for males) suggesting that women are either less likely to seek or receive employment in the Dominican context. 39 PUBLIC-PRIVATE PARTNERSHIPS The ability to develop public-private partnerships for job-training depended on the individual implementing organizations. Entrena did not play a strong role in establishing these partnerships. Instead, individual personnel from implementers were responsible for making connections with local businesses. Yet, this was not a specific criteria in sub-awardee selection. Organizations such as Caminante already established contacts with local businesses, and were more successful in using their contacts to place youth in jobs and internships. Business contacts were important as employers offered trainings and were the source of “the most important internships” in the ARY Activity, according to one program director, despite partnerships with larger companies such as Grupo Ramos. Organizations with a visible presence on the local and international level, particularly those already working in this area, had more successes in establishing these partnerships. The partnership with INFOTEP was likely the most important as they provided the majority of technical courses for over 17,000 youth by end of year three. While the partner organizations and youth were largely satisfied with the courses provided, undocumented youth were not eligible for enrollment. Key factors for achievement Offering job training programs within the community, as was done under the ARY Activity, is necessary for making them accessible to youth. Options need to be available for undocumented youth. Life skills training (e.g. communication, time management, and team work capacities) was viewed by employers as equally important to the technical training provided in the courses, and perceived to be an important aspect of the workforce development program. Introducing managers and business owners to ARY Activity youth through job fairs was viewed as necessary to overcoming the stigma of youth from at-risk communities. Increasing public-private partnerships to place trained youth in their first job is crucial. This was accomplished with mixed success in the ARY Activity and the strongest relationships were with small local enterprises. Jobs should provide a minimum of transportation and meals to be accessible and desirable to youth who cannot invest their own funds. Selecting courses described as “bridge” courses (60-80 hours) was viewed as important as they did not present a burdensome time commitment for youth but were long enough to impart important skills, especially life skills. CONCLUSIONS Achievement and failure to achieve the expected results in each intervention area depended on a variety of factors internal and external to the program. Qualitative interviews with Entrena staff and implementing partners suggest that an established record working in an area, clear methodologies, and quality staff were key to success across intervention sectors. Sustained contact with youth in extended programs also contributed to their valuation among community implementers and beneficiaries. 39 “USAID/Dominican Republic Cross-Sectoral At-Risk Youth Assessment” Education Development Center Inc. August 2010, Pg. 29 33 The failure to erect formal agreements with public and private institutions was a key factor limiting the ability to succeed in the areas of job insertion, documentation, and entrepreneurship. Most employment outcomes were generated from local businesses, suggesting that organizations struggled most to establish macro-level agreements. Formal agreements to provide microcredit, jobs, and public policies to ensure access to employment and education are necessary to ensure the success of these programs. Offering resources within the community (technical courses, medical care, documentation services, etc.) was a fundamental factor in the success of local uptake. Social and cultural attitudes were a central external factor moderating program success. Interviews with community implementers and Entrena staff suggest that providing direct knowledge of and contact with the ARY Activity programs and beneficiaries was a successful tactic for reducing stigma.40 Entrena and sub￾awardees emphasized the importance of soft skills as a key factor in educational and employment success. Youth association with the ARY Activity programs, according to employers and sub-awardees worked to reduce stigma related to crime and violence. Addressing stigma is also necessary for HIV prevention initiatives to reach younger populations, women, commercial sex workers, MSM, and FSM41. 42 40 While there is no data on the perceptions of youth involvement in violence and crime, a 2016 Gallup poll found that in comparison with Central American States, the Dominican Republic had the highest incidence of those affected by a recent crime (25%). These results suggest a widespread sentiment of insecurity. http://www.listindiario.com/las￾mundiales/2016/03/19/412281/republica-dominicana-con-el-porcentaje-mas-alto-en-centroamerica-y-el-caribe-de-hogares￾victima-de-delincuencia. 41 Christine Tagliaferri Rael (2015) argues that stigma is a significant barrier to health interventions for sex workers. Sexuality and Culture. 4, Pg. 674-684. http://ezproxy.fiu.edu/login?url=http://search.proquest.com.ezproxy.fiu.edu/docview/172039807 4?accou ntid=10901 42 Although the highest in the Caribbean region, only 34.5% of Dominicans support the right of homosexuals to run for public office. Researchers argue that this is a better measurement of stigma because running for office is fundamental to exercising citizenship. Montalvo, Daniel and Emily Saunders. Americas Barometer: Topical Brief. Nov. 10, 2015. Pg. 2. 34 V. EVALUATION QUESTION 3: SUSTAINABILITY ARE THE SERVICES OFFERED TO YOUTH BY ENTRENA AND SUB- AWARDEES SUSTAINABLE, IN TERMS OF FINANCIAL RESOURCES AND INSTITUTIONAL CAPACITY BUILT UNDER THE PROJECT AND POTENTIAL LOCAL UPTAKE? FINDINGS To answer this question, the ET relied on in-depth interviews with representatives from 14 organizations, the Entrena staff team, government officials, community leaders, school representatives, health services providers, and private sector representatives. SERVICES OFFERED BY ENTRENA The ARY Activity has offered multi-disciplinary services to more than 100,000 at-risk youth in the corridor between Santo Domingo and Puerto Plata, with a budget of approximately USD 20 million funded by USAID for five years from 2012 to 2017. The specific services, carried out by community￾based organizations, include: youth policy and civil participation; identity documentation; HIV/AIDS prevention and attention services; reproductive health and sex education; technical-vocational and entrepreneurship skills training; as well as interventions in school retention, insertion and completion. All ARY Activity implementers stated a will and a commitment to continue providing these services after the program ends. While the number of target beneficiaries has now far exceeded the initial goal of reaching 85,000 youth, it took more than two years for the project to gain traction. According to consortium members and sub￾awardees, several factors explained the project’s slow start, including the fact that it is a very ambitious project, implemented by a large number of organizations with different levels of institutional capacity and experiences who had to handle all five ARY Activity components, such as public policy or entrepreneurship, for which they probably had no previous experience. These organizations also had to learn new methodologies, namely strict monitoring and evaluation techniques and the electronic tracking of all beneficiaries. Training all sub-awardees took time. More importantly, the tracking system tried several different programs and did not really become functional until the third year. Informants indicated that by 2014, it was only 25% operational. For the first two years all reports, data collected and information coming in from the original 25 organizations had to be tracked in Excel files. Moreover, the project covered an extensive geographical area—9 provinces and 500 neighborhoods across the country, adding more complexity to implementation. Finally, changes in USAID leadership in the early stages meant changes in project expectations, which required adjustments that contributed to initial delays. All these elements introduced challenges in coordination and consistency in implementation. “Sustainability starts from day 1 of the project… we need to ingrain the mentality that ‘if this initiative is not sustainable beyond the five years of USAID support, then we have failed.’” ~Year 1 Work Plan, At-Risk Youth Activity Dominican Republic 35 SUSTAINABILITY SOURCES Public and Private Dominican Sources The ability of organizations to continue services once funding for the ARY Activity is over, hinges, according to Entrena documents, on the development of public-private alliances. Entrena has been working on building these alliances with large companies that have greater capability to assign resources for at-risk youth activities than the businesses located within vulnerable communities that are too small and too unstable to sustain their commitments. Several companies have already committed resources, but their contributions do not yet match the ambitious requirements of the ARY Activity. Entrena believes that one reason for the lukewarm result of the public-private alliances is that the private sector has not been involved in the project from inception. In fact, informants argue that strong alliances and partnerships should have been sought with main business associations, such as the CONEP (National Council of Private Enterprise) from the start. Funding commitment from the government will be essential for the continued support to at-risk youth. Entrena expects the government to provide funding comparable to that raised by the private sector. More importantly, Entrena believes the Dominican government should be primarily responsible for the continuation of the activities and services provided by the ARY Activity: “This project cannot end because USAID funding ends. The government is the one that should take it over, with support from all sectors, but the government should be the backbone of all this. It must be a government program.” ~Staff member from implementing partner Entrena visualizes the continuation of the ARY Activity through a “co-financing mechanism funded by the government but independently managed.” Officials from the Dominican government interviewed, however, stated that the government has yet to allocate a budget to continue providing ARY Activity services. In fact, evaluators could not identify a coherent official policy for the project’s continuation among those officials interviewed. The Ministry of Youth, for example, believes that sustainability could come from synergies between Entrena, the implementing organizations, and the Dominican Government, but it has yet to create those synergies with the ARY Activity. For the Ministry of Planning Economy and Development, sustaining the ARY Activity would require that the State create policies to benefit youth, especially the most vulnerable youth, at an early age. Policies outlined by an official of this ministry included providing seed funds or low-interest loans to young entrepreneurs to start businesses. Another proposal relied mostly on the current sub-awardees, arguing that the services would be sustained because local organizations have learned and adopted the methodologies and training provided by the At-Risk Youth Activity to instructors. These officials, however, doubt that sustainability would apply to vulnerable neighborhoods “because many of these young people go because it is USAID, a renowned institution they are familiar with, not a foundation,” according to a government official. While officials interviewed seemed to indicate that the commitment of the Dominican government to sustain the services provided by the ARY Activity has not yet been defined, Entrena staff expressed moderate certainty that Dominican institutions were committed to adopting at least some of the ARY Activity’s programs and methodologies. Several sub-awardees and some government officials are 36 convinced that the ARY Activity has filled a void left by a lack of comparable Dominican institutions, and that it has helped the government realize that it had not paid enough attention to vulnerable youth. In fact, some institutions are adapting and implementing the ARY Activity methods and practices. The Office of the President has reached out to the project implementers to identify common grounds to reach out to vulnerable kids. The Office of the Attorney General is modeling a prevention unit for youth in conflict with the law on the ARY Activity’s program. The Ministry of Youth has supported the ARY Activity on improving public policies and the legal framework for youth through collaboration for a guide to understand the General Youth Law 49-00. Through a joint effort with Entrena, the Ministry of Youth also supported the issuance of a guide for the youth entrepreneur, the establishment of the youth municipal councils along the intervention areas, and two National Youth Forums. While these were all important activities during the life of the project, the level of involvement of the Ministry of Youth was not as high overall as had been expected at the outset of the ARY Activity. At-Risk Youth Activity implementers, particularly Entrena, have also reached out to the media to provide information and data bases on research sponsored by the project. This helps raise awareness among a larger audience as to the extent of at-risk youth issues, and ways to address them. Despite the collaboration between these DR institutions and the ARY Activity, the evaluators could not identify indications of collaboration and coordination on an at-risk youth policy among the institutions. Each institution carries out its own set of activities. ENTRENA SUB-AWARDEES Entrena staff informed the ET that when sub-awardees prepared a sustainability plan at the end of the third year, most organizations indicated that they would be able to continue with some of the initiatives with existing resources. However, while all 11 sub-awardees interviewed for this evaluation have the skills and institutional capability required to continue implementing the same activities they carried out under the ARY Activity, they lack the financial resources to continue providing these services at the same level as with USAID funding. In fact, some of these sub-awardees cannot continue to provide any activities. A lack of financial resources is the major factor that hinders sustainability of the services, and is a challenge for all intervention types. The majority of organizations could continue disseminating information and adopting methodologies and practices into their own work, but cannot continue to offer the same services. For example, health services and technical trainings are very expensive, and had been funded directly by the ARY Activity. In fact, even organizations like Profamilia, which has its own clinics, could not continue offering the same volume of health services that were previously paid by vouchers offered to youth by the project and/or funded by PEPFAR.43 In the case of Profamilia, however, their long-standing structure built on peer-to-peer multipliers will probably continue to disseminate information. IDDI, despite being one of the best-established organizations with more than 14 years of experience in microcredit, health, and other sectors, faces the same scarcity of funds. An IDDI staff member stated, “We cannot do a technical course if there are no resources available; whether we like it or not, we need to pay the teacher, buy materials, you need a space.” Other organizations, which were already providing services, but have increased the number of beneficiaries with ARY Activity funding, will again reduce the scale of their services. In the case of FPUM: 43 Alerta Joven provided health vouchers to its beneficiaries to pay for services at PROFAMILIA and other health clinics. PEPFAR is the United States President’s Emergency Plan for AIDS Relief. It funded most of the HIV/AIDS initiatives of Alerta Joven. Funding for the Dominican Republic in 2015 was approximately $15 million, the year the funding ended. 37 “We are going to continue working, at a lesser scale, of course, because we cannot keep up with the level of expenses we had before…. the At-Risk Youth Activity gave us 50% and the Foundation [FPUM] the other 50%. So, if in 1 year and 18 months, I reached 1,400 young people, now I can reach 700 young people.” ~FPUM staff member The few sub-awardees that might be able to continue to provide ARY Activity services are those that have international links. JADOM, for example, is usually financed by private companies, and even with USAID’s support for the ARY Activity, 70% of their funding was still derived from the business sector. Children International and Project Hope may also be able to incorporate some of the ARY Activity’s programs into their normal curricula. Still, both institutions indicated that level of beneficiaries served could not be sustained. In general, regardless of the goodwill of sub-awardees, and despite their sustainability plans, their lack of resources is a challenging reality. These sub-awardees report a desire to do their best to continue with the ARY Activity mission and vision, but cannot do so with the same volume of beneficiaries. Several explanations were provided to the evaluators as to why the sub-awardees were not fully prepared to sustain the services. One reason is that the sustainability plans developed by sub-awardees under Entrena guidance came too late; several sub-awardees were essentially closing down as they drafted sustainability plans. Additionally, the ARY Activity missed the opportunity to train sub-awardees on how to provide services with their own resources and how to generate new funding with the newly installed capabilities. According to several sub-awardees, the sustainability plans did not include contacts with other sources of funding or helping sub-awardees with the proposal process. Still, effectively supporting sub-awardees in proposals for future projects would have taken much more time and resources than allowed by the ARY Activity funding for sustainability work. BENEFICIARIES AND VOLUNTEERS As of December 2016, the At-Risk Youth Activity has reported reaching more than 130,000 at-risk youth with their information, training, and support. Successes in particular interventions include the following: ▪ Almost 8,000 youth received information and/or participated in workshops about youth civil rights and duties44 ▪ Almost 2,000 received legal assistance on documentation ▪ At least 22,000 adolescents were helped to stay in school ▪ More than 900 teachers were trained in the Quantum Learning methodology45 used in the Espacios para Crecer (EPC) program ▪ 34,000 youth have received technical and/or vocational training (and almost 5,000 found a new or better job) 44 Entrena SRL, “Proyecto Alerta Jóven: Annual Report 2015-2016,” p. 6 and “Dominican Youth At- Risk Activity, Alerta Jóven: Annual Report 2014-2015,” p. 8. 45 Alerta Joven 2015-2016 Annual Report explains that Quantum Learning (QL) methodology “is a recognized Methodology for its brain-based, holistic approach to learning.” The methodology is used by Alerta Joven “to turn potentially boring or ineffective teaching/learning experiences into exactly the opposite: an environment where teachers and students are engaged.” Entrena SRL, “Proyecto Alerta Joven: Annual Report 2015-2016”, p. 15. 38 ▪ More than 60,000 received reproductive health assistance46 ▪ The At-Risk Youth Activity created 26 youth committees among participants and more than 30 support networks with more than 6,000 young multipliers (peer-to-peer facilitators, many of them also beneficiaries) in ten municipalities47 The volume of stakeholders reached by the project, particularly the young beneficiaries, is probably the best sustainability resource of the ARY Activity. While conducting fieldwork, the ET met dozens of these youth participants at focus groups, who confirmed their enthusiasm about the ARY Activity and how it changed their lives. These youth are the ARY Activity’s ambassadors of lessons learned. It is this small army of youth who may be the most reliable and committed to carry on the ARY Activity’s mission. CONCLUSIONS The ARY Activity is a pertinent and much-needed project. Its success in creating opportunities for at-risk youth, providing them with confidence and a renewed sense of future, merits the continuation/extension of the project. Initial evaluation results suggest, however, that the project may have been too ambitious in its dual goals of reaching a large number of beneficiaries over the extended Duarte Corridor and having sustained impact over a wide variety of intervention types. During qualitative interviews, the ET did not find evidence of a real commitment on the part of Dominican government officials to assume such an enormous endeavor. This does not preclude that the commitment might be there among institutions that the ET did not reach. Even if this were to happen, the amount of funding provided by the Dominican Government would almost certainly be less than what was provided by the US government. Sustainability through private sector partnerships represents a challenge in the Dominican Republic, as in many other countries. One could expect that large companies would be willing to provide funding for at￾risk youth initiatives as part of their social corporate responsibility. However, this funding has been slow, and contributions are not as substantial as are needed to sustain the ARY Activity. Entrena explained that businesses would be willing to donate perhaps USD 20,000, but not any amount near the total ARY Activity funds. Thus, the private sector’s expected contributions would not be enough to sustain the project. Still, Entrena continues to work ardently at seeking private sector support and will probably be successful in ensuring substantial support through its contacts and tenacity. The key would be to link the private sector resources to government funding. Despite its ability to garner funding from the private and public sector, Entrena believes that such an ambitious program cannot be accomplished without the continued support of the international community. Despite their willingness to continue implementing the activities and providing services to at-risk youth, individual sub-grantees are unlikely to be able to sustain the services offered, and expectations in this regard should be lowered. Sub-awardees overflow with good intentions, but their resources are meager. Despite the newly acquired skills and knowledge that will hopefully remain with the institutions and be put to good use, lack of financial resources will prevent continuing implementation at the same scale. Finally, the thousands of youth touched by the ARY Activity are the best hope for the continuation of the process if not the services. They are the ones that could become the real multipliers of the project’s 46 USAID and Entrena, “Proyecto Alerta Joven: “Comunicado de Prensa,” Santo Domingo, DR: October 11, 2016. 47Entrena SRL, “Proyecto Alerta Jóven: Annual Report 2015-2016,” p. 3. 39 teachings, by sharing with other vulnerable youth their own experiences and passionate support for the program and its activities. 40 VI. EVALUATION QUESTION 4: CAPACITY BUILDING TO WHAT EXTENT HAS THE CAPACITY BUILDING TRAINING OFFERED BY ENTRENA HELPED SUB-AWARDEES TO IMPROVE THEIR TECHNICAL AND FINANCIAL MANAGEMENT CAPABILITIES WHICH COULD HELP THEM ACCESS OTHER SOURCES OF FUNDING ONCE AT-RISK YOUTH INITIATIVE IS OVER? FINDINGS To answer this question, the ET conducted qualitative interviews with representatives of 14 sub-awardee organizations, as well as with Entrena and RTI staff. Perceptions from some government officials who had collaborated with the ARY Activity were also considered. SUB-AWARDEE CAPACITY BUILDING The services provided by the ARY Activity are carried out by 23 community-based organizations that became sub-awardees through a competitive process under the Entrena consortium. In addition to the selection criteria explained earlier, an important requirement was that the organization be community￾based and enjoy recognition and respect within the neighborhood in which it operated. This was important to ease introduction of the ARY Activity, to reduce transportation costs and to increase potential for sustainability. Entrena was in charge of providing training and skills to the sub-awardees for the proper compliance with USAID regulations. This entailed strengthening each organization’s institutional capacity. When joining the ARY Activity, the experiences and skills of the community organizations differed significantly. A few sub-awardees, including the Instituto Dominicano de Desarrollo Integral (IDDI) and Profamilia, are long-established service providers already operating in poor neighborhoods. Several sub￾awardees are local affiliates of international organizations, among them Children International, Project Hope, and Junior Achievement Dominican Republic (JADOM). Each of these organizations had functional administrative and financial management procedures in place. In contrast, the management procedures of ACOPRO, a labor union organization serving the Sabana Perdida neighborhood, were rudimentary. Table 6: Profiles of sub-awardees interviewed during evaluation NAME ESTABLISHED NATURE FINANCIAL SUPPORT GOAL BENEFICIARIES AT-RISK YOUTH INITIATIVE NO. MO. CONTRA CT ACOPRO 1994 Local Domestic 3,400 36 Caminante 1994 Local Domestic 3,600 44 Children International 1979 Local Chapter Charity; Int’l donors 3,600 Boca Chica 3,600 Sto. Domingo Este 36 CSA 2004 Local Int’l donors 1,471 24 FPUM 2004 Local n/a 900 18 41 IDDI 1984 Local Int’l donors 19,530 56.5 JADOM 2000 Local Chapter Junior Achievement Int’l donors 9,639 24 MOSTCHA 1985 Dominican & Haitian Domestic & Int’l 1,700 36 Profamilia 1966 Local Services; gov’t; int’l donors 12,014 55 Project Hope 1996 Chapter Order of Malta Int’l 5.295 36 Sur Futuro Foundation 2001 Local Domestic 3,400 24 Source: ARY Activity webpage (www.alertajoven.org); organization’s websites The large differences in organizational structures, experiences and capabilities of the sub-awardees meant that some organizations would gain more than others from Entrena’s expertise-sharing and trainings. All representatives from sub-awardees organizations interviewed reflected positively on the ARY Activity and on the support and training provided by Entrena. The training included technical, administrative, financial, and communication skills, and all claimed to have learned and improve their capabilities with the trainings. IMPROVING TECHNICAL CAPABILITIES It became clear during interviews that Entrena trainings equipped these community-based organizations with new skills and capabilities that placed them in a better position to bid for contracts. The first new skill acquired was the ability to work with at-risk youth, broadening the scope for many organizations. All organizations joining the ARY Activity had a specific field of expertise, be it health, education, employment or the environment. Nearly all were accustomed to working with youth in vulnerable neighborhoods but few had significant experience working specifically with at-risk youth. The ET was told by several sub￾awardees that initially their staff had no idea how to find, contact, and approach the at-risk youth that they would be serving, much less how to talk to them. “We had worked all our lives with young people and we work with many young people like the ones in ALERTA, but there is a segment within ALERTA with whom we had never worked before, which we call ‘pepper’ - pepper because they’re spicy like pepper - which are young people who are in gangs, that are on drugs.” ~Sub-awardee staff member An RTI representative explained how the experience of working with the highly vulnerable population influenced the implementers: “They had worked at the schools but, to go house by house, or the boy who is on the court or smoking in the corner with a small group, how are you going to get 42 in? How are you going to talk with that person? You don't know. …The perspective really changed, uncovering a reality that people know that exists, but they weren't too open to it…. …. it was a very positive aspect, after that, the approach of organizations is not going to be the same.” ~RTI staff member Entrena provided the methodologies that each organization would utilize to work with at-risk youth. These methodologies endowed sub-awardees with new skills in the areas of education, health, entrepreneurship, employment, youth protection policies and advocacy, as well as documentation and outreach to potential clients, such as at-risk youth. Among the most notable and innovative methodologies for the Dominican Republic are the Spaces for Growth Model (EPC); the Quantum Learning used by EPC; and the Chilean model for Entrepreneurship and Employability. Sub-awardees became familiar with new methods unknown to them until now that they could market in the future. During KII interviews with the implementers, the ET was told that another added benefit for sub-awardees is that the community-based organizations became aware of youth-related national issues. Entrena provided them with information on national youth legislation passed or pending at the Dominican Congress. Also, all sub-awardees became familiar with the challenges of documentation that confront many at-risk youth. While sub-awardees claimed that they were overwhelmed by the travails of the legal process, Entrena trained these organizations on how to help youth and their families navigate the complex process of being legally registered in the country. IMPROVING ORGANIZATIONAL AND FINANCIAL MANAGEMENT Entrena staff explained (and sub-awardees confirmed) that broad organizational and financial management training was provided to all community-based organizations. This included training on proposal preparation, creating a logical framework, monitoring and evaluation plans, defining deliverables, creating communication strategies, and more. RTI, in charge of monitoring and evaluation, explained that sub￾awardees were required to complete self-assessments to identify weaknesses and how to improve them. RTI and Entrena staff worked with sub-awardees to improve their managerial and organizational skills. As a result, the sub-awardees learned or improved their report drafting skills, prepared a financial statement that met international standards, and more importantly, RTI explained, learned to meet deadlines. Several sub-awardees had not worked previously with international funds and were not accustomed to strict deadlines. RTI and Entrena officials claim that it was essential that sub-awardees understood that lagging behind had repercussions on the project’s cycle. Throughout the interviewees, most sub-awardees explained that they are confident that the training received will stay with them and help their organization to plan and operate more strategically. Also, despite being overwhelmed by the complexity and demands of the data collection platform, they learned to use the data bank and monitoring platform to follow up with the youth. However, some organizations cited a challenge to retaining the newly acquired capacity: personnel receiving training (including accounting, management, or financial staff) have moved on or needed to be dismissed for lack of funding once the ARY Activity ended. They expressed concerns that that the skills learned might not stay with the organization. Because of their differences, some organizations benefited more than others. Less structured organizations benefited the most in acquiring new skills ranging from simple filing to human resource 43 management. As a MOSCTHA project implementer suggested, the support from Entrena was essential for strengthening the organization: “The ARY Activity was like a push, it was like a school… [Entrena] taught us how they worked on everything, how they worked as a team, how to do things on time, to plan them to obtain better results. You would say ‘the ARY Activity demands too much and will not give you any slack’ but that taught us, their demand caused an impression on us, on the job, in every way….These situations helped us get out and have success….we have been able to strengthen different departments.” ~Sub-awardee staff member In the case of Alianza para el Servicio Comunitario (CSA), the technical assistance provided by Entrena could allow them to expand their reach in future programming: “The project helped us to think about methodology, to organize it and once we could organize it, multiply it in communities where we do not have ARY Activity support but, we were able to duplicate it there. We can really thank Entrena in this sense.” ~Sub-awardee staff member Among sub-awardees, those from well-established organizations and with international links profited from learning new skills and methodologies. In that sense, JADOM and Children International, for example, feel that they are now more capable of competing for external funding. Some organizations, such as Sur Futuro, have already reaped the benefits of participation in the ARY Activity. According to a Sur Futuro official, other institutions have already reached out to them for help implement similar projects. Entrena staff recognizes that project implementation requirements were a difficult task for most organizations but that it strengthened the organizations: “Most of the organizations are accustomed to working with projects but the ARY Activity represented a challenge, a real challenge, and that made them grow, all of them, and strengthened them, be it regarding the platform, the technical or administrative part, but they really came out stronger.” ~Entrena staff member For the sub-awardees, project requirements demanded preparing budgets, doing monthly projections and ensuring transparency and accountability in accordance with USAID regulations. Entrena and RTI trainers explained that through the ARY Activity, sub-awardees learned how to adjust activities to budget and measure results, receiving constant feedback from Entrena. While it was a tedious process for both Entrena and sub-awardee, financial reports, invoices, billing and documentation would be turned back to the implementer until they got it right. Despite the challenges for Entrena and for the sub-awardees, all are now better prepared for future programming. An Entrena interviewee opined: 44 “The project leaves behind a culture that will stay with them and I think that if at any moment they deviate from there, if they are going to do it wrong, they will have to think and say, ‘no, we are going to do it the correct way.’” ~Entrena interviewee CONCLUSIONS Managing a USD 20 million contract and training and coordinating the activities of 23 sub-awardees are not easy tasks. Yet, all sub-awardees interviewed are very grateful for the training and support provided by Entrena. Indeed, all ARY Activity sub-awardees benefitted from the experience by acquiring new technical and administrative skills. The institutional capacity of all organizations was strengthened, allowing them to operate more strategically and more efficiently in terms of time and resources. The knowledge and experience gained on working with at-risk youth should be valuable and provide the opportunity to implement similar tasks in future programming. Strengthening, and often creating, solid financial management procedures may be the greater contribution offered by Entrena to the sub-awardees. Strict supervision and monitoring by Entrena prepared these organizations to better understand the requirements of international donors, including USAID, on record keeping, technical and financial reporting, communication policies and more. The biggest risk to maintaining improved capacity is personnel turnover, which could cause these community-based organizations to lose the skills and knowledge acquired by specific members of their staff. In that sense, well-established organizations with more stable funding and lower turnover will have better probabilities of gaining new funding due to newly acquired skills. 45 VII. EVALUATION QUESTION 5: PERCEPTIONS HOW IS THE ARY ACTIVITY PERCEIVED AND VALUED BY THE BENEFICIARIES, THE COMMUNITY, AND OTHER RELATED PARTIES, INCLUDING PERTINENT NATIONAL AUTHORITIES? FINDINGS To answer this evaluation question, the team collected the results of the youth survey and the qualitative responses from youth beneficiaries at focus groups, as well as the input from more than thirty interviews with key stakeholders. BENEFICIARY PERCEPTIONS The vast majority of beneficiaries interviewed for this evaluation expressed a very favorable opinion of the ARY Activity. Across sex and age groups, survey respondents and focus group participants indicated that they were satisfied with the services received. When asked to what extent they agreed with the statement, “I am satisfied with the programs overall,” over 96% of survey respondents either strongly or somewhat agreed. Youth highlighted that the ARY Activity helped them raise their self-esteem and feel more confident in their ability to move ahead, as well as stay out of trouble and have a better life. For the beneficiaries, the youth themselves, gaining self-esteem and hope seems to be more valuable and more life-changing that even the numerous trainings received. Almost 90% of youth surveyed found the programs in which they participated to be very useful. Particularly useful was the information received on health, human behavior, and employment opportunities. The youth valued the quality and dedication of the staff and instructors, as well as the opportunity to obtain practical experience through internships. An indicator of the level of satisfaction with the services received among youth is that 84% of survey respondents showed a willingness to pay for services similar to those they had received.48 Youth valued highly the fact that someone had reached out to them and offered help. The ET had learned from health and education specialists that these vulnerable kids often lack a safe and loving family and that even community organizations feared them, and as a result, they became isolated and untouchable. Thus, when the ARY Activity reached out to them in the street corner or in schools, the youth saw in the project an alternative that brought hope to them. One of the young men at a focus group expressed his feelings as follows: “the ARY Activity, as the name says, is an alert, a warning, a project that wants to warn young people, to lead them, and to light their path so they will not fall in delinquency.” During focus groups discussions, youth expressed similar opinions on how the ARY Activity had helped them. It offered much needed support to “develop physically and mentally,” or to avoid pregnancies and sexually transmitted diseases. Participants went on to explain that the ARY Activity helped them stop “wasting time and getting into trouble.” They also valued the assistance received from the sub-awardees to obtain birth certificates and proper documentation, which is essential to staying in school and getting a job. Over 95% of youth surveyed indicated that they would be very or somewhat likely to recommend the ARY Activity to others (74% of them very likely). These results appeared consistent across sex and age groups. 48 Their willingness to pay must be tempered with the youth’s ability to pay, which in many cases is minimal. Sub-awardees had to subsidize transportation costs for beneficiaries to be able to attend trainings. 46 Less than 10% (44 of 590) of survey respondents found the project only somewhat useful or not useful. Of these, only eight provided a reason for their answers. These included not being interested in the subject, poor teaching, no opportunity to apply, not in-depth enough, and time was too short. Except for EPC, which lasted the full school year, youth found that most training courses were too short. COMMUNITY PERCEPTIONS Several key informants explained that the Dominican Republic had never seen an investment with the characteristics of the ARY Activity in terms of amount of funding, focus on at-risk youth, and methodologies utilized to implement the project. This specific focus has made an impression in the communities. Although RTI claims that it has found no clear knowledge of the ARY Activity within the communities, and that it is often confused with other projects and other institutions engaged in similar activities, including the Office of the First Lady and the Ministry of Education, the communities recognize that there are fewer unproductive youth and less delinquency, even they do not seem to associate that improvement with the results of the ARY Activity. Part of the reason the ARY Activity is not easily recognized in the communities, as Entrena explained, is that the services are provided by community-based organizations and those are the names that beneficiaries recognize and mention at home, not the ARY Activity. Entrena hired a staff member to support branding, but this did not happen until late in the program. Community leaders interviewed for this evaluation concur that the community saw the ARY Activity positively because of the changes seen in the behavior of the youth. They argue that the project had the capacity and ability to turn youth around. “The youth I know are project multipliers, they have changed a lot their attitudes as well as their habits and their outlook on life. I think they have a completely different outlook. I think that people who know them start comparing the ‘before’ and the ‘after’ and it’s totally different, you can tell from a mile away.” ~Community leader School principals and staff interviewed also perceive that the ARY Activity brought about a “totally unexpected benefit to teachers and students.” School authorities credit the project for helping them with improving discipline and behavior at schools. As a school staff told the evaluators, the ARY Activity gave the youngest children the chance to think that they could make it in life: “The children have decided to behave and dress well and that has taught me that [the ARY Activity] has achieved something at the base.” Another education specialist in a similarly vulnerable location described the community context that the ARY Activity encountered: very poor and uneducated families, unfamiliar with being punctual or organized and lacking a desire to improve in their life. Yet, the project offered youth much needed guidance, support and motivation. Judging the ARY Activity as a “very good initiative,” the school staff said: “Youth need guidance and support and to be motivated. While they might not all become entrepreneurs or set up a music group, as they would like, but if they are guided to see that this is the path they must thread, their future can be better.” ~School staff representative 47 During qualitative interviews, project implementers explained that the ARY Activity’s relations with the community frequently had a rough start and having access to households was not always easy. This was more acute in rural communities, particularly in the bateyes. A sub-awardee explained as follows: “Here the population is different, mobile, sometimes undocumented, and often afraid of the programs or simply because of their condition, they have to work and put money on the table and don’t give priority to education; and the family, since they didn’t have that opportunity, they don’t understand much about the importance of keeping [the kids] in school. Those were some obstacles.” ~Sub-awardee staff representative Gradually, the community realized the benefits of the ARY Activity for the youth and themselves. The ARY Activity implementers explained that to assist youth obtain proper documentation, for example, sub￾awardees had to work with the parents and often with the whole family. In that sense, the ARY Activity “went into the homes of the rural areas and that is why the project is so well regarded in the community,” an implementer explained. He goes on to say: “We wanted to make them part of the progress. It wasn’t just people that came from somewhere else to do something. It was that we allowed them the opportunity to get involved in this process to bring change to their own community.” ~Sub-awardee staff representative Several sub-awardees also pointed out that the ARY Activity helped minimized conflicts between communities. Evaluators were told that gang activities often created problems and fighting across neighborhoods and by bringing youth from different communities together, relationships improved. A sub￾awardee explained that: “Out of everything that could be done in the project, the youth were able to interact among one another; we had people who belonged to different communities, which sometimes had years’ worth of problems with each other, always fighting. Thanks to the intervention of the ARY Activity those fights were minimized. The ARY Activity arrived to be an ally, to create ties between the communities in Boca Chica.” ~Sub-awardee staff representative Finally, community leaders argue that one major contribution of the ARY Activity is that it helped “create awareness on educators and teachers on how to avoid violence, how to provide support to a teenager who is possibly being abused, how to support the family and guide them to be more attentive of their children.” The community welcomed the changes. 48 NATIONAL AUTHORITY PERCEPTIONS The ARY Activity could not maintain steady and continuous relations with national officials during the first three years of implementation due to changes in government administrations. In May 2016, general and congressional elections were held and although, President Danilo Medina was reelected, several cabinet ministries and high level officials were changed. According to Entrena, these changes represented some delays and required re-building ties with new personnel at several ministries. Some interviewees, however, alleged that these obstacles could have been avoided if the ARY Activity had created formal alliances with the different ministries before initiating the project, rather than on the march. Yet, there have been some significant accomplishments. The Ministry of Youth, responsible for implementing official youth policies, should have been the ideal partner for the ARY Activity. Yet, Entrena found that with the exception the Santiago office, the Ministry of Youth was institutionally weak and politicized.49 While some collaboration has been achieved, the official of the Ministry interviewed for this report was not fully familiar with the ARY Activity. The Ministry of Economic Planning and Development (MEPyD) became a stronger ally for the ARY Activity. According to Entrena, the MEPyD maintains “constant communication” with the ARY Activity. The MEPyD was responsible for coordinating the assistance provided by the Chilean Agency for International Cooperation for implementing the training on “Entrepreneurship and Employability” offered to at-risk youth through the Chilean Social Investment and Solidarity Fund—Fondo de Solidaridad e Inversión Social de Chile—FOSIS). This program trained certified instructors from INFOTEP, who then trained at-risk youth participants. The MEPyD argues that the ARY Activity has influenced how the government looks at its own initiatives: “Before the ARY Activity there was no talk of vulnerable youth or that these young people must be inserted into the labor market…Now you can talk about any subject touched by the ARY Activity more fluently. We can see how much it costs the State to sponsor a person in conflict with the law while detained and compare it to the costs of inserting the same youth into the labor market, to keep them off the street. Then, you say: …‘why didn’t they do it before if it costs 3 times as much to keep them in custody?’” ~Government representative The ARY Activity documents and interviews indicate that the project has also worked with the Ministry of Education, the Ministry of Labor, the Ministry of Public Health, the Ministry of Industry and Commerce and with the National Technical and Professional Training Institute (INFOTEP). It has also worked with the Office of the Attorney General with whom it signed an agreement prior to the general elections. Relations with the new Attorney General have been developing rapidly and this office is in the process of creating youth centers based on the ARY Activity model. This indicates that the Office of the Attorney General, like the other Ministries, find the results of the ARY Activity effective for the needs of the country in relation to youth. The ARY Activity has worked closely with INFOTEP, which provides the bulk of the technical and entrepreneurial trainings to ARY Activity beneficiaries through the sub-awardees. INFOTEP is very 49 Entrena SRL, “Dominican Republic At-Risk Youth Activity: Alerta Joven,” ANNUAL Report 2014-2015, p. 19. 49 enthusiastic about the ARY Activity. Referring to the ARY Activity as a “fantastic project that has done a great job, and is much needed,” INFOTEP officials believe that: “It was a project that in a short span of time rendered good results, quick and measurable results; the way it has worked was very transparent, very honest, there was never a situation where I, nor the institution, could feel uncomfortable. It shows [us] that [international donor] contributions are manageable in a short time span, that with few resources you can much. Resources were put into good use.” ~Government representative Gathering from interviews both with INFOTEP and sub-awardees, the collaboration between the agency and ARY Activity implementers was successful and productive. INFOTEP appreciated “the sensibility of the people of the United States for supporting an education program for young people at-risk in our country” and sub-awardees perceived that the trainings offered by INFOTEP to beneficiaries made a difference in their ability to find jobs. PRIVATE SECTOR PERCEPTIONS The ET interviewed five business representatives working with the ARY Activity through different sub￾awardees. From these interviews and those with the sub-awardees, the team gathered that the support offered by the business community to the ARY Activity fluctuates from very supportive to supportive, but not cooperating. Business owners interviewed seem to appreciate that youth graduating from the ARY Activity have personal skills that many vulnerable young people lack, that they have learned to be responsible and punctual, and are being supervised by the sub-awardee while doing an internship. Though the ARY Activity vocational training was designed to address specific demands in the private sector for specific jobs and skills, the targeted positions were generally low-skilled positions, and beneficiaries often joined a crowded field of available labor. Entrepreneurs find that the ARY Activity youth lack high levels skills that would make them more competitive within a high unemployment market. Nonetheless, the private sector representatives interviewed for this report viewed the project favorably for the opportunity it offered to young people. As one entrepreneur explained, “the ARY Activity is rescuing those youth from the streets and providing them skills and training in human behavior.” Small businesses view the ARY Activity favorably in part because it provides them a service not available in poor vulnerable neighborhoods. Business owners interviewed explained that they ask community organizers or sub-awardees that they trust to recommend interns or potential employees. One such business owner in Boca Chica explained to the team that the ARY Activity is “like a job agency: one says ‘send me this type of youth’ and that is how it works.” The familiarity between sub-awardee and business community has facilitated granting internships to many youth and a stable flow of labor to small businesses in poor neighborhoods around Santo Domingo. Middle-sized businesses, those with more than 100 employees, usually require a minimum of one year of experience for new employees. Sub-awardees explained to the team that some businesses have made an exception for the ARY Activity youth and hired them first as interns and then as regular employees. Such is the case of one of our key informants, who has hired three ARY Activity beneficiaries. He explained that ARY Activity participants are well prepared, particularly skillful in human relations, and that they have 50 shown great commitment to their jobs. For this private sector representative, the ARY Activity “is an excellent program that in one way or another, takes the youth off the streets.” Sub-awardees and business leaders told the ET that for larger businesses, the ARY Activity-trained youth do not represent an advantage. These enterprises demand more experience and higher-level skills than those acquired by ARY Activity beneficiaries. As a result, very few large companies have hired ARY Activity graduates. CONCLUSIONS In general terms, the perception among all ARY Activity stakeholders has been extraordinarily positive. The view that the project has brought significant opportunities for youth to change the course of their lives appears as the strongest and most consistent one, practically unanimous, among all interviewees. Changing youth also changes communities, even if many of the communities do not realize that this is the results of the ARY Activity. While crime and violence affects families and communities as a whole, supporting youth or preventing their involvement in illegal activities or irresponsible early parenting also impacts the family and the community. Unfortunately, the extent of the youth vulnerability is so broad in Dominican Republic that the results of the ARY Activity within the national context are practically minute. While the ARY Activity has brought national attention to the plight of vulnerable children in the Dominican Republic, the government has missed the opportunity to be more engaged in its initiatives and to commit itself to assume the responsibilities now filled by international donors. This lack of commitment on the part of the government does not augur well for the future of vulnerable youth in the Dominican Republic. Considering the high unemployment in the Dominican Republic, offering employment opportunities to at￾risk youth is very difficult. The ARY Activity had been successful in forging public-private partnerships, and these should be further developed. 51 VIII. RECOMMENDATIONS In this section, the ET provides several recommendations to enhance ARY Activity programming during the limited time left to conclude its implementation and as considerations for future programming. The recommendations are addressed to USAID and Entrena, to the Dominican national and local authorities, and on specific project areas. Also included are specific recommendations for sustainability. FOR USAID AND ENTRENA IMPROVE TARGETING OF THE AT-RISK YOUTH INITIATIVE PROGRAM For future programming, some organizational changes could contribute to making the project more focused and manageable. ▪ IMPROVE GEOGRAPHICAL TARGETING TO BETTER SERVE THE BENEFICIARIES, EASING PROJECT MANAGEMENT FOR USAID AND IMPLEMENTING PARTNERS. To improve targeting, clear and consistent criteria for the selection of municipalities should be used. Such criteria should include: level of crime and violence, number of at-risk youth in the community, and other defining factors. ▪ THE NUMBER AND SELECTION OF SUB-AWARDEES SHOULD BE BETTER TARGETED. This targeting would make implementation more manageable and consistent, given the human and financial resources available. In selecting potential partners, USAID should pay strong attention to the current level of capacity for implementing all the desired programs, and, in cases where there might be capacity constraints, appropriate plans for building the necessary capacity should be anticipated. ▪ IMPROVE COORDINATION AND IMPLEMENTATION ACROSS PARTNER ORGANIZATIONS. This could include facilitated meetings and workshops with all implementers, more consistent implementation guidelines for all program types, and additional support to help organizations retain staff trained by the ARY Activity. Increased collaboration between implementers will both facilitate consistency across partners and also allow for improved collaborative problem solving and sharing of lessons learned. IMPROVE THE MONITORING SYSTEM The evaluation’s quantitative team had the opportunity to use the ARY Activity’s tracking system, with mixed results. As noted in the limitations section of this report, the data collection firm had significant difficulties with the accuracy of the data included in the system. In particular, the tracking system had difficulties with beneficiaries that were very mobile, changed phone numbers frequently and may have dropped out of school. Given the number of beneficiaries who participated in the ARY Activity’s programs, significant upgrades to the system and keeping it up to date could take substantial financial and human resources, however. Thus, USAID will need to balance the benefits of improved beneficiary tracking with the costs necessary to develop implement, and update a system that explicitly has benefits for the organization even after USAID funding ends. Regardless of the solution selected, however, USAID should anticipate the effect of the accuracy (or lack thereof) of this information on follow-up activities and evaluation efforts. 52 FOR THE DOMINICAN NATIONAL AND LOCAL AUTHORITIES CONTINUE TO MAKE IMPROVEMENTS, INCLUDING INFRASTRUCTURE, TO REDUCE VIOLENCE IN COMMUNITIES The evaluation encountered difficulties in estimating the effect of the ARY Activity on reducing crime and violence in the communities. In part, this was due to the low number of respondents who admitted participating in criminal and violent behaviors. However, it is also due to the reality that the ARY Activity beneficiaries are not the only potential source of criminal and violent behavior within communities. When asked about crime in their communities, community leaders all talked about the deteriorating conditions around them. Considering the factors that contribute to creating conditions of risk for youth, USAID and the USG government should discuss with Dominican national and local authorities the need to reduce those conditions by seeking to relieve overcrowding of school classes, improve infrastructure in the most vulnerable communities. Though limited resources are often cited as reasons for not providing more resources to these efforts, there are international partners with whom Dominican authorities could collaborate, including the Inter￾American Bank and the Central American Bank. DEFINE A COHERENT PUBLIC POLICY FOR AT-RISK YOUTH AND ESTABLISH COORDINATION AMONG INSTITUTIONS IMPLEMENTING IT. At present, initiatives to support at-risk youth appear to be implementing ad hoc. In that sense, they borrow from the ARY Activity in carrying out some activities. However, the ARY Activity’s results could be maximized by coordinating closely with the GoDR to draft an integrated strategy for at-risk youth outreach and programming. THEMATIC RECOMMENDATIONS EDUCATION ▪ CONTINUE THE TRAINING AND INCORPORATION OF EPC METHODS IN SCHOOLS. The EPC methods, the methodology of Quantum Learning, and the Advantage of Staying in School workshops are highly regarded by all ARY Activity stakeholders. The team recommends continuing and expanding the training for school teachers on these methodologies. ▪ CONTINUE SUPPORTING FAMILY ENGAGEMENT IN CHILDREN’S EDUCATION. Essential for school retention and reinsertion is family commitment and engagement. Educational programs should develop and/or expand tools to work with families to promote youth enrollment and educational achievement. An integrated methodology to reach youth who have left the educational spaces should also be emphasized. ▪ INCREASE SUPPORT FOR EXTRACURRICULAR ACTIVITIES. Youth’s opportunities for misbehaving increase with the amount of idle time they have, which is of particular concern among those attending school half day and/or who are unemployed. Future programs could consider allocating budget to provide extra-curricular activities and create more spaces of diversion to prevent drug and alcohol use. Beneficiaries indicated interest in art, music, dance and singing classes, sports, languages, and other courses that could help them develop their talents. 53 HEALTH ▪ CONTINUE THE ARY ACTIVITY HEALTH INTERVENTIONS AND EXPAND TO INCLUDE SEX WORKERS. Services provided through PEPFAR and the ARY Activity health vouchers offered youth valuable information and opportunities for improved health. These services would be highly recommended for future funding, as well as programs aimed at reducing teen pregnancy. The ARY Activity should also create a specific program to reach commercial sex workers as they are a particularly vulnerable group that is a target of the ARY Activity, but has been historically underrepresented. ▪ PROVIDE PSYCHOLOGICAL ASSISTANCE TO YOUTH AND FAMILY. Working with at￾risk youth and their families requires psychological assistance that should be included in any future programming, including psychological services for witnesses of domestic violence, intimate partner violence, and preventing gender-based violence. ▪ BE MORE PROACTIVE IN REACHING YOUTH ON HEALTH ISSUES. Sub-awardees consider that to reach youth in schools, implementers need to be more proactive and take the medical teams to the schools to carry out tests there rather than wait for them to come to a workshop. EMPLOYMENT OPPORTUNITIES ▪ PROVIDE HIGHER-LEVEL TRAINING. Consider offering higher level training that places ARY Activity beneficiaries above the large pool of young unemployed Dominicans. Such training could include management skills and technical skills in the medical and financial domains. ▪ ADDITIONAL INTERNSHIP OPPORTUNITIES. Beneficiaries that completed internships gained skills and experience that prepared them for full time employment. Additional options include: - The ARY Activity could also influence companies to include the employment of vulnerable youth within philosophies of corporate responsibility - Create more opportunities for companies to interact with youth through job fairs. While job fairs were not a major source of employment opportunities, they served to get employers familiar with ARY Activity beneficiaries - Request implementing partners to hire interns for USAID projects - Encourage national institutions, such as Congress, ministries, local governments to offer internships to ARY Activity beneficiaries, where they could learn new skills and experience, Internships in Congress and national institutions would contribute to developing skills in advocacy and public policy ▪ SELF-EMPLOYMENT: EXPAND FUNDING OPPORTUNITIES AND CREATE NETWORKS. Dreaming of owning a business contributed to a renewed sense of future among ARY Activity beneficiaries. The reality, however, was that they lacked the funding and/or credit to carry out their plans. This challenge requires creative solutions and working with local banking institutions to develop a loan product for young entrepreneurs. Program models that have been successfully implemented elsewhere, such as the Caribbean Development Bank’s VYBZING program, can be incorporated into future programming to increase the availability of funding for young entrepreneurs. And, to ensure that young entrepreneurs continue to receive support, the project should create networks of entrepreneurs that can provide monitoring and support. 54 SUSTAINABILITY ▪ PLANNING FOR SUSTAINABILITY MUST START AT THE BEGINNING. Initiatives like the ARY Activity cannot be sustained solely by international donors. The concern for at-risk youth is a primary responsibility for national and local authorities. Despite the generous support of the American people, the project must have domestic ownership in order to be truly sustainable. ▪ PROMOTE DOMINICAN OWNERSHIP OF THE PROJECT. Dominican society as a whole must become familiar with the at-risk youth issues and learn how investing in prevention could directly impact their own neighborhoods and wellbeing. The ET recommends USAID invest in a massive education and public awareness campaign to bring to light these vulnerable neighborhoods and youth with the purpose of creating conditions for Dominican society to put pressure on its government to attend to these youth. Dominican society needs to own the problem and demand solutions. ▪ CREATE FUNDING CONSORTIUM. Future programming should create a consortium between the Dominican national and local authorities and large and medium businesses in a public-private partnership that could gradually take the project over from USAID. However, for it to be successful, this sustainability plan would need to be a central focus of the project from the very beginning. Indeed, it would be preferable for there to be not only a notional commitment of support, but actual financial support from the Dominican government funding from the start, with a clear path for takeover by the end of the program. In order to be truly sustainable, the Dominican government, society, and private sector must take ownership of this very valuable and needed initiative. 55 IX. DISSEMINATION AND UTILIZATION PLAN Utilization is a key focus of this evaluation, and as such, throughout the contract, the SI team has engaged with USAID/DR on how the results from the evaluation can be best used to inform improvements to the current project as well as updates to future program design. As mentioned during the kickoff meeting, while the ARY Activity is currently in its fourth year, there is still time to introduce some changes to the program as well as more generally to contribute to the body of knowledge of whether early stage education interventions are effective at reducing criminal activity. USAID/DR’s focus on youth programming to reduce crime and violence is likely to continue beyond the scope of the ARY Activity, which makes the findings of this evaluation useful to inform future policy in addition to project level improvements. There are two primary methods of dissemination currently planned with this contract. The first comes in the form of stakeholder feedback. At each stage of the evaluation, key stakeholders have been and will continue to be asked to provide feedback on the design and results from this evaluation. Throughout the design process, SI held meetings with Entrena and USAID/DR and shared the evaluation design/methodology with them to solicit their feedback. That feedback was instrumental in the final design of the evaluation, particularly in the handling of baseline data, and was incorporated into the final approach. Additionally, following submission of this draft report, the SI team will develop a power point presentation on the findings from the evaluation. The current budget does not provide for a return visit to the Dominican Republic to present findings, however, SI plans to hold a teleconference presentation with USAID/DR. The final evaluation report, along with data files, will be uploaded to the Development Experience Clearing House by Social Impact within one week of receiving USAID approval of the final report. 56 U.S. Agency for International Development/ Dominican Republic Avenida República de Colombia #57 Altos de Arroyo Hondo Santo Domingo, Dominican Republic 57 ANNEXES ANNEX A. EVALUATION SCOPE OF WORK SECTION C - STATEMENT OF WORK I. PURPOSE OF THE EVALUATION. The purpose of this task order is to independently conduct a mid-term performance evaluation of USAID/DR’s five-year 19.9 million dollar At-Risk Youth Initiative [Alerta Joven Project (2012-2017)], implemented by ENTRENA, SRL (ENTRENA). The contractor will use project data collected by ENTRENA and Research Triangle International (RTI) through the monitoring and evaluation system in place for the USAID/DR At-Risk Youth Initiative (Alerta Joven) to produce an independent mid-term performance evaluation report. The evaluation responds to the USAID Evaluation Policy released in January 2011. This policy can be reviewed at http://www.usaid.gov/evaluation. The Contractor must be familiar with and comply with the policy requirements, as appropriate, as well as with the definitions included in it, such as the one outlined below: “Performance evaluations focus on descriptive and normative questions: what a particular project or program has achieved (either at an intermediate point in execution or at the conclusion of an implementation period); how it is being implemented; how it is perceived and valued; whether expected results are occurring; and other questions that are pertinent to program design, management and operational decision making. Performance evaluations often incorporate before/after comparisons, but generally lack a rigorously defined counterfactual.” The evaluation will contribute to knowledge and learning at USAID regarding the effectiveness of “Alerta Joven”, especially in successfully providing a second-chance for youth in the targeted areas to be impacted. Based on the source of funds it will also be important to gauge the program’s effectiveness in shifting the minds of youth from negative behavior to a positive one and which will assist in reducing the allure to a life of crime and violence. The performance evaluation study will be closely coordinated with USAID/DR and ENTRENA. This evaluation will be informative and will help the Mission determine which intended results are being met or have been met, and specifically, the effectiveness of implementation, lessons learned and best practices that can be used to take any corrective course of action during the remaining implementation period of “Alerta Joven” and improve future at-risk youth’s project design. II. BACKGROUND. Over the past two decades, USAID/DR has played an important role in supporting programming to address the needs of the Dominican Republic’s at-risk youth. USAID provides support for at-risk youth 58 throughout its portfolio including the areas of education, health and employability. USAID/DR provides support in collaboration with other USG agencies, donors and international organizations. USAID/DR’s Development Objective 1(DO 1): Crime Prevention Strengthened under current Mission Strategy ending 2018. “Alerta Joven” contributes to USAID/DR’s DO 1, Crime Prevention Strengthened, through the DO’s Intermediate Result 1.1 (IR 1.1) Youth Involvement in Criminal Activities. The goal of the Dominican Republic At-Risk Youth Initiative ”Alerta Joven” is to create a sustainable, multidisciplinary safety net for at-risk youth with the intended result of preventing crime. The project builds institutional capacity of local youth-service providers and provider networks/consortia, while simultaneously increasing youth engagement to craft solutions that will help their own generation. Collaborative partnerships among government entities, civil society, youth, the private sector, and donors are promoted as part of project sustainability efforts. Alerta Joven’s goal is expected to be achieved through the following three specific intermediate results (IRs):  IR 1: Youth protection policy improved in practice. o IR 1.1 Youth stakeholders empowered to influence policy decisions. o IR 1.2 Implementing agencies understand and comply with youth legal framework. o IR 1.3 Documented children and youth increased.  IR 2: Youth opportunities to be productive citizen improved. o IR 2.1 Formal and alternative education opportunities increased. o IR 2.2 Access to quality health prevention and care services improved. o IR 2.3 Employment for youth increased.  IR 3: Integrated youth system developed. o IR 3.1 A comprehensive information system implemented. o IR 3.2 Multidisciplinary working groups effectively coordinating and implementing innovative solutions to reduce crime and gang activities. o IR 3.3 Public Private Partnerships enhanced to support and advocate for vulnerable children and youth. Four critical assumptions have been considered for the achievement of project goal: 1. Government of the Dominican Republic (GODR) continues to welcome USAID assistance. 2. GODR is committed to funding and implementing youth policy reform. 3. USAID funding is maintained or increased. 4. GODR and local institutions will prioritize efforts for youth development and reduction of youth participation in violence and crime. 59 It is expected that the long-term benefits of this youth project will result in more positive youth participation in society. Poverty and lack of employment opportunities will be reduced with a resultant decrease in rates of crime and delinquency. In turn the program will address the additional barriers such as lack of proper identification documents and corruption that limit youth involvement. Recent studies indicate that youth want a sense of purposefulness, an opportunity to be involved in the processes that affect them. The program will inspire youth and equip them with the ability to influence policy and prevent crime not only on “youth issues” but also in areas that are broader in scope and include societal concerns. III. USAID/DR AT-RISK YOUTH INITIATIVE “ALERTA JOVEN” PROJECT. i. Overview. This five-year and almost 20 million dollar project, funded by the Caribbean Basin Security Initiative (CBSI) and the President’s Emergency Plan for AIDS Relief (PEPFAR), is expected to innovatively support Dominican government and civil society to strengthen or develop systems to ensure youths’ health, safety, and success in school, work, family, and community, especially for out-of-school and at-risk youth aged 11 - 24. The project integrates interventions in empowerment, public action, legal and policy issues, documentation, productive citizenship, information systems, and sustainability to develop a powerful initiative that provides practical solutions to the challenges Dominican society faces today with at-risk youth. The interventions focus in the corridor between Santo Domingo and Puerto Plata, also known as the northern corridor, where 70% of the population and the majority of the at risk youth population live. Other intervention areas are Haina and Boca Chica, near Santo Domingo, and San Francisco de Macorís, in the Cibao valley. “Alerta Joven” provides interventions in school retention, insertion, and completion; technical-vocational and entrepreneurship skills training; HIV/AIDS prevention and attention services; identity documentation (including Haitians in the DR); and youth policy and civic participation. “Alerta Joven” focuses on vulnerable young people who are at-risk of or have left school, committed offenses against the law, or become involved in gangs. It also targets teenage mothers, HIV-positive youth, and other vulnerable groups. “Alerta Joven” also strengthens a youth protection network using a sustainable multidisciplinary approach to empower at-risk youth to be the authors of their own improvement. It works closely with government agencies, the private sector, non-governmental organizations, and other civil society partners regarding actions that will help achieve project sustainability and important leveraging of resources. During the life of the project 85,000 direct beneficiaries are expected to be reached distributed as follows: 51,000 beneficiaries, ages 11-17, 34,000 beneficiaries, ages 18-24. In July 2012 Cooperative Agreement number AID-517-A-12-00002 was signed between USAID/DR and ENTRENA, in the amount of $19,972,130, for the implementation of this initiative, a multidisciplinary project working in partnership with the GODR and civil society. ENTRENA, a Dominican private firm, has worked since 1982 in projects related to Peace Corps, USAID, baseball industry and education for the Dominican private sector, GODR and universities. 60 This initiative is being implemented in partnership with RTI International, Instituto Dominicano de Desarrollo (IDDI), and, Asociación Dominicana Pro-Bienestar de la Familia (PROFAMILIA). Other local organizations have access to the sub-grants fund managed by ENTRENA and generally received through open competition, join “Alerta Joven” as implementing sub-grantees and youth service providers all over the geographic areas of intervention. IV. STATEMENT OF WORK. USAID/DR requires a mid-term performance evaluation to be conducted on its At-Risk Youth Initiative “Alerta Joven” project described herein. i. Objective of the Evaluation & Key Questions. The evaluation will contribute to increase knowledge and learning in USAID/DR’s Mission Management and ENTRENA, and will provide them with: (a) An improved understanding of what works best to ensure a successful “Alerta Joven” project programming; (b) Information on the effectiveness of this at-risk youth crime prevention different interventions; and (c) Guidance on future project and activity design, and implementation. The results of the evaluation will also assist partners as well as local implementing entities to strengthen their project interventions going forward. The evaluation is expected to analyze and synthesize existing project data to support the achievement of the objective and shall include, but not limited to, the following evaluation questions: (a) What is the likelihood that “Alerta Joven” youth beneficiaries will: 1) return and stay in school, 2) reduce risky behaviors, 3) obtain and maintain jobs, 4) start their own business. (b) Which major factors have contributed so far to the achievement (or failure to achieve) of the results through each one of the project interventions by the end of the third year of implementation and to what extent have public-private partnerships contributed or not to results achieved? What are the essential components for the workforce development program under “Alerta Joven” to be successful? (c) Are the services offered to youth by ENTRENA and sub-awardees sustainable, in terms of financial resources and institutional capacity built under the project and potential local uptake? 61 (d) To what extent has the capacity building training offered by ENTRENA helped sub-awardees to improve their technical and financial management capabilities which could help them access other sources of funding once “Alerta Joven” is over? (e) How is “Alerta Joven” perceived and valued by the beneficiaries, the community and other related parties including pertinent national authorities? For each one of the questions above and regarding individual and beneficiary information, sex disaggregated data should be analyzed and findings should reflect any gaps and differences in the results as well as the outcome of the project. Gender issues approached through project interventions are to create a platform for behavioral change among at-risk youth. The contractor shall propose the most appropriate methodologies and implementation plan to address the evaluation questions. The proposal should clearly describe the proposed data collection and analysis methods and a clear justification for the proposed methodology. Based on the results of the mid-term evaluation, the contractor shall provide programmatic directions and prioritized recommendations, which along with the results of USAID regular monitoring of the project, will contribute to the implementation of pertinent corrective actions to be taken by USAID/DR and ENTRENA for the remaining project period of implementation and will assist USAID/DR in the future design of vulnerable youth-focused programs. ii. Sources of Information. The Evaluation team will review relevant documentation regarding the USAID/DR At-risk Youth Initiative “Alerta Joven”, provided by USAID/DR. Documents for review may include, but will not be limited to the following:  USAID Evaluation Policy of January 2011  CBSI Guidance  PEPFAR Guidance  USAID/DR Strategy, 2014-2018  Cooperative Agreement between USAID/DR and ENTRENA  Monitoring and Evaluation Plan from ENTRENA, primary implementer  Yearly Work Plans  Project Quarterly and Annual Reports from ENTRENA  USAID/DR Cross-Sectoral At-Risk Youth Assessment performed by Education Development Center, Inc. (EDC) in August 2010. USAID/DR will assist the Evaluation Team as needed. At or shortly after the team planning meeting, the team will confer with USAID Mission personnel to discuss evaluation planning prior to arrival in the field, and will present its proposed methodology to USAID in accordance with Section iv below. 62 iii. Evaluation Plan. The Evaluation Team shall develop and submit for USAID/DR’s approval an evaluation plan prior to the start of field work. This plan shall include: the key evaluation questions; methods; main features of data collection instruments; and data analysis plans. The plan will be used to guide the process of the evaluation. In accordance with USAID’s policy, the evaluation plan must be shared with stakeholders in the countries prior to being finalized to ensure quality of the plan and the inputs of these external clients. The Contractor will be responsible for this action which will be supported by the appropriate introductory letters from USAID/DR as necessary. The plan must include:  Evaluation objective  Identification of evaluation research questions and sub-questions  Justification of proposed methodology  Sampling methods  Implications for program activities and their sequencing and timing  Data collection and analysis methods (quantitative and qualitative)  Justification for any additional necessary data collection and timing of proposed data collection  Process of data collection quality assessments and management  Key implementation challenges and risks  Estimated Cost  Dissemination Plan for evaluation findings  Data collection instruments that are available should be included in the annex iv. Evaluation Methodology. The Evaluation Team should consider starting its work with a review of documents cited above. There is flexibility in the methodology used for the evaluation and therefore the Evaluation Team is encouraged to develop a design that is appropriate to answer the evaluation questions. It is expected that the Evaluation Team will conduct key interviews with USAID, ENTRENA and implementing partners and sub-grantees, project beneficiaries and public sector partners in order to satisfy the objectives of the evaluation. Data must be disaggregated, for example by gender, to identify how project inputs are benefiting various groupings of youth. USAID suggests a mixed research approach, including the following evaluation methods which are illustrative:  Key informant interviews  Focus groups  Mini-surveys  Personal narratives  Secondary data collection (document and data review); and  Observations through field visits to local implementation sites. v. Evaluation Data Analysis. The evaluation team is expected to provide a plan for data analysis and for review and obtain approval by USAID prior to its commencement. This plan must directly address each evaluation question with specific 63 methods for collecting and for analyzing the data that will be used to answer it. It should also provide details on how focus group interviews for example, will be transcribed and analyzed; what procedures will be used to analyze qualitative data from key informant and other stakeholder interviews; and how the evaluation will weigh and integrate qualitative data from these sources with quantitative data to reach conclusions about the effectiveness and efficiency of the program. The information collected shall be analyzed by the Evaluation Team to identify correlations and establish what are the major trends and issues. Data must be disaggregated for example by gender to identify how project inputs are benefiting various groupings of youth. V. SPECIFIC TASKS (a) Evaluation Plan. Within ten (10) days of contract award, the team will prepare a work plan, to include a data analysis plan outlining the methodologies to be used in the evaluation. The work plan will be submitted to the TOCOR prior to arrival in Dominican Republic. (b) USAID Debriefing. Upon completion of the field work, the team will conduct a debriefing to present the major findings of the evaluation to USAID through a power point presentation, for discussion and to receive initial feedback. This debriefing will include a discussion of findings and recommendations and shall occur prior to submission of the draft report. (c) Implementing Partners debriefing. After (b) above is completed, a debriefing shall be conducted with the project’s Implementing partner prior to departure from Dominican Republic for discussion and to receive feedback. The debriefing will include a discussion of findings and recommendations. (d) Draft Evaluation Report. All evaluation findings, conclusions and recommendations shall be documented in a draft report, to be presented to USAID/DR within seven (7) weeks of completing in￾country work. USAID/DR feedback will be provided within 10 working days. (e) Final Evaluation Report. The final evaluation report, in print and electronic format, shall be presented to USAID within three (3) weeks of receiving USAID’s feedback on the draft report. The Evaluation Team shall incorporate or address comments provided on the draft report and submit an electronic PDF and Word versions to the TOCOR. The final report will include a table of contents, executive summary (3-5 pages), methodology, findings, and recommendations. The document shall include in annex form at a minimum the following: evaluation scope of work; all evaluation tools; all sources of information/data; and statement of differences if applicable. All data and records should be provided to the TOCOR in an organized, electronic format. A second version of the report excluding any potentially procurement sensitive information will be submitted for dissemination among implementing partners and stakeholders. The final evaluation report must be submitted in English and Spanish. END OF SECTION 64 ANNEX B. SCHEDULE OF ACTIVITIES AND INTERVIEWS Date Method Sub Awardee Tool Interviewee Position City Location # October 14, 2016 Qual. KII´s Interview Businessman Santo Domingo Sub awardee´s center 1 KII´s Interview Project Coordinator Santo Domingo Sub awardee´s center 1 KII´s Interview Health Coordinator Santo Domingo Sub awardee´s center 1 KII´s Interview School Principal Santo Domingo School 1 KII´s Interview Center Director Santo Domingo Sub awardee´s center 1 October 15, 2016 Qual. KII´s Interview Project Manager Santo Domingo Sub awardee´s center 1 KII´s Interview Project Coordinator Santo Domingo Sub awardee´s center 1 October 17, 2016 Qual. KII´s Interview Program Promoter Santo Domingo Sub awardee´s center 1 KII´s Interview Community Leader Santo Domingo Sub awardee´s center 1 KII´s Interview Educator Santo Domingo Sub awardee´s center 1 KII´s Interview Project Coordinator Santo Domingo Sub awardee´s center 1 October 18, 2016 Qual. KII´s Interview Representatives Santo Domingo Sub awardee´s center 1 65 KII´s Interview School Principal Santo Domingo School 1 KII´s Interview Project Coordinator Santo Domingo Sub awardee´s center 1 KII´s Interview Businessman Santo Domingo Sub awardee´s center 1 KII´s Interview Center Director Santo Domingo Sub awardee´s center 1 KII´s Interview Center Director Santo Domingo Sub awardee´s center 1 KII´s Interview Supervisor Santo Domingo Sub awardee´s center 1 October 19, 2016 Cuant. IDDI Youth Survey Interviews - - Santo Domingo Sub awardee´s center 5 Qual. KII´s Interview Project Coordinator Boca Chica Sub awardee´s center 1 KII´s Interview Community Leader Boca Chica Sub awardee´s center 1 KII´s Interview Businessman Boca Chica Sub awardee´s center 1 KII´s Interview Educator Boca Chica Sub awardee´s center 1 KII´s Interview Project Coordinator Santo Domingo Sub awardee´s center 1 October 20, 2016 Cuant. CSA Youth Survey Interviews - - Santo Domingo Sub awardee´s center 6 Children Int. BC Youth Survey Interviews - - Boca Chica Sub awardee´s center 1 4 66 Fundación Pediátrica por un Mañana Youth Survey Interviews - - Santo Domingo Sub awardee´s center 7 IDDI Youth Survey Interviews - - Santo Domingo Sub awardee´s center 1 Children Int. BC Youth Survey Interviews - - Boca Chica School 1 1 Qual. Children Int. SD FGDs Female youths - Santo Domingo Sub awardee´s center 6 Children Int. SD FGDs Male youths - Santo Domingo Sub awardee´s center 5 Project Hope FGDs Female youths - Santo Domingo Sub awardee´s center 4 KII´s Interview Project Coordinator Santo Domingo Sub awardee´s center 1 KII´s Interview Center Director Santo Domingo Sub awardee´s center 1 KII´s Interview School Principal Santo Domingo School 1 KII´s Interview Community Leader Santo Domingo Sub awardee´s center 1 KII´s Interview School Insertion Coordinator Santo Domingo School 1 October 21, 2016 Cuant. IDDI Youth Survey Interviews - - Santo Domingo Sub awardee´s center 3 Qual. Children Int. BC FGDs Male youths - Boca Chica Sub awardee´s center 4 67 Children Int. BC FGDs Female youths - Boca Chica Sub awardee´s center 4 KII´s Interview Project Coordinator Santo Domingo Sub awardee´s center 1 KII´s Interview Health Coordinator Santo Domingo Sub awardee´s center 1 KII´s Interview School Principal Santo Domingo Sub awardee´s center 1 KII´s Interview Community Leader Santo Domingo Sub awardee´s center 1 KII´s Interview Center Director Santo Domingo Sub awardee´s center 1 KII´s Interview Businessman Santo Domingo Sub awardee´s center 1 KII´s Interview Center Director Santo Domingo Sub awardee´s center 1 KII´s Interview Project Coordinator Santo Domingo Sub awardee´s center 1 KII´s Interview Health / Community Leader Santo Domingo Sub awardee´s center 1 KII´s Interview Community Leader Santo Domingo Sub awardee´s center 1 KII´s Interview Health Coordinator Santo Domingo Sub awardee´s center 1 KII´s Interview School Principal Santo Domingo School 1 October 22, 2016 Cuant. ACOPRO Youth Survey Interviews - - Santo Domingo Community 3 68 Project Hope Youth Survey Interviews - - Santo Domingo Community 2 October 24, 2016 Cuant. Children Int. BC Youth Survey Interviews - - Boca Chica Sub awardee´s center 10 Children Int. BC Youth Survey Interviews - - Boca Chica School 11 Qual. - KII´s Interview (Coop. International ) Santo Domingo Sub awardee´s center 1 - KII´s Interview (EDUCA) Santo Domingo Sub awardee´s center 1 - KII´s Interview (RTI) Santo Domingo Sub awardee´s center 1 - KII´s Interview (Entrena) Santo Domingo Sub awardee´s center 1 - KII´s Interview (Entrena) Santo Domingo Sub awardee´s center 1 - KII´s Interview (Entrena) Santo Domingo Sub awardee´s center 1 - KII´s Interview (Entrena) Santo Domingo Sub awardee´s center 1 - KII´s Interview (Entrena) Santo Domingo Sub awardee´s center 1 October 25, 2016 Cuant. Caminante Proyecto Educativo Youth Survey Interviews - - Boca Chica Sub awardee´s center 3 MOSTCHA Youth Survey Interviews - - Santo Domingo Sub awardee´s center 17 69 Qual. CSA FGDs Female youths - Santo Domingo School 5 CSA FGDs Male youths - Santo Domingo School 4 Profamilia FGDs Female youths - Santo Domingo Sub awardee´s center 4 October 26, 2016 Cuant. Fundación Pediátrica por un Mañana Youth Survey Interviews - - Santo Domingo Sub awardee´s center 1 Project Hope Youth Survey Interviews - - Santo Domingo School 2 3 Qual. ACOPRO FGDs Male youths - Santo Domingo Sub awardee´s center 4 ACOPRO FGDs Female youths - Santo Domingo Sub awardee´s center 4 - FGDs Mixed (Entrena Staff) - Santo Domingo Ipsos office 5 October 27, 2016 Cuant. CSA Youth Survey Interviews - - Santo Domingo School 1 2 Children Int. BC Youth Survey Interviews - - Boca Chica Sub awardee´s center 1 0 Qual. Project Hope FGDs Male youths - Santo Domingo Sub awardee´s center 4 KII´s Interview Project Manager Boca Chica Sub awardee´s center 1 KII´s Interview Project Coordinator Boca Chica Sub awardee´s center 1 70 KII´s Interview Community Leader Boca Chica Sub awardee´s center 1 KII´s Interview School Principal Boca Chica School 1 KII´s Interview Businessman Boca Chica Sub awardee´s center 1 October 28, 2016 Cuant. ACOPRO Youth Survey Interviews - - Santo Domingo Sub awardee´s center 1 2 Qual. Caminante Proyecto Educativo FGDs Female youths - Boca Chica Sub awardee´s center 4 Caminante Proyecto Educativo FGDs Male youths - Boca Chica Sub awardee´s center 4 MOSTCHA FGDs Male youths - Santo Domingo Sub awardee´s center 5 MOSTCHA FGDs Female youths - Santo Domingo Sub awardee´s center 4 October 29, 2016 Cuant. Caminante Proyecto Educativo Youth Survey Interviews - - Boca Chica Sub awardee´s center 2 IDDI Youth Survey Interviews - - Santo Domingo Sub awardee´s center 1 9 Profamilia Youth Survey Interviews - - Santo Domingo Sub awardee´s center 6 Profamilia Youth Survey Interviews - - Santo Domingo Community 4 October 30, 2016 Qual. - KII´s Interview (Ministerio de la Juventud) Santo Domingo Sub awardee´s center 1 71 - KII´s Interview (Entrena) Santo Domingo Sub awardee´s center 1 - KII´s Interview (Entrena) Santo Domingo Sub awardee´s center 1 - KII´s Interview (Entrena) Santo Domingo Sub awardee´s center 1 October 31, 2016 Cuant. ACOPRO Youth Survey Interviews - - Santo Domingo Sub awardee´s center 8 Qual. Profamilia FGDs Male youths - Santo Domingo Sub awardee´s center 4 Novembe r 1, 2016 Cuant. IDDI Youth Survey Interviews - - Santo Domingo Sub awardee´s center 1 3 Novembe r 2, 2016 Cuant. ACOPRO Youth Survey Interviews - - Santo Domingo Sub awardee´s center 3 Novembe r 3, 2016 Cuant. Children Int. SD Youth Survey Interviews - - Santo Domingo School 8 IDDI Youth Survey Interviews - - Santo Domingo Sub awardee´s center 1 Project Hope Youth Survey Interviews - - Santo Domingo Sub awardee´s center 1 0 Novembe r 4, 2016 Qual. - KII´s Interview (Sur Futuro) Santo Domingo Sub awardee´s center 1 Novembe r 5, 2016 Cuant. ACOPRO Youth Survey Interviews - - Santo Domingo Sub awardee´s center 6 MOSTCHA Youth Survey Interviews - - Santo Domingo Sub awardee´s center 8 72 Profamilia Youth Survey Interviews - - Santo Domingo Sub awardee´s center 16 Novembe r 8, 2016 Cuant. CSA Youth Survey Interviews - - Santo Domingo School 6 Children Int. SD Youth Survey Interviews - - Santo Domingo School 27 JADOM Youth Survey Interviews - - Santo Domingo School 13 Novembe r 9, 2016 Cuant. Children Int. SD Youth Survey Interviews - - Santo Domingo School 18 IDDI Youth Survey Interviews - - Santo Domingo Sub awardee´s center 6 JADOM Youth Survey Interviews - - Santo Domingo School 9 Novembe r 10, 2016 Cuant. IDDI Youth Survey Interviews - - Santo Domingo Sub awardee´s center 6 IDDI Youth Survey Interviews - - Santo Domingo Community 44 Project Hope Youth Survey Interviews - - Santo Domingo School 12 Novembe r 11, 2016 Cuant. Children Int. SD Youth Survey Interviews - - Santo Domingo Sub awardee´s center 19 IDDI Youth Survey Interviews - - Santo Domingo Community 26 Profamilia Youth Survey Interviews - - Santo Domingo Sub awardee´s center 2 73 Novembe r 12, 2016 Cuant. ACOPRO Youth Survey Interviews - - Santo Domingo Sub awardee´s center 4 IDDI Youth Survey Interviews - - Santo Domingo Community 25 Project Hope Youth Survey Interviews - - Santo Domingo Community 4 Profamilia Youth Survey Interviews - - Santo Domingo Sub awardee´s center 14 Novembe r 15, 2016 Cuant. Project Hope Youth Survey Interviews - - Santo Domingo Sub awardee´s center 2 Children Int. SD Youth Survey Interviews - - Santo Domingo Sub awardee´s center 16 JADOM Youth Survey Interviews - - Santo Domingo School 12 Project Hope Youth Survey Interviews - - Santo Domingo School 13 Novembe r 16, 2016 Cuant. Caminante Proyecto Educativo Youth Survey Interviews - - Boca Chica School 22 Project Hope Youth Survey Interviews - - Santo Domingo Community 1 Novembe r 17, 2016 Cuant. Caminante Proyecto Educativo Youth Survey Interviews - - Boca Chica Community 10 JADOM Youth Survey Interviews - - Santo Domingo School 15 Novembe r 18, 2016 Cuant. Project Hope Youth Survey Interviews - - Santo Domingo School 6 74 Novembe r 19, 2016 Cuant. ACOPRO Youth Survey Interviews - - Santo Domingo Sub awardee´s center 8 Novembe r 21, 2016 Cuant. Project Hope Youth Survey Interviews - - Santo Domingo School 1 JADOM Youth Survey Interviews - - Santo Domingo School 1 0 75 ANNEX C. DATA COLLECTION INSTRUMENTS (ENGLISH) YOUTH OUTCOMES SURVEY INTERVIEW DETAILS Please complete this section, yourself, before the start of the interview. Do not ask the interviewees these questions. A1 Interviewer: A2 Interviewee code: OBTAINED FROM PRE￾GENERATED LIST A3 Date of Interview: A4 Province: A5 Municipality: A6 Neighborhood (Barrio): A7 Intake month and year: OBTAINED FROM PRE￾GENERATED LIST A8 Respondent's date of birth OBTAINED FROM PRE￾GENERATED LIST A9 Respondent's age at entry into Alerta Joven programs: OBTAINED FROM PRE￾GENERATED LIST A10 Alerta Joven programs participated in (check all that apply): OBTAINED FROM PRE￾GENERATED LIST  Entrepreneurship Program  Vocational / Educational Training  Life Skills Program  School Insertion Program  School Retention Program  Psychological Therapy  Public Policy Advocacy Program 76  HIV Prevention Program  HIV Testing Program  Crime and Violence Prevention Program  Legal Documentation Program  Microcredit Program  Health Services  Espacios para Crecer Educational Program  Youth Mediation Program A11 Start Time: A12 End Time: A13 Respondent's sex Male Female 1 2 A14 Revisor/Supervisor: A15 Date reviewed: 77 INFORMED CONSENT Hi, (smile and greet the respondent). My name is ______________, and I am a researcher conducting an evaluation of the Alerta Joven program. The purpose of this survey is to learn about your experiences in school, working, and things you and your friends do that may affect your health. As a participant, your responses could be helpful to improve the Alerta Joven program and other programs for youth like yourself. It will last around 45 minutes. Some of the questions I will be asking you will be sensitive and personal, for example, questions about your sexual and reproductive health and perceptions about crime and violence in your neighborhood. If you do not feel comfortable answering a question, that is okay. Your participation in this study is completely voluntary. You have the right to stop the interview at any time or say ‘no response’ or ‘don’t know’ to any questions that you don’t want to answer, and can stop the interview at any point in time. There are no right or wrong answers and you will not get in trouble if you choose not to respond. If you have any questions during the survey, please ask me to explain. Only the research team will know your name and responses and the information provided will not be linked to you. We won’t share your answers with anyone, not even your parents/guardian/in loco parentis and your responses will not affect your current or future participation in Alerta Joven or other programs. If you have any concerns about the interview, you may contact REDACTED for any reason and at any time. You may also contact REDACTED for questions about the study. Do you have any questions? Do you agree to participate in the survey? CONSENT STATEMENT: I understand and agree to participate in this study. Respondent provided consent: Yes_____ No_____ 78 SECTION 1: DEMOGRAPHICS AND HOME ENVIRONMENT 1.1 What is your marital status? Single Living together but not legally married Married Divorced Widowed Don’t know Prefer not to answer 1 2 3 4 5 97 98 1.2 Do you have children? Yes No Don’t know Prefer not to answer 1 2 97 98 1.3 Not including yourself, how many of the following people live in the house where you usually stay? a) Children or teenagers under 18 b) Parents or step-parents c) Other adults (such as aunts, uncles, grandparents, friends) Insert number Don’t know Prefer not to answer 97 98 1.4 How many times have you moved in the last 5 years? Insert number Don’t know Prefer not to answer 97 98 79 SECTION 2: EDUCATION Read aloud: Now I will ask you some questions about your educational level and goals. 2.1 Can you read and write? Yes No Prefer not to answer 1 2 98 2.2 What is the highest educational level you have completed? Básica Bachillerato Universitario Post grado Ninguno Prefer not to answer 1 2 3 4 5 98 2.3 Currently, you are attending… If attending school (responses 2-5), skip to 2.5 Ninguno Básica Bachillerato Universitario Post grado Prefer not to answer 1 2 3 4 5 98 2.4 What are the main reasons why you are out of school? Select all that apply. You should read aloud all answers to the respondent. Completed education Failed examinations Did not enjoy school Wanted to start working 1 2 3 4 80 NOTE FOR INTERVIEWER: Explain that “completed education” means the respondent has completed the highest level of schooling he/she feels is necessary for his/her future, but must include completion of all grades in the level of schooling the respondent chooses. Now skip to 2.10 Got married Parents/guardians did not want you to continue schooling Financial reasons Other (specify) ___________ Don’t know Prefer not to answer 5 6 7 8 97 98 2.5 What grade are you currently in? Insert number Don’t know Prefer not to answer 97 98 2.6 In the last 12 months, how many times, if any, did you miss school for a full day without being excused? Insert number Don’t know Prefer not to answer 97 98 2.7 In the last 12 months, how many times, if any, were you suspended from school? Insert number Don’t know Prefer not to answer 97 98 2.8 In the last 12 months, did you fail any courses at school? Insert number Don’t know Prefer not to answer 97 98 81 Read aloud: Now I would like to ask you how you feel about school. For each sentence that I read, can you tell me if you agree, disagree, or feel neutral about it (neither agree nor disagree)? You can also say “don’t know” or “prefer not to answer.” 2.9 a) I am interested in going to school. b) I am happy about my performance in school. c) I try hard in school. d) I complete my homework. e) I get along with my teachers. f) I get along with my peers. g) I feel that my education is important. h) I feel safe at school. i) I spend time outside of school reading for fun (not for school). j) I have achieved my educational goals Agree Disagree Neutral Don’t know Prefer not to answer 1 2 3 97 98 Read aloud: Now I would like you to think back to the time when you were ___ (insert age when respondent began participating in Alerta Joven minus 1 year): 2.10 Were you attending school then? If answer is 1 or 3, skip to 2.12 If answer is 97 or 98, skip to 2.13 Yes No I was enrolled but did not attend often Don’t know Prefer not to answer 1 2 3 97 98 2.11 What are the main reasons why you were out of school then? Select all that apply. Completed education Failed examinations Did not enjoy school 1 2 3 82 NOTE FOR INTERVIEWER: Read all answers aloud to the respondent. Explain that “completed education” means the respondent has completed the highest level of schooling he/she feels is necessary for his/her future, but must include completion of all grades in the level of schooling the respondent chooses. Now skip to 2.13 Wanted to start working Got married Parents did not want you to continue schooling Financial reasons Other (specify) ___________ Don’t know Prefer not to answer 4 5 6 7 8 97 98 2.12 What grade were you in? Insert number Don’t know Prefer not to answer 97 98 NOTE TO INTERVIEWER: Read aloud: Now I would like to ask you one question about how you see your education in the future. 2.13 With respect to education, what would you like to achieve in the future, if anything? Select all that apply. You should read aloud all answers to the respondent. To read better. To get better in mathematics. To get better in science. To be in school. To speak Spanish better. To speak English better. To learn more about jobs. To finish basic education. To finish the bachillerato. To go to university. To get along better with my teachers. 1 2 3 4 5 6 7 8 9 10 11 83 To feel safer at school. To improve my qualifications for the job market I do not have any educational goals. Don’t know Prefer not to answer 12 13 14 97 98 For those who are not in school but have not yet completed their education (generated by tablet based on answer to question 2.4): NOTE TO INTERVIEWER: Read aloud: Now I would like to ask you a couple of questions about how likely you think you are to return to school. 2.14 How likely are you to return to school? You should read aloud all answers to the respondent. Highly likely Somewhat likely Uncertain Somewhat unlikely Highly unlikely Don’t know Prefer not to answer 1 2 3 4 5 97 98 2.15 Please rate the influence of the following factors on your likelihood of returning to school: a) Participation in Alerta Joven programs b) Enjoyment of school c) Peers d) Financial reasons Strong positive influence Moderate positive influence No influence Moderate negative influence Strong negative influence 1 2 3 4 5 84 e) Marriage f) Family, including parents/guardians g) Other (please specify) Don’t know Prefer not to answer 97 98 For those who are currently in school (generated by tablet based on answer to question 2.3): NOTE TO INTERVIEWER: Read aloud: Now I would like to ask you a couple questions about how likely you think you are to remain in school. 2.16 How likely are you to stay in school until you have completed your education? You should read aloud all answers to the respondent. Very likely Somewhat likely Uncertain Somewhat unlikely Very unlikely Don’t know Prefer not to answer 1 2 3 4 5 97 98 2.17 Please rate the influence of the following factors on your likelihood of staying in school until you have completed your education: a) Participation in Alerta Joven programs b) Peers c) Enjoyment of school d) Financial reasons e) Marriage f) Family, including parents/guardians g) Other (please specify) Strong positive influence Moderate positive influence No influence Moderate negative influence Strong negative influence Don’t know Prefer not to answer 1 2 3 4 5 97 98 NOTE TO INTERVIEWER: Read aloud: This concludes the section on education. 85 SECTION 3: EMPLOYMENT NOTE TO INTERVIEWER: Read aloud: Now I will ask you some questions about employment and remuneration. 3.1 Have you ever worked for a salary or other financial remuneration? If answer is not YES, skip to 3.13 Yes No Don’t know Prefer not to answer 1 2 97 98 3.2 Are you currently working for financial remuneration? If answer is not YES, skip to 3.13 Yes No Don’t know Prefer not to answer 1 2 97 98 3.3 How many jobs do you currently have? Insert number Don’t know Prefer not to answer 97 98 3.4 How many hours do you work per week total? Insert number Don’t know Prefer not to answer 97 98 3.5 On average, how much do you earn per week total? Insert number 86 Don’t know Prefer not to answer 97 98 NOTE TO INTERVIEWER: Read aloud: For the following questions, please think about your primary job. Your primary job is the one for which you work the most hours each week. 3.6 For how many months have you been working at this job? Insert number Don’t know Prefer not to answer 97 98 3.7 Where do you work? At my home Plantation/ranch/garden At my boss’s house Construction site On the street Maquiladora/factory Shop/market/kiosk Mining site Other (specify) ______________ Don’t know Prefer not to answer 1 2 3 4 5 6 7 8 9 97 98 3.8 At this job, you are: Please read all responses, and select all that apply. A salaried worker An hourly or daily wage earner The boss Family assistant 1 2 3 4 87 General assistant Self-employed Other (please specify) _________ Prefer not to answer 5 6 7 98 3.9 Overall, how satisfied are you with this job? Very satisfied Somewhat satisfied Neither satisfied nor unsatisfied Somewhat unsatisfied Very unsatisfied Don’t know Prefer not to answer 1 2 3 4 5 97 98 3.10 What aspect of your job has the greatest positive effect on your satisfaction? Boss or supervisor Job responsibilities Wage Hours worked Benefits received Co-workers Prospects for the future Interest in the sector Other (specify) _____________ Don’t know Prefer not to answer 1 2 3 4 5 6 7 8 9 97 98 88 3.11 What aspect of your job has the greatest negative effect on your satisfaction? Boss or supervisor Job responsibilities Wage Hours worked Benefits received Co-workers Prospects for the future Interest in the sector Other (specify) _____________ Don’t know Prefer not to answer 1 2 3 4 5 6 7 8 9 97 98 3.12 How likely are you to stay in this sector for at least another year? Very likely Somewhat likely Uncertain Somewhat unlikely Very unlikely Don’t know Prefer not to answer 1 2 3 4 5 97 98 NOTE TO INTERVIEWER: Read aloud: Now I will ask you some questions about entrepreneurship. 3.13 How easy or difficult would it be for you to start a business? Very easy Somewhat easy Uncertain 1 2 3 89 Somewhat difficult Very difficult Don’t know Prefer not to answer 4 5 97 98 3.14 Did you ever own a business before participating in the Alerta Joven program? Yes No Prefer not to answer 1 2 98 3.15 Do you own a business now? If not Yes, skip to 3.17 Yes No Prefer not to answer 1 2 98 3.16 What type of business do you have? Skip to end of section Describe Don’t know Prefer not to answer 97 98 3.17 Do you want to start your own business? If No, skip to end of section Yes No Don’t know Prefer not to answer 1 2 97 98 3.18 How likely is it that you will start your own business? Very likely 1 90 Read aloud all answers to the respondent. Somewhat likely Uncertain Somewhat unlikely Very unlikely Don’t know Prefer not to answer 2 3 4 5 97 98 3.19 Do you have access to the following elements needed to start a business? a) Funding b) Business support services c) Knowledge of regulations d) Business skills Yes No Don’t know Prefer not to answer 1 2 97 98 NOTE TO INTERVIEWER: Read aloud: Now I will ask you some questions about financial remuneration. 3.20 Are you currently working with no financial remuneration? If answer is not YES, skip to close of section 3 and move onto section 4. Yes No Don’t know Prefer not to answer 1 2 97 98 3.21 Why are you currently working with no financial remuneration? Please select all that apply. Read aloud all answers to the respondent. I am doing an internship/traineeship I couldn’t find a paying job I want to volunteer for a while I work in my family’s business or home I am starting my own business 1 2 3 4 5 6 91 I receive in-kind remuneration (food, transportation, housing, other benefits) Don’t know Prefer not to answer 97 98 3.22 Do you expect to be able to receive financial remuneration for your work within the next few months? If answer is YES, skip to 3.24 Yes No Don’t know Prefer not to answer 1 2 97 98 3.23 Why not? You should read aloud all answers to the respondent. Skip to section 4 I am not looking for a paying job I don’t expect to find a paying job I don’t expect my business to be making a profit yet Other (explain) ____________ Don’t know Prefer not to answer 1 2 3 4 97 98 3.24 Why? You should read aloud all answers to the respondent. I am looking for a paying job My current job will begin to pay me Other (explain) ___________ Don’t know Prefer not to answer 1 2 3 97 98 NOTE TO INTERVIEWER: Read aloud: This concludes the section on employment. 92 SECTION 4: HEALTH AND DRUG AND ALCOHOL USE (self-administered) NOTE TO INTERVIEWER: Read aloud: This section will ask you questions about your health and well￾being, and drug and alcohol use. I will hand you my tablet so you can complete the questions in this section. By doing so, I will not be able to see or hear your responses. If you are uncomfortable answering any questions or wish to skip any questions, all you have to do is select “prefer not to answer” on the tablet. The survey will take you immediately to the next question. I also want to remind you that your answers will be kept confidential and will not be shared with anyone outside the research team. We will not identify the data you provide with your name; only a number. When you reach the end of this section, please hand the tablet back to me. INSTRUCTIONS FOR INTERVIEWER: You may now hand your tablet to the respondent. INSTRUCTIONS FOR RESPONDENT: The following questions are about your health and well-being, including drug and alcohol use. 4.1 In the last 12 months, have you had a physical exam or a check-up from a doctor or nurse? Yes No Don’t know Prefer not to answer 1 2 97 98 4.2 In the last 12 months, have you consumed alcohol, meaning a whole glass, not just a sip? If answer is not YES, skip to 4.5. Yes No Don’t know Prefer not to answer 1 2 97 98 4.3 In the last 12 months, how often have you had 5 or more glasses of an alcoholic beverage in one day? More than once a week Once a week More than once a month Once a month Less than once a month 1 2 3 4 5 93 Never Don’t know Prefer not to answer 6 97 98 4.4 During the next 12 months, how likely is it that you will consume more than 5 glasses of an alcoholic beverage in one day more than once? Very likely Somewhat likely Uncertain Somewhat unlikely Very unlikely Don’t know Prefer not to answer 1 2 3 4 5 97 98 4.5 In the last 12 months, have you used any illegal or recreational drugs? If answer is not Yes, skip to 4.10. Yes No Don’t know Prefer not to answer 1 2 97 98 4.6 In the last 12 months, have you used marijuana? If answer is not Yes, skip to 4.10 Yes No Don’t know Prefer not to answer 1 2 97 98 94 4.7 In the last 12 months, have you used any illegal or recreational drugs other than marijuana? If answer is not Yes, skip to 4.10 Yes No Don’t know Prefer not to answer 1 2 97 98 4.8 In the last 12 months, how often have you used the drugs you have tried? More than once a week Once a week More than once a month Once a month Less than once a month Don’t know Prefer not to answer 1 2 3 4 5 97 98 4.9 During the next 12 months, how likely is it that you will use drugs? Very likely Somewhat likely Uncertain Somewhat unlikely Very unlikely Don’t know Prefer not to answer 1 2 3 4 5 97 98 4.10 Please rate the influence of the following factors on your likelihood of using drugs in the next 12 months: a) Participation in Alerta Joven programs Strong positive influence Moderate positive influence No influence Moderate negative influence 1 2 3 4 95 b) Peers c) Financial reasons d) Marriage e) Family f) Other (please specify) Strong negative influence Don’t know Prefer not to answer 5 97 98 96 SECTION 5: SEXUAL AND REPRODUCTIVE HEALTH (self-administered) NOTE: This section will only appear if respondent is 13 or older and has been participating in Alerta Joven activities for at least three months, based on cover sheet data entry of Date of Birth and Intake Date. If not, the survey will progress to section 6. This section will ask you questions about your knowledge, experience, and beliefs regarding sexual and reproductive health. If you are uncomfortable answering any questions or wish to skip any questions, all you have to do is select “prefer not to answer” on the tablet. When you reach the end of this section, please hand the tablet back to me. INSTRUCTIONS FOR RESPONDENT: For the next set of statements, please tell me if you believe them to be true, false, or if you don’t know. 5.1 For the next set of statements, please tell me if you believe them to be true, false, or if you don’t know. a) A woman can get pregnant the very first time she has sexual intercourse with a man. b) A woman is most likely to get pregnant if she has sexual intercourse with a man half way between her periods. c) It is possible for a woman to get pregnant while she is breastfeeding. True False Don’t know Prefer not to answer 1 2 97 98 5.2 Have you ever had sexual relations? If answer is not YES, skip to 5.14 If answer is YES and respondent is under age 15, skip to 5.5. Yes No Don’t know Prefer not to answer 1 2 97 98 97 5.3 How old were you when you first had sexual relations? Insert number Don’t know Prefer not to answer 97 98 5.4 Have you ever had sexual relations in exchange for money, goods (cell phones, housing, etc.) or services (protection, food, etc.)? If answer is not YES, skip to 5.6 Yes No Don’t know Prefer not to answer 1 2 97 98 5.5 When was the last time you had sex in exchange for money, goods or services? Last week In the last 6 months In the last 12 months More than a year ago More than five years ago Don’t know Prefer not to answer 1 2 3 4 5 97 98 5.6 Have you ever had sexual intercourse? If answer is not YES, skip to 5.14 Yes No Don’t know Prefer not to answer 1 2 97 98 98 5.7 How many people have you had sexual intercourse with in the last 12 months? Insert number Don’t know Prefer not to answer 97 98 5.8 What type of birth control, if any, did you or your partner use the last time you had sexual intercourse? Answers not to be read out loud. Select correct option based on answer. No birth control method was used Condom Female condom Birth control pill Fertility calendar (period abstinence) Pull out Breastfeeding IUD Injections Implant Female sterilization Male sterilization Other (specify) ____________ Don’t know Prefer not to answer 1 2 3 4 5 6 7 8 9 10 11 12 13 97 98 5.9 How often do you or your most recent partner use a condom or other birth control method when having sexual intercourse? Always Sometimes Rarely Never Don’t know Prefer not to answer 1 2 3 4 97 99 98 5.10 MALES: How many times have you made a woman pregnant? Please include all pregnancies, regardless of whether it was carried to term. FEMALES: How many times have you been pregnant? Please include all pregnancies, regardless of whether it was carried to term. Never Once Twice Three times Four times Five or more times Don’t know Prefer not to answer 1 2 3 4 5 6 97 98 5.11 Have you ever tested positive for a sexually transmitted infection (STI)? Yes No Don’t know Prefer not to answer 1 2 97 98 5.12 If you want to have sexual intercourse, do you believe you can convince your partner to use a condom if he/she does not want to? Yes No Sometimes Don’t know Prefer not to answer 1 2 3 97 98 5.13 In the next 12 months, how likely are you to: a) Have sexual intercourse? Very likely Somewhat likely 1 2 100 If answer is not 1 Very Likely or 2 Somewhat Likely, skip to 5.14. Otherwise, proceed with b) and c). b) Use a condom during sexual intercourse? c) Use any form of birth control during sexual intercourse? Uncertain Somewhat unlikely Very unlikely Don’t know Prefer not to answer 3 4 5 97 98 5.14 Do you have an adult with whom you are able to have an open and honest conversation about sex? Yes No Don’t know Prefer not to answer 1 2 97 98 5.15 Has anyone ever taught you about sexually transmitted infections, including HIV/AIDS? If answer is not YES, skip to 5.17 Yes No Don’t know Prefer not to answer 1 2 97 98 5.16 Where did you receive this information? Select all that apply. Counselor or psychologist at school Teacher at school Friends Father or mother Internet Health worker 1 2 3 4 5 6 7 101 Alerta Joven Program Other (please specify) ______ Don’t know Prefer not to answer 8 97 98 5.17 Has anyone ever taught you about pregnancy prevention? If answer is not YES, skip to end. Yes No Don’t know Prefer not to answer 1 2 97 98 5.18 Where did you receive this information? Select all that apply. Counselor or psychologist at school Teacher at school Friends Father or mother Internet Health worker Program Alerta Joven Other (please specify) _______ Don’t know Prefer not to answer 1 2 3 4 5 6 7 8 97 98 INSTRUCTIONS FOR RESPONDENT: End of section 5. Please hand the tablet back to the interviewer. 102 SECTION 6: CRIME AND PERSONAL SAFETY (self-administered) Read aloud: This section will ask you questions about your personal safety and perceptions of crime in your neighborhood. I will now hand you my tablet again so you can complete the questions in this section. By doing so, I will not be able to see or hear your responses. Remember, if you are uncomfortable answering any questions or wish to skip any questions, all you have to do is select “prefer not to answer” on the tablet. As with the rest of the survey, all of your answers are confidential. When you reach the end of this section, please hand the tablet back to me. INSTRUCTIONS FOR INTERVIEWER: You may now hand your tablet to the respondent. 6.1 Currently, how much is your life impacted by the following. Please respond using a scale from 1 to 10, where 1 is no impact and 10 is very high impact. Please enter a number. a) Crime b) Delinquency Insert number Don’t know Prefer not to answer 97 98 6.2 In the last 12 months… a) Have you taken anything from a shop without paying for it? b) Have you threatened anybody? c) Have you carried a weapon? d) Have you participated in a fight? e) Have you injured anyone? f) Have you damaged or destroyed on purpose articles that do not belong to you? g) Have you been attacked or robbed? h) Have you attacked or robbed anyone? i) Have you sold or helped someone sell drugs? j) Have you been involved in a gang? Yes No Don’t know Prefer not to answer 1 2 97 98 103 6.3 In the next 12 months, how likely are you to… a) Take anything from a shop without paying for it? b) Participate in a fight? c) Carry a weapon? d) Threaten anybody? e) Injure anyone? f) Damage or destroy on purpose articles that do not belong to you? g) Be attacked or robbed? h) Attack or rob anyone? i) Sell or help someone sell drugs? j) Be involved in a gang? Very likely Somewhat likely Uncertain Somewhat unlikely Very unlikely Don’t know Prefer not to answer 1 2 3 4 5 97 98 6.4 Please rate the influence of the following factors on your likelihood of participating in any of the activities listed in the previous questions: a) Participation in Alerta Joven programs b) Peers c) Financial reasons d) Marriage e) Family f) Other (please specify) Strong positive influence Moderate positive influence No influence Moderate negative influence Strong negative influence Don’t know Prefer not to answer 1 2 3 4 5 97 98 INSTRUCTIONS FOR RESPONDENT: End of section 6. Please hand the tablet back to the interviewer. 104 SECTION 7: PERCEPTION OF PROGRAM ALERTA JOVEN Read aloud: This is the last section of our survey. I will now ask you questions about your experience with various interventions you participated in through the Alerta Joven program. Program 1 Program 2 Program 3 … more as needed Program lists to be automatically generated based on responses in Section A. Respondents will only answer these questions for programs they participated in. Most respondents participated in 1-3 programs. 7.1 How useful did you find this program? Very useful Somewhat useful Not useful Don’t know Prefer not to answer If Not Useful, go to 7.2. Other responses skip to 7.3 1 2 3 97 98 7.2 Why was the programming not useful? Check all that apply. Not interested in the subject Poor teaching No opportunities to apply Not in depth enough Time was too short Didn’t apply to me 1 2 3 4 5 6 105 Other (explain) _______ Don’t know Prefer not to answer 7 97 98 7.3 What aspect was most useful? Check all that apply Staff/teachers Peers in the program A mentor in the program The information/subject matter Gaining hands-on experience Networks or professional connections Other (explain)_______________ Don’t know Prefer not to answer 1 2 3 4 5 6 7 97 98 7.4 Why was that aspect useful? Insert explanation Don’t know Prefer not to answer 97 98 7.5 Would you be willing to pay for this program if you had to? 106 Yes No Don’t know Prefer not to answer 1 2 97 98 7.6 How likely would you be to recommend the program to others like you? Very likely Somewhat likely Uncertain Somewhat unlikely Very unlikely Don’t know Prefer not to answer 1 2 3 4 5 97 98 Read aloud: Please think about all of the programs we just discussed as a whole. I’m going to read a few statements. For each one, I want you to state to what extent you agree or disagree with the statement. 7.7 a) “I am satisfied with the programs overall.” b) “The programs helped me feel more confident.” c) “The programs helped me improve my standard of living.” d) “The programs helped me stay out of trouble.” Strongly agree Somewhat agree Neither agree nor disagree Somewhat disagree Strongly disagree Don’t know Prefer not to answer 1 2 3 4 5 97 98 NOTE TO INTERVIEWER: Thanks the respondent for taking the time to speak with you, and ensure any questions they have are answered. 107 QUALITATIVE QUESTION GUIDE Alerta Joven Group Interview, Key Informant Interview, and Focus Group Protocols ENTRENA GROUP INTERVIEWS Introduction and Consent My name is [____] and this is [_____]. We represent Social Impact Inc., an independent research company contracted by the US Agency for International Development. We are doing a performance evaluation of the Alerta Joven project, in order to gain a better understanding of what works and what can be improved in the project, which will help to inform future programming and approaches. We would like to ask you some questions about your experience with Alerta Joven. This interview is voluntary; you can withdraw at any time or choose not to answer any question. Keep in mind that there are no right or wrong answers, and we are interested in your candid opinions. Your comments will be kept confidential and not linked in any way to you or your organization in our reporting. The opinions you provide will be analyzed with responses from other individuals and groups to arrive at general findings. The interview should not take more than 60 minutes. With this information in mind – do you agree to continue with the interview? We would like to list you as an informant in an annex to our report, along with the names of other informants consulted for this evaluation. Is it okay if we include your name in that list? You may contact the evaluation team after the interview if you have any questions about this evaluation or want to provide additional information. You may contact REDACTED [NOTE: This information to be provided on paper to participants.] 108 Questions 1. TO EACH PERSON: Please tell me your position and role in implementing Alerta Joven. 2. TO EACH PERSON: How long have you been associated with the project? 3. How well have public-private partnerships [for example: between a private company/employer and Alerta Joven workforce training partners who connect youth to that company] worked in Alerta Joven? Please give some examples of successful and unsuccessful partnerships: Sub-awardee Successful PPP Unsuccessful PPP Children International – Boca Chica ACOPRO CAMINANTE Children International – Santo Domingo Este CSA MOSCHTA PROJECT HOPE IDDI PROFAMILIA FPUM FURJUG JADOM 4. What could be done to improve PPPs? Please explain both Alerta Joven’s influence as well as exogenous influences. 5. Which programs (prompt: workforce/entrepreneurship, health, education), do you think have been most successful in reducing the number of at-risk youth? What factors have made them more successful than others? 6. [IF NOT ADDRESSED IN ANSWERS TO PREVIOUS QUESTION] Which programs (prompt: workforce/entrepreneurship, health, education) do you think have been least successful in reducing the number of at-risk youth? What factors have made them less successful than others? 7. To what extent are the services offered to youth by Entrena and Entrena’s sub-awardees sustainable? a) In terms of financial resources [for example: will sub-awardees have the funds to continue services after Alerta Joven funding ends]? b) In terms of institutional capacity? c) In terms of local uptake? 109 Services offered by sub-awardees Financial resources to continue services? Institutional capacity to continue services? Local uptake to continue services? Entrepreneurship Vocational education training Assistance to legal documentation Espacios para crecer Public policy advocacy School insertion Youth mediation Microcredit HIV prevention HIV testing School retention Health services Psychological therapy 8. What additional training or support would sub-awardees benefit from? 9. To what extent has the capacity building training offered by Entrena helped sub-awardees to: a) improve their technical capabilities, b) improve their financial management capabilities, and c) access other sources of funding once Alerta Joven ends? Please give specific examples for each Santo Domingo-based sub-awardee: Sub-awardee Technical capabilities improved? How? Financial management capabilities improved? How? Ability to access other sources of funding once AJ ends? Children International – Boca Chica ACOPRO CAMINANTE Children International – Santo Domingo Este 110 CSA MOSCHTA PROJECT HOPE IDDI PROFAMILIA FPUM FURJUG JADOM 10. What additional training or support would sub-awardees benefit from? 11. Overall, do you view the Alerta Joven project positively or negatively? Please explain a. How do the youth you work with view the project? b. How do other partners (sub-awardees, employers, government, the community) that you work with view the project? c. To what extent do these beneficiaries and stakeholders value Alerta Joven? 12. What is your overall opinion on the Youth Alert project (could it be in terms of effectiveness, efficiency, management, coordination, etc.)? 13. What is the most important piece of advice that you can give for improving the project? 111 ENTRENA KII (NOTE: to be tailored based upon the sector specialist that is being interviewed.) Introduction and Consent My name is [____] and this is [_____]. We represent Social Impact Inc., an independent research company contracted by the US Agency for International Development. We are doing a performance evaluation of the Alerta Joven project, in order to gain a better understanding of what works and what can be improved in the project, which will help to inform future programming and approaches. We would like to ask you some questions about your experience with Alerta Joven. This interview is voluntary; you can withdraw at any time or choose not to answer any question. Keep in mind that there are no right or wrong answers, and we are interested in your candid opinions. Your comments will be kept confidential and not linked in any way to you or your organization in our reporting. The opinions you provide will be analyzed with responses from other individuals and groups to arrive at general findings. The interview should not take more than 60 minutes. With this information in mind – do you agree to continue with the interview? We would like to list you as an informant in an annex to our report, along with the names of other informants consulted for this evaluation. Is it okay if we include your name in that list? By the way, you may contact the evaluation team after the interview if you have any questions about this evaluation or want to provide additional information. You may contact REDACTED. 112 Questions 1. TO EACH PERSON: Please tell me your position and role in implementing Alerta Joven. 2. TO EACH PERSON: How long have you been associated with the project? 3. [FOR THOSE FAMILIAR WITH EDUCATION PROGRAMS] What is the likelihood that Alerta Joven youth beneficiaries will return to or stay in school? Please explain your reasoning. 4. [FOR THOSE FAMILIAR WITH HEALTH PROGRAMS] What is the likelihood that Alerta Joven youth beneficiaries will reduce risky behaviors (prompt: drug and alcohol use, sexual activity, violence, etc.)? Please explain your reasoning. 5. [FOR THOSE FAMILIAR WITH WORKFORCE PROGRAMS] What is the likelihood that Alerta Joven youth beneficiaries will obtain and maintain jobs? Please explain your reasoning. 6. [FOR THOSE FAMILIAR WITH WORKFORCE PROGRAMS] What is the likelihood that Alerta Joven youth beneficiaries will start their own businesses? Please explain your reasoning. 7. What factors do you think have contributed to the likelihood that Alerta Joven youth beneficiaries will: a. return to or stay in school; b. reduce risky behaviors; and c. obtain and maintain jobs or start their own businesses? Please explain both Alerta Joven’s influence as well as exogenous influences. 8. Which programs (prompt: workforce/entrepreneurship, health, education), do you think have been most successful in reducing the number of at-risk youth? What factors have made them more successful than others? 9. [IF NOT ADDRESSED IN ANSWERS TO PREVIOUS QUESTION] Which programs (prompt: workforce/entrepreneurship, health, education) do you think have been least successful in reducing the number of at-risk youth? What factors have made them less successful than others? 10. To what extent are the services offered to youth by Entrena and Entrena’s sub-awardees sustainable? a. In terms of financial resources [for example: will sub-awardees have the funds to continue services after Alerta Joven funding ends]? b. In terms of institutional capacity? c. In terms of local uptake? Please explain your reasoning and give examples. What additional training or support would sub-awardees benefit from? 11. To what extent has the capacity building training offered by Entrena helped sub-awardees to: a. improve their technical capabilities; b. improve their financial management capabilities; and c. access other sources of funding once Alerta Joven ends? Please explain your reasoning and give examples. What additional training or support would sub-awardees benefit from? 12. Overall, do you view the Alerta Joven project positively or negatively? Please explain. 113 a. How do the youth you work with view the project? b. How do other partners (sub-awardees, employers, government, the community) that you work with view the project? c. To what extent do these beneficiaries and stakeholders value Alerta Joven? 13. What is the most important piece of advice that you can give for improving the project? 114 SUB-AWARDEES GROUP INTERVIEW AND KII Introduction and Consent My name is [____] and this is [_____]. We represent Social Impact Inc., an independent research company contracted by the US Agency for International Development. We are doing a performance evaluation of the Alerta Joven project, in order to gain a better understanding of what works and what can be improved in the project, which will help to inform future programming and approaches. We would like to ask you some questions about your experience with Alerta Joven. This interview is voluntary; you can withdraw at any time or choose not to answer any question. Keep in mind that there are no right or wrong answers, and we are interested in your candid opinions. Your comments will be kept confidential and not linked in any way to you or your organization in our reporting. The opinions you provide will be analyzed with responses from other individuals and groups to arrive at general findings. The interview should not take more than 60 minutes. With this information in mind – do you agree to continue with the interview? We would like to list you as an informant in an annex to our report, along with the names of other informants consulted for this evaluation. Is it okay if we include your name in that list? By the way, you may contact the evaluation team after the interview if you have any questions about this evaluation or want to provide additional information. You may contact REDACTED. [INTERVIEWER: Begin the FGD or KII by giving the MINI-SURVEY to each respondent. An explanation of the purpose of the mini-survey is described on the survey itself; you may explain the purpose of the mini-survey to participants. Once the mini-survey is completed, begin with the questions below.] 115 Questions 1. TO EACH PERSON: Please tell me your position and role in implementing Alerta Joven. 2. TO EACH PERSON: How long have you been associated with the project? 3. To what extent do you think your organization will be able to continue its services for youth [prompt: financially and with respect to institutional capacity] after support from Alerta Joven stops? Explain. a. How, if at all, has your organization’s capacity to offer these services changed during the Alerta Joven project? What factors have contributed to these changes? 4. [FOR EDUCATION-RELATED ACTIVITIES] What is the likelihood that Alerta Joven youth beneficiaries will return to or stay in school? Please explain your reasoning. a. How do you think Alerta Joven youth compare to youths not participating in the Alerta Joven interventions in this regard? Please provide specific examples. b. What accounts for the difference (please address both Alerta Joven and exogenous factors)? 5. [FOR HEALTH-RELATED ACTIVITIES] What is the likelihood that Alerta Joven youth beneficiaries will reduce risky behaviors (prompt: drug and alcohol use, sexual activity, violence, etc.)? Please explain your reasoning. a. How do you think Alerta Joven youth compare to youths not participating in the Alerta Joven interventions in this regard? Please provide specific examples. b. What accounts for the difference (address both Alerta Joven and exogenous factors)? 6. [FOR WORKFORCE DEVELOPMENT-RELATED ACTIVITIES] What is the likelihood that Alerta Joven youth beneficiaries will obtain and maintain jobs? Please explain your reasoning. a. How do you think Alerta Joven youth compare to youths not participating in the Alerta Joven interventions in this regard? Please provide specific examples. b. What accounts for the difference (address both Alerta Joven and exogenous factors)? 7. [FOR WORKFORCE DEVELOPMENT-RELATED ACTIVITIES] How well have public￾private partnerships worked in Alerta Joven? Please give some examples of the most successful and least successful partnerships. What could be done to improve PPPs? 8. [FOR ENTREPRENEURSHIP-RELATED ACTIVITIES] What is the likelihood that Alerta Joven youth beneficiaries will start their own businesses? Please explain your reasoning. a. How do you think Alerta Joven youth compare to youths not participating in the Alerta Joven interventions in this regard? Please provide specific examples. b. What accounts for the difference (address both Alerta Joven and exogenous factors)? 9. Which components of the [prompt ALL THAT APPLY: workforce/health/education] programming have been most successful in reducing the number of at-risk youth? What has made them more successful than others? 116 10. Which components of the [prompt ALL THAT APPLY: workforce/health/education] programming have been least successful in reducing the number of at-risk youth? What has made them less successful than others? 11. How, if at all, have your organization’s technical capabilities (personnel management, record keeping, proposal writing, etc.) changed during implementation of the Alerta Joven project? What factors have contributed to these changes? 12. [IF NOT ALREADY ADDRESSED] How effective has the capacity-building training from ENTRENA been in improving your organization’s technical capabilities? In what ways? What additional training or other support would you be interested in receiving? 13. How, if at all, have your organization’s financial capabilities changed during implementation of the Alerta Joven project? What factors have contributed to these changes? 14. [IF NOT ALREADY ADDRESSED] How effective has the training from ENTRENA been in improving your organization’s financial capabilities? In what ways? What additional financial training or support would you be interested in receiving? 15. To what extent, if at all, has Entrena’s support affected your access to other sources of funding? How so? 16. What resources, buy-in or other elements are necessary for your organization to continue to provide youth services after the Alerta Joven project ends? 17. Overall, do you view the Alerta Joven project positively or negatively? Please explain. a. How do the youth you work with view the project? b. How do other partners (sub-awardees, employers, government, the community) that you work with view the project? c. To what extent do these beneficiaries and stakeholders value Alerta Joven? 18. What is the most important piece of advice that you can give for improving the project? 117 CBOS AND CSOS GROUP INTERVIEW Introduction and Consent My name is [____] and this is [_____]. We represent Social Impact Inc., an independent research company contracted by the US Agency for International Development. We are doing a performance evaluation of the Alerta Joven project, in order to gain a better understanding of what works and what can be improved in the project, which will help inform future programming and approaches. We would like to ask you some questions about your experience with Alerta Joven. This interview is voluntary; you can withdraw at any time or choose not to answer any question. Keep in mind that there are no right or wrong answers, and we are interested in your candid opinions. Your comments will be kept confidential and not linked in any way to you or your organization in our reporting. The opinions you provide will be analyzed with responses from other individuals and groups to arrive at general findings. The interview should not take more than 60 minutes. With this information in mind – do you agree to continue with the interview? We would like to list you as an informant in an annex to our report, along with the names of other informants consulted for this evaluation. Is it okay if we include your name in that list? By the way, you may contact the evaluation team after the interview if you have any questions about this evaluation or want to provide additional information. You may contact REDACTED. 118 Questions 1. Please tell me your position and association with Alerta Joven. 2. How long have you been associated with the project? 3. [FOR EDUCATION-RELATED ACTIVITIES] What is the likelihood that Alerta Joven youth beneficiaries will return to or stay in school? Please explain your reasoning. a. How do you think Alerta Joven youth compare to youths not participating in the Alerta Joven interventions in this regard? Please provide specific examples. b. What accounts for the difference (address both Alerta Joven and exogenous factors)? 4. [FOR HEALTH-RELATED ACTIVITIES] What is the likelihood that Alerta Joven youth beneficiaries will reduce risky behaviors (prompt: drug and alcohol use, sexual activity, violence, etc.)? Please explain your reasoning. a. How do you think Alerta Joven youth compare to youths not participating in the Alerta Joven interventions in this regard? Please provide specific examples. b. What accounts for the difference (address both Alerta Joven and exogenous factors)? 5. [FOR WORKFORCE DEVELOPMENT-RELATED ACTIVITIES] What is the likelihood that Alerta Joven youth beneficiaries will obtain and maintain jobs? Please explain your reasoning. a. How do you think Alerta Joven youth compare to youths not participating in the Alerta Joven interventions in this regard? Please provide specific examples. b. What accounts for the difference (address both Alerta Joven and exogenous factors)? 6. [FOR ENTREPRENEURSHIP-RELATED ACTIVITIES] What is the likelihood that Alerta Joven youth beneficiaries will start their own businesses? Please explain your reasoning. a. How do you think Alerta Joven youth compare to youths not participating in the Alerta Joven interventions in this regard? Please provide specific examples. b. What accounts for the difference (address both Alerta Joven and exogenous factors)? 7. Which components of the [prompt ALL THAT APPLY: workforce/health/education] programming have been most successful in reducing the number of at-risk youth? 8. Which components of the [prompt ALL THAT APPLY: workforce/health/education] programming have been least successful in reducing the number of at-risk youth? 9. To what extent do you believe that the benefits for youth will be retained once the Alerta Joven project stops? For the community? Please explain. a. What resources, buy-in or other elements (policies, training, etc.) are needed to make them more sustainable? 10. Alerta Joven is now in its 4th year of existence. How do you believe the issues of youth (e.g., participation in risky behaviors, lack of education, unemployment) in this community have changed in that time? What accounts for these changes (please address both Alerta Joven’s and exogenous factors’ influence)? 119 11. Overall how do you feel the Alerta Joven project is perceived by the community? To what extent does the community value Alerta Joven? What could the project do to improve community perceptions and value for the community? 12. What recommendations do you have for improving Alerta Joven? 13. Please share any other experiences or examples that would give me a clear picture of how Alerta Joven operates. 120 SCHOOL REPRESENTATIVES KII Introduction and Consent My name is [____] and this is [_____]. We represent Social Impact Inc., an independent research company contracted by the US Agency for International Development. We are doing a performance evaluation of the Alerta Joven project, in order to gain a better understanding of what works and what can be improved in the project, which will help to inform future programming and approaches. We would like to ask you some questions about your experience with Alerta Joven. This interview is voluntary; you can withdraw at any time or choose not to answer any question. Keep in mind that there are no right or wrong answers, and we are interested in your candid opinions. Your comments will be kept confidential and not linked in any way to you or your organization in our reporting. The opinions you provide will be analyzed with responses from other individuals and groups to arrive at general findings. The interview should not take more than 60 minutes. With that in mind – do you agree to continue with the interview? We would like to list you as an informant in an annex to our report, along with the names of other informants consulted for this evaluation. Is it okay if we include your name in that list? We will be asking you for specific numbers of students enrolled in your school including Alerta Joven youth. Do you have this information readily available? [IF THIS INFORMATION IS NOT AVAILABLE] Will you be able to estimate these numbers or provide them to me sometime this week? By the way, you may contact the evaluation team after the interview if you have any questions about this evaluation or want to provide additional information. You may contact REDACTED. 121 Questions 1. Please tell me your position 2. Do you have any knowledge of the Alerta Joven project? If so, what is your association with Alerta Joven? a. How long have you been associated with the project? 3. Approximately how many students aged 11- 24 are enrolled in your school? Approximately what percentage do you think have participated in the Alerta Joven program? 4. How do you think youth who have received Alerta Joven services compare to their peers who have not been involved with Alerta Joven, in the below areas? What factors account for these differences? a. Enrollment b. Attendance c. Academic performance d. Social behavior (for example: drug and alcohol use, sexual activity, violence, legal troubles, etc.) 5. Given your answers to the previous question, what is the likelihood that Alerta Joven youth beneficiaries will return to or stay in school? Please explain your reasoning. 6. To what extent do you believe that Alerta Joven’s benefits will be retained by participating youth once the Alerta Joven project stops? Please explain. What resources, buy-in or other elements (policies, training, etc.) are needed to make them more sustainable? 7. Alerta Joven is now in its 4th year of existence. How do you believe the issues of youth (e.g., participation in risky behaviors, lack of education, unemployment) in this community have changed in that time? What accounts for these changes (please address both Alerta Joven’s and exogenous factors’ influence)? 8. Overall, how do you feel the project is perceived by the community? To what extent does the community value the project? What could the project do to improve community perceptions and value? 9. Overall how do you feel the project is perceived by youth? To what extent do youth value the project? What could the project do to improve youth perceptions and value? 10. What recommendations do you have for improving Alerta Joven? 11. Please share any other experiences or examples that would give me a clear picture of how Alerta Joven operates. 122 EMPLOYERS KII Introduction and Consent My name is [____] and this is [_____]. We represent Social Impact Inc., an independent research company contracted by the US Agency for International Development. We are doing a performance evaluation of the Alerta Joven project, in order to gain a better understanding of what works and what can be improved in the program which will help to inform future programming and approaches. We would like to ask you some questions about your experience with Alerta Joven. This interview is voluntary; you can withdraw at any time or choose not to answer any question. Keep in mind that there are no right or wrong answers, and we are just interested in your candid opinions. Your comments will be kept confidential and not linked in any way to you or your organization in our reporting. The opinions you provide will be analyzed with responses from other individuals and groups to arrive at general findings. The interview should not take more than 60 minutes. With that in mind – do you agree to continue with the interview? We would like to list you as an informant in an annex to our report, along with the names of other informants consulted for this evaluation. Is it okay if we include your name in that list? By the way, you may contact the evaluation team after the interview if you have any questions about this evaluation or want to provide additional information. You may contact REDACTED. 123 Questions 1. Please tell me your position and association with Alerta Joven/[SUB-AWARDEE]. 2. How long have you been associated with the project? 3. How many people do you employ? How many of your employees are under the age of 26? Of employees under 26 years of age, how many are Alerta-Joven/[SUB-AWARDEE] trained youth? [CALCULATE PERCENTAGE]. a. Please confirm: approximately [INSERT PERCENTAGE] of employees under 26 years old are Alerta Joven youth? b. What kinds of positions have you typically employed Alerta Joven/[SUB-AWARDEE] youth in (administrative, low/high-skilled, etc.), both for youth who have worked for you in the past and youth currently working? 4. How do Alerta Joven/[SUB-AWARDEE]-trained youth compare to other (non-Alerta Joven-trained) youth employees in terms of the appropriateness of their skills to the positions for which you have hired them? Are there other skills which Alerta Joven/[SUB-AWARDEE] youth lack but which you need in your employees? 5. Based upon your experience, what is the likelihood that Alerta Joven/[SUB-AWARDEE] youth beneficiaries will continue to maintain and obtain jobs? Please explain. 6. How effective has your business’s partnership with Alerta Joven/[SUB-AWARDEE] been? (prompts: matching youths’ skills and employer’s needs, process efficiency, adequacy of needs and skills assessments, etc.) How effective would you say the partnership between Alerta Joven and the private sector in general has been (prompts: matching youths’ skills and employer’s needs, process efficiency, adequacy of needs and skills assessments, etc.)? Please give some examples. What could be done by Alerta Joven/[SUB￾AWARDEE] or others to improve the project’s relationship with businesses like yours? 7. [IF NOT ALREADY ANSWERED] Please describe any skills gaps between Alerta Joven/[SUB￾AWARDEE] -trained youth and what businesses like yours need them to have? a. How do these skills gaps compare to the skills gaps among youth before Alerta Joven programming began in 2011? b. Are there any differences between youth who participate in Alerta Joven/[SUB-AWARDEE] training versus youth who have participated in other (non-Alerta Joven/[SUB-AWARDEE]) trainings? c. What aspects or components of the Alerta Joven/[SUB-AWARDEE] trainings make them more or less successful in preparing youth for employment? 8. To what extent do you think the workforce development trainings Alerta Joven/[SUB-AWARDEE] has been supporting will continue after the project ends? Please explain. What will be necessary to ensure they can continue? 9. To what extent do you think youth will retain the workforce development skills they learned through Alerta Joven/[SUB-AWARDEE]? Please explain. 10. If you were to offer Alerta Joven/[SUB-AWARDEE] advice on improving its workforce readiness training for youth, what would it be? 124 11. Overall how do you feel the private sector perceives Alerta Joven/[SUB-AWARDEE]? To what extent do employers value the project? Please give concrete examples. What could the program do to improve perceptions and value? 12. Please share any other experiences or examples that would give me a clear picture of how Alerta Joven/[SUB-AWARDEE] works with employers in the community. 125 GOVERNMENT ENTITIES KII Introduction and Consent My name is [____] and this is [_____]. We represent Social Impact Inc., an independent research company contracted by the US Agency for International Development. We are doing a performance evaluation of the Alerta Joven project, in order to gain a better understanding of what works and what can be improved in the project, which will help inform future programming and approaches. We would like to ask you some questions about your experience with Alerta Joven. This interview is voluntary; you can withdraw at any time or choose not to answer any question. Keep in mind that there are no right or wrong answers, and we are interested in your candid opinions. Your comments will be kept confidential and not linked in any way to you or your organization in our reporting. The opinions you provide will be analyzed with responses from other individuals and groups to arrive at general findings. The interview should not take more than 60 minutes. With this information in mind – do you agree to continue with the interview? We would like to list you as an informant in an annex to our report, along with the names of other informants consulted for this evaluation. Is it okay if we include your name in that list? By the way, you may contact the evaluation team after the interview if you have any questions about this evaluation or want to provide additional information. You may contact REDACTED. 126 Questions 1. Please tell me your position and association with Alerta Joven. 2. How long have you been associated with the project? 3. In your experience with Alerta Joven programs, which components of the [prompt ALL THAT APPLY: workforce/health/education] programming have been most successful in reducing the number of at-risk youth? 4. In your experience with Alerta Joven programs, which components of the [prompt ALL THAT APPLY: workforce/health/education] programming have been least successful in reducing the number of at-risk youth? 5. How do you think Alerta Joven youth compare to youths not participating in the Alerta Joven interventions? Please provide specific examples. What accounts for the difference (please address both Alerta Joven and exogenous factors)? 6. To what extent do you believe that the benefits for youth will be retained once the Alerta Joven project stops? For the community? Please explain. What resources, buy-in or other elements (policies, training, etc.) are needed to make them more sustainable? 7. To what extent do you think the [prompt: workforce/health/education] services that Alerta Joven has been supporting will continue after the project ends? Please explain. What will be necessary to ensure they can continue? 8. How do you view the work that ENTRENA has been doing to build capacity of sub-awardees (local NGOs) to carry out future work? Has this capacity building work been more successful or less successful, and how? Please provide specific examples that you know of. 9. Alerta Joven is now in its 4th year of existence. How do you believe the issues of youth (e.g., participation in risky behaviors, lack of education, unemployment) in this community have changed in that time? What accounts for these changes (please address both Alerta Joven’s and exogenous factors’ influence)? 10. Overall, do you view the Alerta Joven project positively or negatively? Please explain. How do other government ministries involved [prompt: Ministry of Youth, Attorney’s Office, Ministry of the Presidency, Ministry of Labor, National Institute for Vocational Training] view the project? 11. Overall how do you feel the Alerta Joven project is perceived by the community? What could the project do to improve community perceptions and value for the community? 12. What recommendations do you have for improving Alerta Joven? 13. Please share any other experiences or examples that would give me a clear picture of how Alerta Joven operates. 127 ALERTA JOVEN YOUTH FOCUS GROUP PROTOCOL Introduction and Consent My name is [____] and this is [_____]. We represent Social Impact Inc., an independent research company contracted by the US Agency for International Development. We are doing a study on the Alerta Joven project so we can see what went well and what could have gone better. This information will help USAID develop other youth programs. We would like to ask you some questions about your experience with Alerta Joven. You can drop out of the group at any time or choose not to answer any question. There are no right or wrong answers, and we are interested in your honest opinions and ideas. We want to hear from all of you, and ask you to respect each other’s views at all times which means you cannot make fun or tease anyone about what they have said. I may sometimes act as a traffic cop by encouraging someone who has been quiet to talk, or by asking someone to hold off for a few minutes. The group interview should not take more than an hour and a half. We would like to record this conversation so that we do not miss any of your comments; however, all of what you say will be kept secret. No one will know what ‘he’ said or ‘she’ said (point to a couple of people) outside of this group. We will not include your names in any reports. We also ask each of you not to share any information about what anyone in this group said with other people after this session. By the way, you can contact the evaluation team later if you have any questions about this study or want to provide additional information. You can contact REDACTED. With that in mind – does everyone agree to participate? If it is OK with you, we will start the recorder and begin this session. 128 Instructions FGDs will be stratified by age cohort; 11-17 or 18-26 Each FGD will include an IPSOS facilitator (as the lead) and an SI facilitator (as support). While the IPSOS facilitator should lead the majority of the FGD in order to establish a positive rapport with youth, the SI facilitator should feel free to jump in with pertinent questions. Have the groups play a short game to get them warmed up and feeling comfortable (ideas to be provided at facilitator training) 10-15 minutes. 1. Do you remember when you started AJ? If yes, describe yourself then. Describe yourself now. What changed? What didn't change? Depending on the group, if they seem amenable - give each young person a piece of paper to draw on with colored pencils or markers and ask them to visually show how they are different now than they were before their participation in Alerta Joven. Ask them to specifically show how they have changed any of their ideas or behaviors around: education, sports, friends, employment, self￾employment and also how they perceive their family and community (in a positive or negative way). They can use pictures and words. Give them 15-20 minutes. Bring back the group and have them discuss the key points they want to say from what they drew – note these on a flip chart. If the group seems like they will not engage in this activity (maybe more likely for older youth), they can describe orally in the group. Prompts for this question: a. Did you attend school? What grades did you get? Did you get a job? Did you start your own business? Did you get arrested? Did you make different decisions about your sexual relationships? Did you change your drinking or drug use patterns? Etc. Follow up questions as they give their responses might be: 2. What caused these changes and how? a. How did Alerta Joven help you to learn something new or make a change in your life? (Potential prompts: with regard to school, grades, attendance, getting a job, setting up your own business, legal troubles/arrests, different patterns with sexual relationships, drinking or drug use patterns, etc.) b. How did you get to know new people through Alerta Joven? c. What did Alerta Joven do to help you see your family or community differently? Take pictures of their drawings to keep as data collection record and let them keep their pictures if they want to. 3. What do you like about Alerta Joven (prompt: games, activities, staff, training)? 4. What don’t you like about Alerta Joven (prompt: not enough activities, no food or drinks, not held in the summer)? 5. What do your friends think about Alerta Joven? What do others in the community (parents, teachers, relatives) think about the program? What do they like or not like? 6. What do your friends and classmates say to you about your participation in AJ? What do your parents, teachers, and relatives say to you about your participation in AJ? How does what they say about your participation in AJ make you feel? 129 7. What would you like to see changed about the project to make it better? a. What programs or services would you like to see for youth in your community? 8. If you were to tell a friend who doesn’t participate about Alerta Joven, what three words would you use to describe the program? 9. Does anyone have any other thoughts or feelings they want to share about Alerta Joven? 130 MINI SURVEY ON CAPACITY OF SUB AWARDEES NOTE TO ENUMERATOR (DO NOT READ TO PARTICIPANT) The purpose of this mini survey is to provide a quantitative measure of the technical and financial management capacity of sub awardees to serve as a data point to answer Evaluation Question 4. This brief questionnaire will be administered at the beginning of key informant interviews and group discussions with sub awardees as a way to introduce the conversation and will be triangulated with data from similar questions collected from Entrena. EQ4: To what extent has the capacity building training offered by Entrena helped sub-awardees to improve their technical and financial management capabilities which could help them access other sources of funding once Alerta Joven is over? CONSENT LANGUAGE: (READ TO PARTICIPANT) My name is [____] and this is [_____]. We represent Social Impact Inc., an independent research company contracted by the US Agency for International Development. We are doing a performance evaluation of the Alerta Joven project, in order to gain a better understanding of what works and what can be improved in the project, which will help to inform future programming and approaches. We would like to ask you some questions about your experience with Alerta Joven. This interview is voluntary; you can withdraw at any time or choose not to answer any question. Keep in mind that there are no right or wrong answers, and we are interested in your candid opinions. Your comments will be kept confidential and not linked in any way to you or your organization in our reporting. The opinions you provide will be analyzed with responses from other individuals and groups to arrive at general findings. This survey should not take more than 10 minutes and the interview overall should not take more than 60 minutes. With this information in mind – do you agree to continue with the interview? We would like to list you as an informant in an annex to our report, along with the names of other informants consulted for this evaluation. Is it okay if we include your name in that list? By the way, you may contact the evaluation team after the interview if you have any questions about this evaluation or want to provide additional information. You may contact REDACTED. 131 Question 1: How would you rate the change in your organization’s knowledge of financial management now compared to before receiving the training provided by Entrena? 1 2 3 4 5 Decreased Somewhat Decreased No change Somewhat Increased Increased Question 2: How would you rate the change in your organization’s knowledge of personnel management now compared to before receiving the training provided by Entrena? 1 2 3 4 5 Decreased Somewhat Decreased No change Somewhat Increased Increased Question 3: How would you rate the change in your organization’s knowledge of record keeping and data management now compared to before receiving the training provided by Entrena? 1 2 3 4 5 Decreased Somewhat Decreased No change Somewhat Increased Increased Question 4: How would you rate the change in your organization’s knowledge of writing proposals for new business now compared to before receiving the training provided by Entrena? 1 2 3 4 5 Decreased Somewhat Decreased No change Somewhat Increased Increased Question 5: How would you rate the change in your organization’s knowledge of monitoring and evaluation now compared to before receiving the training provided by Entrena? 1 2 3 4 5 Decreased Somewhat Decreased No change Somewhat Increased Increased Question 6: To what extent do you agree with the following statement, “As a result of the capacity building training my organization received from Entrena, I feel my organization is better placed to access funding from other sources.” 1 2 3 4 5 Completely disagree Somewhat disagree Neither agree nor disagree Somewhat agree Completely agree 132 ANNEX D. DATA COLLECTION INSTRUMENTS (SPANISH) YOUTH OUTCOMES SURVEY PORTADA Nota: Las instrucciones para los entrevistadores estarán siempre en mayúsculas. No deben leerse a los encuestados. INSTRUCCIONES PARA LOS ENTREVISTADORES Por favor, rellene esta sección antes de comenzar la entrevista. No haga estas preguntas a los encuestados. A1 Entrevistador: A2 Código del encuestado: OBTENIDO DE UNA LISTA PRE-GENERADA. A3 Fecha de la entrevista: A4 Provincia: A5 Municipio: A6 Sector (barrio): A7 Mes y año de inscripción: OBTENIDO DE UNA LISTA PRE-GENERADA. A8 Fecha de nacimiento del encuestado OBTENIDO DE UNA LISTA PRE-GENERADA A9 Edad del encuestado en el momento de ingresar en Alerta Joven: OBTENIDO DE UNA LISTA PRE-GENERADA A10 Programas de Alerta Joven en los que ha participado  Programa Empresarial  Formación Vocacional/Educativa 133 (comprobar todos los aplicables): OBTENIDO DE UNA LISTA PRE-GENERADA  Programa de Habilidades para la Vida  Programa de Inserción Escolar  Programa de Retención Escolar  Terapia Psicológica  Programa de Promoción de Políticas Públicas  Programa de Prevención del VIH  Programa de Tamizaje del VIH (diagnóstico)  Programa de Prevención del Crimen y de la Violencia  Programa de Documentación Legal  Programa de Microcréditos  Servicios de Salud  Programa Educativo Espacios para Crecer  Programa de Mediación Juvenil A11 Hora de inicio: A12 Hora de finalización: A13 Sexo del encuestado Hombre Mujer 1 2 A14 Revisor/Supervisor: A15 Fecha de revisión: 134 CONSENTIMIENTO INFORMADO Hola, . Mi nombre es ______________, y estoy llevando a cabo una evaluación del programa Alerta Joven. El propósito de esta encuesta es conocer tu experiencia escolar, laboral y sobre cosas que hacen tú y tus amigos que puedan afectar tu salud. Al participar, tus respuestas pueden ser muy útiles para mejorar el programa Alerta Joven y otros programas para otros jóvenes como tú. Durará unos 45 minutos. Algunas de las preguntas que te haré serán delicadas y personales, por ejemplo preguntas relativas a tu salud sexual y tus percepciones sobre la delincuencia y la violencia en tu barrio. Si no te sientes cómodo/a respondiendo alguna pregunta, no hay ningún problema. Tu participación en este estudio es totalmente voluntaria. Puedes detener la entrevista en cualquier momento o decir «no respondo» o «no sé» a cualquier pregunta que no quieras responder y, puedes parar la entrevista en un momento dado. No hay respuestas correctas o equivocadas y no hay ningún problema si decides no responder. Si tienes alguna duda durante la entrevista, por favor, pídeme que te lo explique. Únicamente el equipo de investigación conocerá tu nombre y respuestas y la información facilitada no se vinculada contigo. No compartiremos tus respuestas con nadie, ni siquiera con tus padres/tutores y tus respuestas no afectarán tu participación presente o futura en Alerta Joven o en cualquier otro programa. Si tienes alguna duda o preguntas sobre la entrevista, puedes ponerte en contacto con REDACTED por cualquier motivo o en cualquier momento. También puedes ponerte en contacto REDACTED para preguntas relativas al estudio. ¿Tienes alguna pregunta? ¿Aceptas participar hoy en la encuesta? DECLARACIÓN DE CONSENTIMIENTO: Entiendo y acepto participar en este estudio. Consentimiento del encuestado otorgado: Si_____ No_____ 135 SECCIÓN 1: DATOS DEMOGRÁFICOS Y ENTORNO FAMILIAR 1.1 ¿Cuál es tu estado civil? Soltero(a) Unión Libre Casado(a) Divorciado(a) Viudo(a) No lo sé Prefiero no contestar 1 2 3 4 5 97 98 1.2 ¿Tienes hijos? Si No No lo sé Prefiero no contestar 1 2 97 98 1.3 Sin incluirte tú, ¿cuántas de las siguientes personas viven en la casa en la que sueles permanecer? d) Niños o jóvenes menores de 18 años e) Padres o padrastros f) Otros adultos (como tías, tíos, abuelos, amigos) Introducir número No lo sé Prefiero no contestar 97 98 1.4 ¿Cuántas veces has cambiado de domicilio en los últimos 5 años? Introducir número No lo sé Prefiero no contestar 97 98 136 SECCIÓN 2: EDUCACIÓN Leer en voz alta: Ahora voy a hacerte algunas preguntas sobre tu nivel y tus metas educativas. 2.1 ¿Sabes leer y escribir? Si No Prefiero no contestar 1 2 98 2.2 ¿Cuál es el nivel educativo más alto que has aprobado? Básica Bachillerato Universitario Post grado Ninguno Prefiero no contestar 1 2 3 4 5 98 2.3 Actualmente estas asistiendo a... Si está asistiendo a la escuela o al colegio (respuestas 2-5), pasar a 2.5. NOTA PARA EL ENTREVISTADOR: Generalmente la “escuela” es pública y los “colegios” privados. Luego de identificar en cual sector estudia el participante continuar refiriéndose al mismo según sea el caso. Ninguno Básica Bachillerato Universitario Post grado Prefiero no contestar 1 2 3 4 5 98 2.4 ¿Cuáles son las razones principales para que no asistas a la escuela o colegio? Seleccionar todos los que sean aplicables. Educación finalizada Exámenes no superados No me gustaba el colegio 1 2 3 137 Debe leer en voz alta todas las respuestas al/a la encuestado/a. NOTA PARA EL ENTREVISTADOR: Explique que «educación finalizada» significa que el encuestado ha finalizado el nivel más alto de escolarización que él/ella cree que es necesario para su futuro, pero que debe incluir la finalización de todos los cursos en el nivel de escolarización que seleccione el encuestado. Pase ahora al 2.10 Quería empezar a trabajar Me casé Mis padres/tutores no querían que continuase en el colegio Razones económicas Otras (especificar) _______________ No lo sé Prefiero no contestar 4 5 6 7 8 97 98 2.5 ¿Qué grado estas cursando actualmente? Introducir número No lo sé Prefiero no contestar 97 98 2.6 En el último año, ¿cuántas veces has faltado a la escuela por un día completo sin excusa? Introducir número No lo sé Prefiero no contestar 97 98 2.7 En los últimos 12 meses, ¿cuántas veces has sido suspendido/a de la escuela? Introducir número No lo sé Prefiero no contestar 97 98 2.8 En los últimos 12 meses, ¿cuántas veces has repetido algún curso en la escuela? Introducir número No lo sé Prefiero no contestar 97 138 98 Leer en voz alta: Ahora me gustaría preguntarte cómo te sientes en la escuela o colegio. En relación con cada frase que lea, podrías decirme si estás de acuerdo, en desacuerdo o neutro (ni de acuerdo ni en desacuerdo)? También puedes decir «no lo sé» o «prefiero no contestar». 2.9 a) Estoy interesado en ir a la escuela o al colegio. b) Estoy contento con mi rendimiento en escuela o colegio. c) Me esfuerzo en mis estudios. d) Hago todos mis deberes. e) Me llevo bien con mis profesores. f) Me llevo bien con mis compañeros. g) Creo que mi educación es importante. h) Me siento seguro en el escuela o colegio. i) Cuanto no estoy en la escuela o colegio, paso tiempo leyendo por diversión (no por tareas del de la escuela) j) He alcanzado mis metas educativas De acuerdo En desacuerdo Neutro No lo sé Prefiero no contestar 1 2 3 97 98 Leer en voz alta: Ahora me gustaría que volvieras mentalmente a la época en que tenías ____ (introducir la edad a la que el encuestado comenzó a participar en Alerta Joven menos 1 año): 2.10 Ibas entonces al colegio? Si la respuesta es 1 o 3, saltar a 2.12 Si la respuesta es 97 o 98, saltar a 2.13 Si No Estaba matriculado, pero no solía ir a menudo. 1 2 3 97 139 No lo sé Prefiero no contestar 98 2.11 ¿Cuáles eran las razones principales para que no asistieras entonces al colegio? Seleccionar todas los que sean aplicables. NOTA PARA EL ENTREVISTADOR: Leer todas las respuestas en voz alta al/a la encuestado/a. Explique que «educación finalizada» significa que el encuestado ha finalizado el nivel más alto de escolarización que él/ella cree que es necesario para su futuro, pero que debe incluir la finalización de todos los cursos en el nivel de escolarización que seleccione el encuestado. Pase ahora al 2.13 Educación finalizada Exámenes no superados No me gustaba el colegio Quería empezar a trabajar Me casé Mis padres no querían que continuase en el colegio Razones económicas Otras (especificar) _______________ No lo sé Prefiero no contestar 1 2 3 4 5 6 7 8 97 98 2.12 En qué curso estabas? Introducir número No lo sé Prefiero no contestar 97 98 NOTA PARA EL ENTREVISTADOR: Leer en voz alta: Ahora me gustaría hacerte una pregunta acerca de cómo ves tu educación en el futuro. 2.13 Con respecto a tu educación, ¿qué te gustaría lograr en el futuro? Marca todo lo que aplique Me gustaría leer mejor. Me gustaría mejorar en matemáticas. Me gustaría mejorar en ciencias. 1 2 3 140 Debe leer en voz alta todas las respuestas al/a la encuestado/a. Me gustaría estar en la escuela. Me gustaría hablar mejor el español. Me gustaría hablar mejor el inglés. Me gustaría saber más acerca de empleos. Me gustaría terminar la básica. Me gustaría terminar el bachillerato. Me gustaría ir a la universidad. Me gustaría llevarme mejor con mis maestros. Me gustaría sentirme más seguro en la escuela. Me gustaría mejorar mis calificaciones. No tengo ninguna meta. No lo sé Prefiero no contestar 4 5 6 7 8 9 10 11 12 13 14 97 98 Para aquellos que no asisten al colegio pero que aún no han finalizado su educación (generado por tablet basándose en la pregunta 2.4): NOTA PARA EL ENTREVISTADOR: Leer en voz alta: Ahora me gustaría hacerte un par de preguntas sobre la probabilidad que ves de volver al colegio. 2.14 ¿Qué probabilidad ves de volver a la escuela o al colegio? Debe leer en voz alta todas las respuestas al/a la encuestado/a. Muy probable Probable Ni probable ni improbable Improbable 1 2 3 4 141 Muy improbable No lo sé Prefiero no contestar 5 97 98 2.15 Por favor, valora la influencia de los factores siguientes en la probabilidad de tu vuelta a la escuela o al colegio: h) Participación en los programas de Alerta Joven i) Disfrutas en el colegio j) Los compañeros k) Razones económicas l) Matrimonio m) Familia, incluyendo los padres/tutores n) Otra (especificar por favor) Gran influencia positiva Moderada influencia positiva No tiene influencia Moderada influencia negativa Gran influencia negativa No lo sé Prefiero no contestar 1 2 3 4 5 97 98 Para aquellos que asisten actualmente al colegio (generado por tablet basándose en la pregunta 2.3): NOTA PARA EL ENTREVISTADOR: Leer en voz alta: Ahora me gustaría hacerte un par de preguntas sobre la probabilidad que sigas en el colegio. 2.16 ¿Qué probabilidad hay de que sigas en la escuela o el colegio hasta que termines tu educación? Debe leer en voz alta todas las respuestas al/a la encuestado/a. Muy probable Probable Ni probable ni improbable Improbable Muy improbable No lo sé Prefiero no contestar 1 2 3 4 5 97 98 142 2.17 Por favor, valora la influencia de los siguientes factores en la probabilidad de que permanezcas en la escuela o el colegio hasta que hayas terminado tu educación: h) Participación en los programas de Alerta Joven i) Los compañeros j) Disfrutas en el colegio k) Razones económicas l) Matrimonio m) Familia, incluyendo los padres/tutores n) Otra (especificar por favor) Gran influencia positiva Moderada influencia positiva No tiene influencia Moderada influencia negativa Gran influencia negativa No lo sé Prefiero no contestar 1 2 3 4 5 97 98 NOTA PARA EL ENTREVISTADOR: Leer en voz alta: Con esto finaliza la sección de educación 143 SECCIÓN 3: EMPLEO NOTA PARA EL ENTREVISTADOR: Leer en voz alta: Ahora voy a hacerte algunas preguntas sobre el trabajo y la remuneración. 3.1 Has realizado alguna vez un trabajo por paga o ganancia? Si la respuesta no es SÍ, saltar a 3.13 Si No No lo sé Prefiero no contestar 1 2 97 98 3.2 Durante la última semana, ¿realizaste algún trabajo por paga o ganancia? Si la respuesta no es SÍ, saltar a 3.13 Si No No lo sé Prefiero no contestar 1 2 97 98 3.3 ¿Cuántos trabajos tienes actualmente? Introducir número No lo sé Prefiero no contestar 97 98 3.4 ¿Cuántas horas trabajas semanalmente en total? Introducir número No lo sé Prefiero no contestar 97 98 144 3.5 En promedio, ¿cuánto ganas en una semana? Introducir número No lo sé Prefiero no contestar 97 98 NOTA PARA EL ENTREVISTADOR: Leer en voz alta: Para las siguientes preguntas piensa en tu trabajo principal. Tu trabajo principal es aquel en el que trabajas más horas semanales. 3.6 ¿Cuántos meses llevas en ese trabajo? Introducir número No lo sé Prefiero no contestar 97 98 3.7 ¿En qué lugar realizas este trabajo? En mi casa Plantación/rancho/jardín En la casa del jefe/patrón Sitio de construcción En la calle Maquiladora/fábrica Colmado/Tienda/Mercado/Kiosco/ Almacén En una mina Otros (especifica) No lo sé Prefiero no contestar 1 2 3 4 5 6 7 8 9 97 98 3.8 En este trabajo tú eres…? Empleado con salario Empleado sin salario 1 2 145 Por favor, lea todas las respuestas y seleccione todas las aplicables. Jefe Ayudante familiar Ayudante general Por cuenta propia Otro (especifica) Prefiero no contestar 3 4 5 6 7 98 3.9 En general, ¿Qué tan satisfecho/a estas con este trabajo? Muy satisfecho Satisfecho Ni satisfecho ni insatisfecho Insatisfecho Muy insatisfecho No lo sé Prefiero no contestar 1 2 3 4 5 97 98 3.10 ¿Qué aspecto de tu trabajo tiene el mayor efecto positivo en tu satisfacción? El jefe o supervisor Las responsabilidades del trabajo El sueldo Las horas de trabajo Los beneficios recibidos Los compañeros Las perspectivas de futuro El interés en el sector Otros (especificar) _______________ No lo sé Prefiero no contestar 1 2 3 4 5 6 7 8 9 97 98 146 3.11 ¿Qué aspecto de tu trabajo tiene el mayor efecto negativo en tu satisfacción? El jefe o supervisor Las responsabilidades del trabajo El sueldo Las horas de trabajo Los beneficios recibidos Los compañeros Las perspectivas de futuro El interés en el sector Otros (especificar) _______________ No lo sé Prefiero no contestar 1 2 3 4 5 6 7 8 9 97 98 3.12 ¿Qué probabilidad existe de que continues en este sector de trabajo por un año más? Muy probable Probable Ni probable ni improbable Improbable Muy improbable No lo sé Prefiero no contestar 1 2 3 4 5 97 98 NOTA PARA EL ENTREVISTADOR: Leer en voz alta: Ahora voy a hacerte algunas preguntas sobre la empresa. 3.13 ¿Te resultó fácil o difícil iniciar un negocio? Muy fácil Fácil Ni probable ni improbable Difícil 1 2 3 4 147 Muy difícil No lo sé Prefiero no contestar 5 97 98 3.14 ¿Habías tenido alguna vez un negocio antes de participar en el programa Alerta Joven? Si No Prefiero no contestar 1 2 98 3.15 ¿Eres dueño de tu negocio ahora? Si la respuesta no es SÍ, saltar a 3.17 Si No Prefiero no contestar 1 2 98 3.16 ¿Qué tipo de negocio tienes? Si ha terminado, saltar a 3.20 Describir No lo sé Prefiero no contestar 97 98 3.17 ¿Quieres poner en marcha tu propio negocio? Si No, saltar a 3.20 Si No No lo sé Prefiero no contestar 1 2 97 98 3.18 ¿Qué probabilidad ves de iniciar tu propio negocio? Muy probable Probable 1 2 148 Lea en voz alta todas las respuestas al/a la encuestado/a. Ni probable ni improbable Improbable Muy improbable No lo sé Prefiero no contestar 3 4 5 97 98 3.19 ¿Tienes acceso a los siguientes elementos necesarios para iniciar un negocio? e) Fondos f) Servicios de apoyo empresarial g) Conocimiento de la normativa h) Habilidades empresariales Si No No lo sé Prefiero no contestar 1 2 97 98 NOTA PARA EL ENTREVISTADOR: Leer en voz alta: Ahora voy a hacerte algunas preguntas sobre la remuneración económica. 3.20 Actualmente, realizas algún trabajo sin paga? Si la respuesta no es SÍ, salte para cerrar la sección 3 y pasar a la sección 4. Si No No lo sé Prefiero no contestar 1 2 97 98 3.21 ¿Por qué estás trabajando actualmente sin remuneración económica? Seleccionar todos los que sean aplicables. Lea en voz alta todas las respuestas al/a la encuestado/a. Estoy haciendo prácticas/en formación No pude encontrar un trabajo remunerado Quiero ser voluntario durante un tiempo Trabajo en un negocio familiar o en casa 1 2 3 4 5 6 149 Estoy poniendo en marcha mi propio negocio Recibo remuneración en especie (comida, transporte, alojamiento, otros beneficios) No lo sé Prefiero no contestar 97 98 NOTA PARA EL ENTREVISTADOR: Leer en voz alta: Con esto finaliza la sección de trabajo. 150 SECCIÓN 4: SALUD Y CONSUMO DE DROGAS Y ALCOHOL (auto-administrado) NOTA PARA EL ENTREVISTADOR: Leer en voz alta: En esta sección se te harán preguntas sobre tu salud y bienestar y sobre el consumo de drogas y de alcohol. Te pasaré mi tableta para que puedas rellenar las preguntas de esta sección. De este modo yo no podré ver ni oír tus respuestas. Si no te sientes cómodo/a respondiendo alguna pregunta o quieres saltarte alguna, lo único que tienes que hacer es seleccionar «prefiero no contestar» en la tableta. La encuesta te llevará inmediatamente a la siguiente pregunta. También quiero recordarte que tus respuestas se guardarán confidencialmente y no se compartirán con nadie fuera del equipo de investigación. No identificaremos los datos que proporciones con tu nombre; únicamente con un número. Cuando llegues al final de esta sección, vuelve a pasarme la tableta por favor. INSTRUCCIONES PARA EL ENTREVISTADOR: Ahora puede pasarle su tableta al/a la encuestado/a. INSTRUCCIONES PARA EL/LA ENCUESTADO/A: Las siguientes preguntas son sobre tu salud y bienestar, incluyendo el consumo de drogas y de alcohol. 4.1 En los últimos 12 meses, ¿has pasado algún examen físico o chequeo por parte de un médico o una enfermera? Si No No lo sé Prefiero no contestar 1 2 97 98 4.2 En los últimos 12 meses, ¿has consumido alcohol, una copa/vaso entero, no sólo un sorbo? Si la respuesta no es SÍ, saltar a 4.5 Si No No lo sé Prefiero no contestar 1 2 97 98 151 4.3 En los últimos 12 meses, ¿Cuántas veces has tomado cinco vasos/copas o más de bebidas alcohólicas en el mismo día? Más de una vez a la semana Una vez a la semana Más de una vez al mes Una vez al mes Menos de una vez al mes Nunca No lo sé Prefiero no contestar 1 2 3 4 5 6 97 98 4.4 Durante los próximos 12 meses, ¿Qué probabilidad hay de que consumas más de una vez, más de 5 vasos de una bebida alcohólica en un día? Muy probable Probable Ni probable ni improbable Improbable Muy improbable No lo sé Prefiero no contestar 1 2 3 4 5 97 98 4.5 En los últimos 12 meses, ¿Has consumido alguna droga ilegal o recreativa? Si la respuesta no es SÍ, saltar a 4.10 Si No No lo sé Prefiero no contestar 1 2 97 98 4,6 En los últimos 12 meses, ¿Has probado marihuana? Si 1 152 Si la respuesta no es SÍ, saltar a 4.10 No No lo sé Prefiero no contestar 2 97 98 4.7 En los últimos 12 meses, ¿Has consumido alguna droga ilegal o recreativa distinta de la marihuana? Si la respuesta no es SÍ, saltar a 4.10 Si No No lo sé Prefiero no contestar 1 2 97 98 4.8 En los últimos 12 meses, ¿Con qué frecuencia has consumido las drogas que has probado? Más de una vez a la semana Una vez a la semana Más de una vez al mes Una vez al mes Menos de una vez al mes No lo sé Prefiero no contestar 1 2 3 4 5 97 98 4.9 Durante los próximos 12 meses, ¿Qué probabilidad hay de que consumas drogas? Muy probable Probable Ni probable ni improbable Improbable Muy improbable No lo sé Prefiero no contestar 1 2 3 4 5 97 98 153 4.10 Por favor, valora la influencia de los factores siguientes en la probabilidad de que consumas drogas durante los próximos 12 meses: g) Participación en los programas de Alerta Joven h) Los compañeros i) Razones económicas j) Matrimonio k) Familia, incluyendo los padres/tutores l) Otra (especificar por favor) Gran influencia positiva Moderada influencia positiva No tiene influencia Moderada influencia negativa Gran influencia negativa No lo sé Prefiero no contestar 1 2 3 4 5 97 98 154 SECCIÓN 5: SALUD SEXUAL Y REPRODUCTIVA (auto-administrado) NOTA: Esta sección aparecerá únicamente si el encuestado tiene 13 o más años y ha estado participando en las actividades de Alerta Joven durante tres meses al menos, basándose en los datos de la portada de fecha de nacimiento y fecha de ingreso. En caso contrario, la encuesta avanza hasta la sección 6. En esta sección se te harán preguntas sobre tus conocimientos, experiencia y creencias relativas a la salud sexual y reproductiva. Si no te sientes cómodo/a respondiendo alguna pregunta o quieres saltarte alguna, lo único que tienes que hacer es seleccionar «prefiero no contestar» en la tableta. Cuando llegues al final de esta sección, vuelve a pasarme la tableta por favor. INSTRUCCIONES PARA EL/LA ENCUESTADO/A: Respecto al siguiente grupo de afirmaciones, dime si crees que son ciertas, falsas o si no lo sabes. 5.1 Respecto al siguiente grupo de afirmaciones, dime si crees que son verdaderas, falsas o si no lo sabes. d) Una mujer puede quedarse embarazada la primera vez que tiene relaciones sexuales con un hombre. e) Es más probable que una mujer se quede embarazada si tiene relaciones sexuales con un hombre en el momento intermedio entre dos periodos. f) Es posible que una mujer se quede embarazada mientras está dando el pecho. Verdadero Falso No lo sé Prefiero no contestar 1 2 97 98 5.2 ¿Alguna vez has tenido relaciones sexuales? Si la respuesta no es SÍ, saltar a 5.14 Si No No lo sé Prefiero no contestar 1 2 97 98 155 5.3 ¿Qué edad tenías la primera vez que tuviste relaciones sexuales? Introducir número No lo sé Prefiero no contestar 97 98 5.4 ¿Alguna vez has tenido relaciones sexuales por dinero, bienes (celulares, vivienda, etc.) o servicios (protección, comida, etc.)? Si la respuesta no es SÍ, saltar a 5.6 Si No No lo sé Prefiero no contestar 1 2 97 98 5.5 ¿Cuándo ha sido la última vez que has tenido relaciones sexuales a cambio de dinero, bienes o servicios? La semana pasada En los últimos 6 meses En los últimos 12 meses Hace más de un año Hace más de cinco años No lo sé Prefiero no contestar 1 2 3 4 5 97 98 5.6 ¿Has tenido relaciones sexuales alguna vez? Si la respuesta no es SÍ, saltar a 5.14 Si No No lo sé Prefiero no contestar 1 2 97 156 98 5.7 ¿Con cuántas personas has tenido relaciones sexuales en los últimos 12 meses? Introducir número No lo sé Prefiero no contestar 97 98 5.8 ¿Qué tipo de método anticonceptivo utilizaste tú o tu pareja la última vez que tuvieron relaciones sexuales, si es que utilizaste alguno? Respuestas para no ser leídas en voz alta. Seleccione la opción correcta basada en la respuesta. No se utilizó ningún método anticonceptivo Condón Condón femenino Píldora anticonceptiva Calendario de fertilidad (periodo de abstinencia) Marcha atrás o retiro Lactancia DIU Inyecciones Implante Esterilización femenina Esterilización masculina Otros (especificar) _______________ No lo sé Prefiero no contestar 1 2 3 4 5 6 7 8 9 10 11 12 13 97 98 5.9 ¿Con qué frecuencia usas un condón, tú o tu última pareja como método anticonceptivo cuando tienen relaciones sexuales? Siempre A veces 1 2 157 Raramente Nunca No lo sé Prefiero no contestar 3 4 97 98 5.10 HOMBRES: ¿Cuántas veces has dejado embarazada a una mujer? Incluye todos los embarazos, independientemente de si han llegado a término. MUJERES: ¿Cuántas veces has estado embarazada? Incluye todos los embarazos, independientemente de si han llegado a término. Nunca Una vez Dos veces Tres veces Cuatro veces Cinco o más veces No lo sé Prefiero no contestar 1 2 3 4 5 6 97 98 5.11 ¿Alguna vez has dado positivo en una prueba de enfermedades de transmisión sexual (ETS)? Si No No lo sé Prefiero no contestar 1 2 97 98 5.12 Si quisieras tener sexo, ¿crees que podrías convencer a tu pareja de que use condón en caso de que él/ella se niegue a usarlo? Si No A veces No lo sé Prefiero no contestar 1 2 3 97 98 158 5.13 En los próximos 12 meses, ¿Qué probabilidad hay de: d) que tengas relaciones sexuales? Si la respuesta es 1 Muy probable o 2 Probable, saltar a 5.14. En caso contrario, seguir con b) y c). e) uses un condón durante las relaciones sexuales? f) uses algún otro método anticonceptivo durante las relaciones sexuales? Muy probable Probable Ni probable ni improbable Improbable Muy improbable No lo sé Prefiero no contestar 1 2 3 4 5 97 98 5.14 ¿Hay algún adulto con el que puedas tener una conversación abierta y honesta sobre sexo? Si No No lo sé Prefiero no contestar 1 2 97 98 5.15 ¿Alguna vez alguien te ha enseñado (o hablado) acerca de las infecciones de transmisión sexual? Si la respuesta no es SÍ, saltar a 5.17 Si No No lo sé Prefiero no contestar 1 2 97 98 5.16 ¿Dónde has recibido información? Marca todas las que apliquen Orientador(a) o Psicólogo(a) de la escuela Profesor(a) de la escuela 1 2 3 159 Amigos Padre o Madre Internet Promotor(a) de Salud Programa Alerta Joven Otro (Especifique): No lo sé Prefiero no contestar 4 5 6 7 8 97 98 5.17 ¿Alguna vez alguien te ha enseñado acerca de la prevención de embarazo? Si la respuesta no es SÍ, saltar al final. Si No No lo sé Prefiero no contestar 1 2 97 98 5.18 ¿Dónde has recibido información? Marca todas las que apliquen Orientador(a) o Psicólogo(a) de la escuela Profesor(a) de la escuela Amigos Padre o Madre Internet Promotor(a) de Salud Programa Alerta Joven Otro (Especifique): No lo sé Prefiero no contestar 1 2 3 4 5 6 7 8 97 98 INSTRUCCIONES PARA EL/LA ENCUESTADO/A: Fin de la sección 5. Por favor, devuélvele la tableta al entrevistador 160 SECCIÓN 6: CRIMEN Y SEGURIDAD PERSONAL (auto-administrado) Leer en voz alta: En esta sección se te harán preguntas sobre tu seguridad personal y sobre tus percepciones del crimen y la violencia en tu vecindario. Ahora volveré a pasarte mi tableta para que puedas rellenar las preguntas de esta sección. De este modo yo no podré ver ni oír tus respuestas. Recuerda, si no te sientes cómodo/a respondiendo alguna pregunta o quieres saltarte alguna, lo único que tienes que hacer es seleccionar «prefiero no contestar» en la tableta. Lo mismo que en el resto de la entrevista, todas tus respuestas son confidenciales. Cuando llegues al final de esta sección, vuelve a pasarme la tableta por favor. INSTRUCCIONES PARA EL ENTREVISTADOR: Ahora puede pasarle su tableta al/a la encuestado/a. 6.1 En la actualidad, ¿qué impacto tiene en tu vida lo siguiente: Por favor, responde utilizando una escala del 1 al 10, en la que el 1 es que no tiene impacto y el 10 que tiene un gran impacto. Por favor introduzca un número. c) Crimen d) Delincuencia Introducir número No lo sé Prefiero no contestar 97 98 6.2 En los últimos 12 meses ... k) ¿Has tomado algo de una tienda sin pagar por ello? l) ¿Has amenazado a alguien? m) ¿Has llevado un arma? n) ¿Has participado en alguna pelea o riña? o) ¿Has herido a alguien? p) ¿Has dañado o destruido intencionadamente artículos que no fueran de tu propiedad? q) ¿Has sido atacado o robado? r) ¿Has atacado o robado a alguien? Si No No lo sé Prefiero no contestar 1 2 97 98 161 s) ¿Has vendido drogas o ayudado a alguien a venderlas? t) ¿Has estado metido en alguna banda? 6.3 En los próximos 12 meses, ¿Qué probabilidad hay de que ... k) te lleves algo de una tienda sin pagarlo? l) participes en una pelea? m) lleves un arma? n) amenaces a alguien? o) hieras a alguien? p) dañes o destruyas intencionadamente artículos que no fueran de tu propiedad? q) seas atacado o robado? r) ataques o robes a alguien? s) vendas drogas o ayudes a venderlas? t) estés metido en una banda? Muy probable Probable Ni probable ni improbable Improbable Muy improbable No lo sé Prefiero no contestar 1 2 3 4 5 97 98 6.4 Por favor, valora la influencia de los factores siguientes en la probabilidad de participar en actividades criminales o peligrosas: g) Participación en los programas de Alerta Joven h) Los compañeros i) Razones económicas j) Matrimonio k) Familia, incluyendo los padres/tutores l) Otra (especificar por favor) Gran influencia positiva Moderada influencia positiva No tiene influencia Moderada influencia negativa Gran influencia negativa No lo sé Prefiero no contestar 1 2 3 4 5 97 98 INSTRUCCIONES PARA EL/LA ENCUESTADO/A: Fin de la sección 6. Por favor, devuélvele la tableta al entrevistador 162 SECCIÓN 7: PERCEPCIÓN DEL PROGRAMA ALERTA JOVEN Leer en voz alta: Esta es la última sección de la encuesta. Ahora te haré algunas preguntas sobre tu experiencia con varias intervenciones en la que has participado a través del programa Alerta Joven. Programa 1 Programa 2 Programa 3 ... más si es necesario Lista de programas que se generará automáticamente basándose en las respuestas de la sección A. Los encuestados únicamente responderán a las preguntas para los programas en los que han participado. La mayoría de los encuestados han participado en 1-3 programas. 7.1 ¿Qué utilidad le encontraste a este programa? Es muy útil Es útil en cierta medida No es útil No lo sé Prefiero no contestar Si No lo encontró útil, vaya a 7.2. En caso de otras respuestas vaya a 7.3 1 2 3 97 98 7.2 ¿Por qué no fue útil el programa? Comprobar todos los que sean aplicables. No estaba interesado en el tema Enseñanza pobre No hay oportunidades de aplicarlo No es suficientemente profundo 1 2 3 4 163 El tiempo fue demasiado corto No era aplicable para mi Otras (explicar) ________ No lo sé Prefiero no contestar 5 6 7 97 98 7.3 ¿Qué aspecto fue el más útil? Clasificación obligatoria en una escala de más  menos útil Personal/profesores Compañeros en el programa Un mentor en el programa La información/la materia Obtención de experiencia práctica Las redes o conexiones profesionales Otras (explicar) __________________ No lo sé Prefiero no contestar 1 2 3 4 5 6 7 97 98 7.4 ¿Por qué este aspecto fue el más útil? Para el aspecto clasificado como el más útil en 7.3 Introducir explicación 164 No lo sé Prefiero no contestar 97 98 7.5 Estarías dispuesto a pagar por este programa si tuvieras que hacerlo? Si No No lo sé Prefiero no contestar 1 2 97 98 7.6 ¿Con qué probabilidad recomendarías el programa a otras personas como tú? Muy probable Probable Ni probable ni improbable Improbable Muy improbable No lo sé Prefiero no contestar 1 2 3 4 5 97 98 Leer en voz alta: Por favor piensa en todos los programas de los que acabamos de hablar en conjunto. Voy a leer algunas afirmaciones. Para cada uno, me gustaría que me dijeses en qué medida estás de acuerdo o en desacuerdo con ella. 7.7 e) «Estoy satisfecho con los programas en general.» Totalmente de acuerdo De acuerdo 1 2 165 f) «Los programas me ayudaron a tener más confianza en mí mismo.» g) «Los programas me ayudaron a mejorar mi nivel de vida.» h) «Los programas me han ayudado a no meterme en problemas.» Ni de acuerdo ni en desacuerdo En desacuerdo Totalmente en desacuerdo No lo sé Prefiero no contestar 3 4 5 97 98 NOTA PARA EL ENTREVISTADOR: De las gracias al encuestado por haberse tomado el tiempo para hablar con usted y asegúrese de que se ha respondido a cualquier pregunta que tengan. 166 QUALITATIVE QUESTION GUIDE ENTREVISTA DE GRUPO A ENTRENA Introducción y consentimiento Mi nombre es [____] y él/ella es [_____]. Somos representantes de Social Impact Inc., una compañía de investigación independiente contratada por la Agencia de los Estados Unidos para el Desarrollo Internacional. Estamos evaluando el desempeño del proyecto Alerta Joven para comprender mejor qué funciona y qué puede mejorarse en el proyecto; ello ayudará a facilitar información para futuros programas y enfoques. Nos gustaría hacerle algunas preguntas acerca de su experiencia con Alerta Joven. Esta entrevista es voluntaria; puede retirarse en cualquier momento o decidir no responder a cualquier pregunta. No hay respuestas correctas o incorrectas; nos interesan sus opiniones sinceras. Sus comentarios serán confidenciales y no se vincularán de modo alguno con usted o su organización en nuestro informe. Los comentarios proporcionados se analizarán junto con las respuestas de otras personas y grupos para llegar a conclusiones generales. La entrevista no debería durar más de 60 minutos. ¿Está conforme y desea continuar con la entrevista? Nos gustaría incluirle como informante en un anexo de nuestro informe junto con los nombres de otros informantes consultados para esta evaluación. ¿Le parece bien que incluyamos su nombre en esa lista? Si tiene alguna pregunta sobre esta evaluación o desea proporcionar información adicional, puede comunicarse con el equipo de evaluación después de la entrevista. Puede contactar a REDACTED Preguntas 14. A CADA PERSONA: Por favor, dígame ¿cuál es su posición y el papel que desempeña en la implementación de Alerta Joven. 15. A CADA PERSONA: ¿Cuánto tiempo ha estado usted asociado/a al proyecto? 16. ¿Qué tan bien han funcionado en Alerta Joven las asociaciones público-privadas [por ejemplo: entre una empresa privada o un empleador y los compañeros de formación laboral de Alerta Joven que vinculan a los jóvenes con dicha empresa]? Por favor, proporcione algunos ejemplos de las asociaciones de más éxito y de las de menos éxito: [NOTA PARA EL COORDINADOR Rellenar la tabla en base a las respuestas dadas a la pregunta 3.] Las Organizaciones Socias/Subcontratados Asociaciones público￾privadas con más éxito Asociaciones público￾privadas con menos éxito Children International – Boca Chica ACOPRO 167 CAMINANTE Children International – Santo Domingo Este CSA MOSCHTA PROJECT HOPE IDDI PROFAMILIA FPUM FURJUG JADOM 17. ¿Qué podría hacerse para mejorar las asociaciones público-privadas? 18. ¿Cuáles programas [indicar: laboral/empresarial, de salud, educación] considera usted que han tenido más éxito en la reducción del número de jóvenes en situación de riesgo? ¿Cuáles factores han hecho que tengan más éxito que otros? 19. [SI NO SE HA MENCIONADO EN PREGUNTAS ANTERIORES] ¿Cuáles programas [indicar: laboral/empresarial, de salud, educación] considera usted que han tenido menos éxito en la reducción del número de jóvenes en situación de riesgo? a. ¿Cuáles factores han hecho que tengan menos éxito que otros? 20. En qué medida cree usted que son sostenibles los servicios ofrecidos a los jóvenes por parte de Entrena y sus Organizaciones Socias? a. En términos de recursos financieros [por ejemplo: tendrán las Organizaciones Socias los fondos necesarios para continuar con los servicios cuando concluya la financiación de Alerta Joven]? b. En términos de capacidad institucional? c. En términos de la captación local? [NOTA PARA EL COORDINADOR Rellenar la tabla en base a las respuestas dadas a la pregunta 7.] Servicios ofrecidos por las Organizaciones Socias/ subcontratados Recursos financieros para continuar los servicios? Capacidad institucional para continuar con los servicios? Captación local para continuar con los servicios? 168 Iniciativa empresarial Formación para la enseñanza profesional Asistencia en la documentación legal Espacios para crecer Promoción de políticas públicas Inserción escolar Mediación para jóvenes Microcrédito Prevención del VIH Prueba del VIH Retención escolar Servicios de salud Terapia psicológica 21. ¿Qué formación o apoyo adicional ayudaría a asegurar la sostenibilidad de estos servicios? 22. ¿En qué medida la formación en desarrollo de capacidades ofrecida por Entrena ha ayudado las Organizaciones Socias a mejorar sus capacidades técnicas, de seguimiento y evaluación, de gestión financiera, y el acceso a otras fuentes de financiación tras la conclusión de Alerta Joven? Por favor, proporcione ejemplos específicos para todas las Organizaciones Socias con base en Santo Domingo del que tenga conocimiento. 169 Las Organizaciones Socias/Subcontratados Capacidades técnicas mejoradas? Cómo? Capacidades de seguimiento y evaluación mejoradas? Cómo? Capacidades de gestión financiera mejoradas? Cómo? Capacidad de acceder a otras fuentes de financiación una vez concluida AJ? Children International – Boca Chica ACOPRO CAMINANTE Children International – Santo Domingo Este CSA MOSCHTA PROJECT HOPE IDDI PROFAMILIA FPUM FURJUG JADOM 170 23. ¿De qué formación o apoyo adicional se beneficiarían las Organizaciones Socias? 24. ¿Tiene usted una visión global positiva o negativa del proyecto Alerta Joven? Por favor, explique su respuesta. a. ¿Cómo ven el proyecto los jóvenes? b. ¿Cómo ven el proyecto otros socios (las Organizaciones Socias, empleadores, autoridades gubernamentales, la comunidad) con los que usted trabaja? c. ¿En qué medida estos beneficiarios y las partes interesadas valoran Alerta Joven? 25. ¿Cuál es su opinión global sobre el proyecto Alerta Joven (podría ser en términos de eficacia, eficiencia, gestión, coordinación, etc.)? 26. ¿Cuál es el consejo más importante que podría dar para mejorar el proyecto? 171 ENTREVISTA A INFORMANTES CLAVE DE ENTRENA (NOTA: a adaptar al especialista del sector que está siendo entrevistado.) Introducción y consentimiento Mi nombre es [____] y él/ella es [_____]. Somos representantes de Social Impact Inc., una empresa de investigación independiente contratada por la Agencia de los Estados Unidos para el Desarrollo Internacional (USAID). Estamos llevando a cabo una evaluación del desempeño del proyecto Alerta Joven para conocer mejor qué funciona y qué es lo que puede mejorarse en el proyecto, lo que ayudará a configurar la programación y enfoques futuros. Nos gustaría hacerle algunas preguntas acerca de su experiencia con Alerta Joven. Esta entrevista es voluntaria; puede abandonarla en cualquier momento o decidir no responder a cualquier pregunta. No hay respuestas correctas o incorrectas; nos interesan sus opiniones sinceras. Sus comentarios serán confidenciales y no se vincularán en modo alguno con usted o su organización en nuestro informe. Los comentarios que nos proporcione se analizarán con las respuestas de otras personas y grupos para alcanzar conclusiones generales. La entrevista no debería durar más de 60 minutos. Teniendo esto en cuenta, desea continuar con la entrevista? Nos gustaría incluirle como informante en un anexo de nuestro informe, junto con los nombres de otros informantes consultados para esta evaluación. ¿Está de acuerdo con que incluyamos su nombre en la lista? Por cierto, puede comunicarse con el equipo de evaluación después de la entrevista. Si tiene alguna pregunta sobre esta evaluación o desea proporcionar información adicional. Puede contactar a REDACTED. 172 Preguntas 1. A CADA PERSONA: Por favor, dígame ¿cuál es su posición y el papel que desempeña en la implementación de Alerta Joven? 2. A CADA PERSONA: ¿Cuánto tiempo ha estado usted asociado/a al proyecto? 3. [PARA AQUELLOS FAMILIARIZADOS CON PROGRAMAS DE EDUCACIÓN] ¿Cuál es la probabilidad de que los jóvenes beneficiarios de Alerta Joven regresen o permanezcan en la escuela? Por favor, explique su razonamiento. 4. [PARA AQUELLOS FAMILIARIZADOS CON PROGRAMAS DE SALUD] ¿Qué probabilidad hay de que los jóvenes beneficiarios de Alerta Joven reduzcan los comportamientos de riesgo (indicar: consumo de drogas y alcohol, actividad sexual, violencia, etc.)? Por favor, explique su razonamiento. 5. [PARA AQUELLOS FAMILIARIZADOS CON PROGRAMAS LABORALES] ¿Qué probabilidad hay de que los jóvenes beneficiarios de Alerta Joven consigan y mantengan un puesto de trabajo? Por favor, explique su razonamiento. 6. [PARA AQUELLOS FAMILIARIZADOS CON PROGRAMAS LABORALES] ¿Qué probabilidad hay de que los jóvenes beneficiarios de Alerta Joven inicien su propio negocio? Por favor, explique su razonamiento. 7. ¿Cuáles factores considera usted han contribuido con la posibilidad de que los jóvenes beneficiarios de Alerta Joven hagan lo siguiente? a. regresar o permanecer en la escuela; b. reducir los comportamientos de riesgo; y c. obtener y mantener puestos de trabajo o iniciar su propio negocio. Por favor, explique la influencia de Alerta Joven y las influencias externas o fuera del contexto del programa. 8. ¿Cuáles programas [indicar: laboral/empresarial, de salud, educación] cree usted han tenido más éxito en la reducción del número de jóvenes en situación de riesgo? Qué factores han hecho que tengan más éxito que otros? 9. [SI NO SE HA MENCIONADO EN PREGUNTAS ANTERIORES] ¿Cuáles programas (indicar: laboral/empresarial, de salud, educación) considera usted han tenido menos éxito en la reducción del número de jóvenes en situación de riesgo? Qué factores han hecho que tengan menos éxito que otros? 10. ¿En qué medida son sostenibles los servicios ofrecidos a los jóvenes por parte de Entrena y las Organizaciones de Entrena? a. ¿En términos de recursos financieros [por ejemplo: tendrán las Organizaciones los fondos necesarios para continuar con los servicios cuando concluya la financiación de Alerta Joven]? b. ¿En términos de capacidad institucional? c. ¿En términos de la captación local? Por favor, comente su razones y proporcione ejemplos. ¿De qué formación o apoyo adicional se beneficiarían las Organizaciones Socias? 11. ¿En qué medida la formación en desarrollo de capacidades ofrecida por Entrena ha ayudado a las Organizaciones Socias a: 173 a. mejorar sus capacidades técnicas; b. mejorar sus capacidades de gestión financiera; y c. acceder a otras fuentes de financiación una vez concluida Alerta Joven? Por favor, comente sus razones y proporcione ejemplos. De qué formación o apoyo adicional se beneficiarían las Organizaciones Socias? 12. ¿Tiene usted una visión global positiva o negativa del proyecto Alerta Joven? Por favor, explique su respuesta. a. Cómo ven el proyecto los jóvenes con los que trabaja? b. Cómo ven el proyecto otros socios (las Organizaciones Socias, empleadores, autoridades gubernamentales, la comunidad) con los que usted trabaja? c. En qué medida estos beneficiarios y las partes interesadas valoran Alerta Joven? 13. ¿Cuál es el consejo más importante que podría dar para mejorar el proyecto? 174 ENTREVISTA AL GRUPO DE LAS ORGANIZACIONES SOCIAS Y ENTREVISTA A INFORMANTES CLAVES Introducción y consentimiento Mi nombre es [____] y él/ella es [_____]. Somos representantes de Social Impact Inc., una empresa de investigación independiente contratada por la Agencia de los Estados Unidos para el Desarrollo Internacional (USAID). Estamos llevando a cabo una evaluación del desempeño del proyecto Alerta Joven para conocer mejor qué funciona y qué es lo que puede mejorarse en el proyecto, lo que ayudará a configurar la programación y enfoques futuros. Nos gustaría hacerle algunas preguntas acerca de su experiencia con Alerta Joven. Esta entrevista es voluntaria; puede abandonarla en cualquier momento o decidir no responder a cualquier pregunta. No hay respuestas correctas o incorrectas; nos interesan sus opiniones sinceras. Sus comentarios serán confidenciales y no se vincularán en modo alguno con usted o su organización en nuestro informe. Los comentarios que nos proporcione se analizarán con las respuestas de otras personas y grupos para alcanzar conclusiones generales. La entrevista no debería durar más de 60 minutos. Teniendo todo esto en cuenta, desea continuar con la entrevista? Nos gustaría incluirle como informante en un anexo a nuestro informe, junto con los nombres de otros informantes consultados para esta evaluación. Está conforme con que incluyamos su nombre en la lista? Por cierto, puede comunicarse con el equipo de evaluación después de la entrevista si tiene alguna pregunta sobre esta evaluación o desea proporcionar información adicional. Puede contactar a REDACTED. [ENTREVISTADOR: Iniciar la discusión de grupo o entrevista a informantes clave entregando la MINI￾ENCUESTA a cada encuestado. En la propia encuesta se halla una explicación de la finalidad de la mini-encuesta; puedes explicar el propósito de la mini-encuesta a los participantes. Una vez se haya completado la mini-encuesta, comenzar con las preguntas a continuación.] 175 Preguntas 1. A CADA PERSONA: Por favor, ¿dígame cuál es su posición y el papel que desempeña en la implementación de Alerta Joven. 2. A CADA PERSONA: Cuánto tiempo ha estado usted asociado/a al proyecto? 3. En qué medida cree usted que su organización será capaz de continuar sus servicios para jóvenes [indicar: en cuestión de financiación y capacidad institucional] cuando cese el apoyo de Alerta Joven? Desarrolle su respuesta. a. ¿Cómo ha cambiado, en su caso, la capacidad de su organización de ofrecer estos servicios durante el proyecto Alerta Joven? ¿Cuáles factores han contribuido a estos cambios? 4. [PARA LAS ACTIVIDADES RELACIONADAS CON LA EDUCACIÓN] ¿Qué probabilidad hay de que los jóvenes beneficiarios de Alerta Joven regresen o permanezcan en la escuela? Por favor, explique su razonamiento. a. En ese sentido ¿Cómo ves los jóvenes de Alerta Joven en comparación con los jóvenes que no participan en el proyecto? Por favor, proporcione ejemplos concretos. b. ¿A qué se debe la diferencia (por favor, refiérase tanto a Alerta Joven como a los factores externos)? 5. [PARA LAS ACTIVIDADES RELACIONADAS CON LA SALUD] ¿Qué probabilidad hay de que los jóvenes beneficiarios de Alerta Joven reduzcan los comportamientos de riesgo (indicar: consumo de drogas y alcohol, actividad sexual, violencia, etc.)? Por favor, explique su razonamiento. a. ¿Cómo considera que son los jóvenes de Alerta Joven en este respecto, en comparación con los jóvenes que no participan en las intervenciones de Alerta Joven? Por favor, proporcione ejemplos concretos. b. ¿A qué se debe la diferencia (refiérase tanto a Alerta Joven como a los factores externos)? 6. [PARA LAS ACTIVIDADES RELACIONADAS CON EL DESARROLLO LABORAL] ¿Qué probabilidad hay de que los jóvenes beneficiarios de Alerta Joven consigan y mantengan un puesto de trabajo? Por favor, explique su razonamiento. a. En ese sentido, ¿Cómo ves los jóvenes de Alerta Joven en comparación con los jóvenes que no participan en las intervenciones de Alerta Joven? Por favor, proporcione ejemplos concretos. b. ¿A qué se debe la diferencia (refiérase tanto a Alerta Joven como a los factores externos)? 7. [PARA LAS ACTIVIDADES RELACIONADAS CON EL DESARROLLO LABORAL] ¿Qué tan bien han funcionado las asociaciones público-privadas en Alerta Joven? Por favor, proporcione algunos ejemplos de las asociaciones de más éxito y de las de menos éxito: ¿Qué podría hacerse para mejorar las asociaciones público-privadas? 8. [PARA LAS ACTIVIDADES RELACIONADAS CON LA INICIATIVA EMPRESARIAL] ¿Qué probabilidad hay de que los jóvenes beneficiarios de Alerta Joven inicien su propio negocio? Por favor, comente su razones. a. En ese mismo sentido ¿Cómo ves a los jóvenes de Alerta Joven en comparación con los jóvenes que no participan en las intervenciones de Alerta Joven? Por favor, proporcione ejemplos concretos. 176 b. ¿A qué se debe la diferencia (refiérase tanto a Alerta Joven como a los factores externos)? 9. ¿Cuáles aspectos de la programación [indicar TODO LO QUE SEA APLICABLE: laboral/salud/educación] han tenido más éxito en la reducción del número de jóvenes en situación de riesgo? ¿Qué ha hecho que tengan más éxito que otros? 10. ¿Qué partes de la programación [indicar TODO LO QUE SEA APLICABLE: laboral/salud/educación] han tenido menos éxito en la reducción del número de jóvenes en situación de riesgo? Qué ha hecho que tengan menos éxito que otros? 11. Cómo han cambiado, en su caso, las capacidades técnicas de su organización (en gestión de personal, mantenimiento de registros, escritos de propuesta, etc.) durante la implementación del proyecto Alerta Joven? ¿Qué factores han contribuido a estos cambios? 12. [SI NO HA SURGIDO YA] ¿Qué tan efectiva ha sido la formación en desarrollo de capacidades de ENTRENA en la mejora de las capacidades técnicas de su organización? ¿De qué maneras? ¿Qué formación adicional u otro apoyo estaría interesado en recibir? 13. ¿Cómo han cambiado, en su caso, las capacidades financieras de su organización de ofrecer estos servicios durante la implementación del proyecto Alerta Joven? Qué factores han contribuido a estos cambios? 14. [SI NO HA SURGIDO YA] Cómo de efectiva ha sido la formación de ENTRENA en la mejora de las capacidades financieras de su organización? De qué maneras? Qué formación o apoyo financiero adicional estaría interesado en recibir? 15. En qué medida ha afectado, en su caso, el apoyo de Entrena a otras fuentes de financiación? De qué manera? 16. Qué recursos, acuerdos u otros elementos necesita su organización para poder seguir proporcionando servicios a los jóvenes cuando finalice el proyecto Alerta Joven? 17. Tiene una visión global positiva o negativa del proyecto Alerta Joven? Por favor, explique su respuesta. a. Cómo ven el proyecto los jóvenes con los que trabaja? b. Cómo ven el proyecto otros socios (las Organizaciones Socias, empleadores, autoridades gubernamentales, la comunidad) con los que usted trabaja? c. En qué medida estos beneficiarios y las partes interesadas valoran Alerta Joven? 18. Cuál es el consejo más importante que podría dar para mejorar el proyecto? 177 ENTREVISTA DE GRUPO DE ORGANIZACIONES BASADAS EN COMUNIDAD (OBC) Y ORGANIZACIONES DE LA SOCIEDAD CIVIL (OSC) Introducción y consentimiento Mi nombre es [____] y él/ella es [_____]. Somos representantes de Social Impact Inc., una empresa de investigación independiente contratada por la Agencia de los Estados Unidos para el Desarrollo Internacional (USAID). Estamos llevando a cabo una evaluación del desempeño del proyecto Alerta Joven para conocer mejor qué funciona y qué es lo que puede mejorarse en el proyecto, lo que ayudará a configurar la programación y enfoques futuros. Nos gustaría hacerle algunas preguntas acerca de su experiencia con Alerta Joven. Esta entrevista es voluntaria; puede abandonarla en cualquier momento o decidir no responder a cualquier pregunta. No hay respuestas correctas o incorrectas; nos interesan sus opiniones sinceras. Sus comentarios serán confidenciales y no se vincularán en modo alguno con usted o su organización en nuestro informe. Los comentarios que nos proporcione se analizarán con las respuestas de otras personas y grupos para alcanzar conclusiones generales. La entrevista no debería durar más de 60 minutos. Teniendo todo esto en cuenta, desea continuar con la entrevista? Nos gustaría incluirle como informante en un anexo a nuestro informe, junto con los nombres de otros informantes consultados para esta evaluación. Está conforme con que incluyamos su nombre en la lista? Por cierto, puede comunicarse con el equipo de evaluación después de la entrevista si tiene alguna pregunta sobre esta evaluación o desea proporcionar información adicional. Puede contactar a REDACTED. 178 Preguntas 1. Por favor, dígame, ¿cuál es su posición y cuál es su asociación con Alerta Joven? 2. ¿Por cuánto tiempo ha estado usted asociado/a al proyecto? 3. [PARA LAS ACTIVIDADES RELACIONADAS CON LA EDUCACIÓN] ¿Qué probabilidad hay de que los jóvenes beneficiarios de Alerta Joven regresen o permanezcan en la escuela? Por favor, explique su razonamiento. a. ¿Cómo considera que son los jóvenes de Alerta Joven a este respecto en comparación con los jóvenes que no participan en las intervenciones de Alerta Joven? Por favor, proporcione ejemplos concretos. b. A qué se debe la diferencia (refiérase tanto a Alerta Joven como a los factores externos)? 4. [PARA LAS ACTIVIDADES RELACIONADAS CON LA SALUD] Qué probabilidad hay de que los jóvenes beneficiarios de Alerta Joven reduzcan los comportamientos de riesgo (indicar: consumo de drogas y alcohol, actividad sexual, violencia, etc.)? Por favor, explique su razonamiento. a. En ese respecto, ¿Cómo ves los jóvenes de Alerta Joven en comparación con los jóvenes que no participan en las intervenciones de Alerta Joven? Por favor, proporcione ejemplos concretos. b. ¿A qué se debe la diferencia (refiérase tanto a Alerta Joven como a los factores externos)? 5. [PARA LAS ACTIVIDADES RELACIONADAS CON EL DESARROLLO LABORAL] ¿Qué probabilidad hay de que los jóvenes beneficiarios de Alerta Joven consigan y mantengan un puesto de trabajo? Por favor, explique su razonamiento. a. ¿Cómo considera que son los jóvenes de Alerta Joven a este respecto en comparación con los jóvenes que no participan en las intervenciones de Alerta Joven? Por favor, proporcione ejemplos concretos. b. ¿A qué se debe la diferencia (refiérase tanto a Alerta Joven como a los factores externos)? 6. [PARA LAS ACTIVIDADES RELACIONADAS CON LA INICIATIVA EMPRESARIAL] ¿Qué probabilidad hay de que los jóvenes beneficiarios de Alerta Joven inicien su propio negocio? Por favor, explique su razonamiento. a. ¿Cómo considera que son los jóvenes de Alerta Joven a este respecto en comparación con los jóvenes que no participan en las intervenciones de Alerta Joven? Por favor, proporcione ejemplos concretos. b. ¿A qué se debe la diferencia (refiérase tanto a Alerta Joven como a los factores externos)? 7. ¿Cuáles partes de la programación [indicar TODO LO QUE SEA APLICABLE: laboral/salud/educación] han tenido más éxito en la reducción del número de jóvenes en situación de riesgo? 8. ¿Cuáles partes de la programación [indicar TODO LO QUE SEA APLICABLE: laboral/salud/educación] han tenido menos éxito en la reducción del número de jóvenes en situación de riesgo? 9. ¿En qué medida cree que se mantendrán los beneficios para los jóvenes cuando finalice el proyecto Alerta Joven? ¿Para la comunidad? Por favor, explique su respuesta. a. ¿Qué recursos, acuerdos u otros elementos (políticas, formación, etc.) son necesarios para hacerlos más sostenibles? 179 10. Alerta Joven se encuentra ya en su 4º año de existencia. ¿Cómo cree que han cambiado en este tiempo los problemas de los jóvenes (por ejemplo, participación en comportamientos de riesgo, falta de educación, desempleo) en esta comunidad? ¿A qué se deben estos cambios (por favor, refiérase tanto a Alerta Joven como a los factores externos)? 11. En general, ¿cómo cree percibe la comunidad el proyecto Alerta Joven? ¿En qué medida valora la comunidad el proyecto Alerta Joven? ¿Qué podría hacer el proyecto para mejorar la percepción y la valoración de la comunidad? 12. ¿Cuáles recomendaciones tiene para mejorar Alerta Joven? 13. Por favor, comparta cualquier otra experiencia o ejemplos que proporcionen una imagen clara de cómo funciona Alerta Joven. 180 ENTREVISTA A INFORMANTES CLAVE: REPRESENTANTES ESCOLARES Introducción y consentimiento Mi nombre es [____] y él/ella es [_____]. Somos representantes de Social Impact Inc., una empresa de investigación independiente contratada por la Agencia de los Estados Unidos para el Desarrollo Internacional (USAID). Estamos llevando a cabo una evaluación del desempeño del proyecto Alerta Joven para conocer mejor qué funciona y qué es lo que puede mejorarse en el proyecto, lo que ayudará a configurar la programación y enfoques futuros. Nos gustaría hacerle algunas preguntas acerca de su experiencia con Alerta Joven. Esta entrevista es voluntaria; puede abandonarla en cualquier momento o decidir no responder a cualquier pregunta. No hay respuestas correctas o incorrectas; nos interesan sus opiniones sinceras. Sus comentarios serán confidenciales y no se vincularán en modo alguno con usted o su organización en nuestro informe. Los comentarios que nos proporcione se analizarán con las respuestas de otras personas y grupos para alcanzar conclusiones generales. La entrevista no debería durar más de 60 minutos. Teniendo esto en cuenta, ¿desea continuar con la entrevista? Nos gustaría incluir su nombre como informante en un anexo a nuestro informe, junto con los nombres de otros informantes consultados para esta evaluación. Está conforme con que incluyamos su nombre en la lista? Le preguntaremos cifras concretas de estudiantes matriculados en su escuela con Alerta Joven. Tiene esta información disponible? [SI ESTA INFORMACIÓN NO ESTÁ DISPONIBLE] ¿Podrá estimar estos números o facilitármelos en algún momento de esta semana? Por cierto, puede comunicarse con el equipo de evaluación después de la entrevista. Si tiene alguna pregunta sobre esta evaluación o desea proporcionar información adicional puede contactar a REDACTED. 181 Preguntas 1. Por favor, dígame ¿cuál es su posición? 2. ¿Tiene algún conocimiento del proyecto Alerta Joven? Si es así, ¿cuál es su asociación con Alerta Joven? a. ¿Por cuánto tiempo ha estado usted asociado/a al proyecto? 3. Aproximadamente, ¿cuántos estudiantes entre 11 y 24 años de edad están matriculados en su escuela? Aproximadamente ¿qué porcentaje cree usted que ha participado en el programa de Alerta Joven? 4. ¿Cómo cree rinden los jóvenes que han recibido servicios de Alerta Joven en comparación con sus compañeros no implicados con Alerta Joven, en los siguientes ámbitos? Qué factores explican estas diferencias? a. Matriculación b. Asistencia c. Desempeño académico d. Comportamiento social (por ejemplo: consumo de drogas y alcohol, actividad sexual, violencia, problemas legales, etc.) 5. Teniendo en cuenta sus respuestas a la pregunta anterior, ¿qué probabilidad hay de que los jóvenes beneficiarios de Alerta Joven regresen o permanezcan en la escuela? Por favor, explique su razonamiento. 6. ¿En qué medida cree que los jóvenes participantes mantendrán los beneficios de Alerta Joven cuando finalice el proyecto Alerta Joven? Por favor, explique su respuesta. ¿Qué recursos, acuerdos u otros elementos (políticas, formación, etc.) son necesarios para hacerlos más sostenibles? 7. Alerta Joven se encuentra ya en su 4º año de existencia. ¿Cómo cree que han cambiado en este tiempo los problemas de los jóvenes (por ejemplo, participación en comportamientos de riesgo, falta de educación, desempleo) en esta comunidad? ¿A qué se deben estos cambios (por favor, refiérase tanto a Alerta Joven como a los factores externos)? 8. En general, ¿cómo cree que se percibe el proyecto por la comunidad? ¿En qué medida es valorado el proyecto por la comunidad? ¿Qué se podría hacer desde el proyecto para mejorar la percepción y la valoración de la comunidad? 9. En general, ¿cómo cree que es percibido el proyecto por los jóvenes? ¿En qué medida valoran los jóvenes el proyecto? ¿Qué podría hacer el proyecto para mejorar la percepción y la valoración? 10. ¿Cuáles recomendaciones tiene para mejorar Alerta Joven? 11. Por favor, comparta cualquier otra experiencia o ejemplos que proporcionen una imagen clara de cómo funciona Alerta Joven. 182 ENTREVISTA A INFORMANTES CLAVE: EMPLEADORES Introducción y consentimiento Mi nombre es [____] y él/ella es [_____]. Somos representantes de Social Impact Inc., una empresa de investigación independiente contratada por la Agencia de los Estados Unidos para el Desarrollo Internacional (USAID). Estamos llevando a cabo una evaluación del desempeño del proyecto Alerta Joven para conocer mejor qué funciona y qué es lo que puede mejorarse en el programa, lo que ayudará a configurar la programación y enfoques futuros. Nos gustaría hacerle algunas preguntas acerca de su experiencia con Alerta Joven. Esta entrevista es voluntaria; puede abandonarla en cualquier momento o decidir no responder a cualquier pregunta. No hay respuestas correctas o incorrectas; solo nos interesan sus opiniones sinceras. Sus comentarios serán confidenciales y no se vincularán en modo alguno con usted o su organización en nuestro informe. Los comentarios que nos proporcione se analizarán con las respuestas de otras personas y grupos para alcanzar conclusiones generales. La entrevista no debería durar más de 60 minutos. Teniendo esto en cuenta, desea continuar con la entrevista? Nos gustaría incluirle como informante en un anexo a nuestro informe, junto con los nombres de otros informantes consultados para esta evaluación. Está conforme con que incluyamos su nombre en la lista? Por cierto, puede comunicarse con el equipo de evaluación después de la entrevista si tiene alguna pregunta sobre esta evaluación o desea proporcionar información adicional. Puede contactar a REDACTED. 183 Preguntas 1. Por favor, dígame ¿cuál es su posición y cuál es su asociación con Alerta Joven/[ORGANIZACIÓN SOCIA]? 2. ¿Cuánto tiempo ha estado usted asociado/a al proyecto? 3. ¿Cuánta gente emplea usted? ¿Cuántos de sus empleados tienen menos de 26 años de edad? De los empleados menores de 26 años, ¿cuántos de ellos son jóvenes que han recibido la formación de Alerta-Joven/[ORGANIZACIÓN SOCIA]? [CALCULAR PORCENTAJE]? a. Por favor, confirme: aproximadamente el [INSERTAR PORCENTAJE] de los empleados menores de 26 años son jóvenes que han recibido formación de Alerta Joven, es correcto? b. ¿Para qué tipo de puestos emplea habitualmente a los jóvenes que han recibido formación de Alerta Joven/[ORGANIZACIÓN SOCIA] (administrativos, baja/alta cualificación, etc.), tanto en el caso de los jóvenes que han trabajado para usted en el pasado como en el de los que trabajan actualmente? 4. ¿Cómo son los jóvenes que han recibido formación de Alerta Joven/[ORGANIZACIÓN SOCIA] en comparación con otros jóvenes empleados (sin formación de Alerta Joven) en cuanto a la idoneidad de sus competencias para los puestos para los que se les ha contratado? ¿Hay otras competencias de las que carezca Alerta Joven/[ORGANIZACIÓN SOCIA], pero que necesite en sus empleados? 5. En base a su experiencia, ¿qué probabilidad hay de que los jóvenes beneficiarios de Alerta Joven/[ORGANIZACIÓN SOCIA] sigan manteniendo y consiguiendo puestos de trabajo? Por favor, explique su respuesta. 6. ¿Qué tan efectiva ha sido su asociación empresarial con Alerta Joven/[ORGANIZACIÓN SOCIA]? (indicar: coinciden las competencias de los jóvenes con las necesidades del empleador, eficacia del proceso, adecuación de las necesidades y evaluaciones de las competencias, etc.) ¿Qué tan efectiva diría usted que ha sido en general la asociación entre Alerta Joven y el sector privado (indicar: coinciden las competencias de los jóvenes con las necesidades del empleador, eficacia del proceso, adecuación de las necesidades y evaluaciones de las competencias, etc.)? Por favor, proporcione algunos ejemplos. ¿Que podría hacer Alerta Joven/[ORGANIZACIÓN SOCIA] u otros para mejorar la relación del proyecto con empresas como la suya? 7. [SI NO SE HA RESPONDIDO YA] Por favor, mencione cualquier carencia de competencias en los jóvenes que han recibido formación de Alerta Joven/[ORGANIZACIÓN SOCIA] respecto a lo que empresas como la suya necesita. a. ¿Cómo estas carencias de competencias se comparan aquellas carencias de competencias entre los jóvenes antes de que empezara el programa Alerta Joven en 2011? b. ¿Hay diferencias entre los jóvenes que participan en la formación de Alerta Joven/[ORGANIZACIÓN SOCIA] y los jóvenes que han participado en otras formaciones (distintas a Alerta Joven/[ORGANIZACIÓN SOCIA])? c. ¿Qué aspectos o partes de las formaciones de Alerta Joven/[ORGANIZACIÓN SOCIA] hacen que tengan más o menos éxito en la preparación de los jóvenes para el empleo? 8. ¿En qué medida cree usted que continuarán las formaciones en desarrollo laboral que Alerta Joven/[ORGANIZACIÓN SOCIA] ha apoyado después de que termine el proyecto? Por favor, explique su respuesta. ¿Qué será necesario para asegurar que puedan continuar? 184 9. ¿En qué medida cree usted que los jóvenes retendrán las competencias de desarrollo laboral que han adquirido gracias a Alerta Joven/[ORGANIZACIÓN SOCIA]? Por favor, explique su respuesta. 10. Si tuviera que ofrecer algún consejo a Alerta Joven/[ORGANIZACIÓN SOCIA] para mejorar su formación en preparación laboral de los jóvenes, ¿cuál sería? 11. En general, ¿cómo siente usted que el sector privado percibe Alerta Joven/[ORGANIZACIÓN SOCIA]? En qué medida valoran los empleadores el proyecto? Por favor, proporcione ejemplos concretos. ¿Qué podría hacer el programa para mejorar las percepciones y la valoración? 12. Por favor, comparta cualquier otra experiencia o ejemplos que proporcionen una imagen clara de cómo funciona Alerta Joven/[ORGANIZACIÓN SOCIA] con los empleadores en la comunidad. 185 ENTREVISTA A INFORMANTES CLAVE: ORGANISMOS GUBERNAMENTALES Introducción y consentimiento Mi nombre es [____] y él/ella es [_____]. Somos representantes de Social Impact Inc., una empresa de investigación independiente contratada por la Agencia de los Estados Unidos para el Desarrollo Internacional (USAID). Estamos llevando a cabo una evaluación del desempeño del proyecto Alerta Joven para conocer mejor qué funciona y qué es lo que puede mejorarse en el proyecto, lo que ayudará a configurar la programación y enfoques futuros. Nos gustaría hacerle algunas preguntas acerca de su experiencia con Alerta Joven. Esta entrevista es voluntaria; puede abandonarla en cualquier momento o decidir no responder a cualquier pregunta. No hay respuestas correctas o incorrectas; nos interesan sus opiniones sinceras. Sus comentarios serán confidenciales y no se vincularán en modo alguno con usted o su organización en nuestro informe. Los comentarios que nos proporcione se analizarán con las respuestas de otras personas y grupos para alcanzar conclusiones generales. La entrevista no debería durar más de 60 minutos. Teniendo esto en cuenta, ¿desea continuar con la entrevista? Nos gustaría incluirle como informante en un anexo a nuestro informe, junto con los nombres de otros informantes consultados para esta evaluación. Está conforme con que incluyamos su nombre en la lista? Por cierto, puede comunicarse con el equipo de evaluación después de la entrevista. Si tiene alguna pregunta sobre esta evaluación o desea proporcionar información adicional puede contactar a REDACTED. 186 Preguntas 1. Por favor, dígame, ¿cuál es su posición y cuál es su asociación con Alerta Joven? 2. ¿Por cuánto tiempo ha estado usted asociado/a al proyecto? 3. Según su experiencia con los programas de Alerta Joven, ¿cuáles partes de la programación [indicar TODO LO QUE SEA APLICABLE: laboral/salud/educación] han tenido más éxito en la reducción del número de jóvenes en situación de riesgo? 4. Qué partes de la programación [indicar TODO LO QUE SEA APLICABLE: laboral/salud/educación] han tenido menos éxito en la reducción del número de jóvenes en situación de riesgo? 5. ¿Cómo considera que son los jóvenes de Alerta Joven en comparación con los jóvenes que no participan en las intervenciones de Alerta Joven? Por favor, proporcione ejemplos concretos. ¿A qué se debe la diferencia (por favor, refiérase tanto a Alerta Joven como a los factores externos)? 6. ¿En qué medida cree que se mantendrán los beneficios para los jóvenes cuando finalice el proyecto Alerta Joven? ¿Para la comunidad? Por favor, explique su respuesta. ¿Qué recursos, acuerdos u otros elementos (políticas, formación, etc.) son necesarios para hacerlos más sostenibles? 7. En qué medida cree usted que los servicios [indicar: laborales/de salud/de educación] que Alerta Joven ha estado apoyando continuarán después de que termine el proyecto? Por favor, explique su respuesta. Qué será necesario para asegurar que puedan continuar? 8. ¿Cómo ve el trabajo que ENTRENA lleva realizando para desarrollar capacidades de las Organizaciones Socias (ONGs locales) para llevar a cabo el trabajo futuro? Este trabajo en desarrollo de capacidades, ¿ha tenido más éxito o menos éxito, y en qué modo? Por favor, proporcione ejemplos concretos de los que tenga constancia. 9. Alerta Joven se encuentra ya en su 4º año de existencia. ¿Cómo cree que han cambiado en este tiempo los problemas de los jóvenes (por ejemplo, participación en comportamientos de riesgo, falta de educación, desempleo) en esta comunidad? ¿A qué se deben estos cambios (por favor, refiérase tanto a Alerta Joven como a los factores externos)? 10. ¿Tiene una visión global positiva o negativa del proyecto Alerta Joven? Por favor, explique su respuesta. ¿Cómo ven el proyecto otros Ministerios implicados [indicar: Ministerio de la Juventud, Oficina del Fiscal, Ministerio de la Presidencia, Ministerio del Trabajo, Instituto Nacional de Formación Profesional]? 11. En general, ¿cómo siente que la comunidad percibe el proyecto Alerta Joven? ¿Qué podría hacer el proyecto para mejorar la percepción y valoración de la comunidad? 12. ¿Qué recomendaciones tiene para mejorar Alerta Joven? 13. Por favor, comparta cualquier otra experiencia o ejemplos que proporcionen una imagen clara de cómo funciona Alerta Joven. 187 PROTOCOLO DE GRUPO FOCAL DE JÓVENES DE ALERTA JOVEN Introducción y consentimiento Mi nombre es [____] y él/ella es [_____]. Somos representantes de Social Impact Inc., una empresa de investigación independiente contratada por la Agencia de los Estados Unidos para el Desarrollo Internacional (USAID). Estamos haciendo un estudio sobre el proyecto Alerta Joven con el fin de conocer que ha ido bien y que podría haber ido mejor. Esta información ayudará a USAID a desarrollar otros programas juveniles. Nos gustaría hacerles algunas preguntas acerca de su experiencia con Alerta Joven. Pueden abandonar el grupo en cualquier momento u optar por no responder a cualquier pregunta. No hay respuestas correctas o incorrectas; nos interesan sus opiniones e ideas sinceras. Queremos oírlos a todos, y les pediremos que respeten las opiniones de los demás en todo momento; lo que significa que no pueden burlarse o molestar a nadie por lo que haya dicho. Puede que en algún momento actúe de policía de tráfico animando a hablar a alguien que ha estado callado, o pidiendo a alguien que se espere unos minutos. La entrevista de grupo no debería tardar más de una hora y media. Nos gustaría grabar esta conversación para que no se pierda ninguno de sus comentarios; no obstante, todo lo que digan se mantendrá confidencial. Nadie sabrá lo que "él" o "ella" ha dicho (señalar a un par de personas) fuera de este grupo. No incluiremos sus nombres en ningún informe. También pedimos a cada uno de ustedes que no comparta con otras personas, después de esta sesión, ninguna información sobre lo que cualquiera haya dicho en este grupo. Por cierto, después pueden comunicarse con el equipo de evaluación si tiene alguna pregunta sobre este estudio o desean proporcionar información adicional. Pueden contactar a REDACTED. Teniendo esto en cuenta, ¿está todo el mundo de acuerdo en participar? Si les parece bien, vamos a iniciar la grabadora y comenzar esta sesión. 188 Instrucciones Las discusiones de grupo focal se estratifican por rangos de edad; 11-17 o 18-26. Cada discusión de grupo incluirá un facilitador Ipsos (como líder) y un facilitador SI (como apoyo). El facilitador Ipsos debe conducir la mayor parte de la discusión de grupo para establecer una relación positiva con los jóvenes, mientras que el facilitador SI debe sentirse libre de intervenir con preguntas pertinentes. Hacer que los grupos jueguen a un juego corto a modo de calentamiento previo y para que se sientan cómodos (las ideas se facilitarán en la formación del facilitador) durante 10-15 minutos. 1. ¿Te acuerdas de cuando empezaste AJ? Si la respuesta es sí, describe como eras entonces. Descríbete ahora. ¿Qué ha cambiado? ¿Qué no ha cambiado? Dependiendo del grupo, si parecen dispuestos, proporcione a cada joven una hoja de papel para dibujar con lápices de colores o marcadores y pídeles que muestren visualmente lo diferentes que son ahora respecto a cómo eran antes de su participación en Alerta Joven. Pídeles que muestren concretamente cómo ha cambiado cualquiera de sus ideas o comportamientos sobre: educación, deportes, amigos, empleo, autoempleo y también cómo perciben a su familia y su comunidad (de manera positiva o negativa). Pueden usar imágenes y palabras. Dales 15-20 minutos. Vuelve a reunir al grupo y haz que debatan los puntos claves que quieren expresar a partir de lo que han dibujado; anótalos en un rotafolio. Si el grupo parece no querer participar en esta actividad (quizá sea más probable en jóvenes de mayor edad), pueden expresarse oralmente en el grupo. Indicar para esta pregunta: a. ¿Asistías a la escuela? ¿Qué calificaciones obtenías? ¿Conseguiste un puesto de trabajo? ¿Comenzaste tu propio negocio? ¿Te arrestaron? ¿Tomaste decisiones diferentes respecto a tus relaciones sexuales? ¿Cambiaste tus patrones de consumo de alcohol o drogas? Etc. Las preguntas de seguimiento, a medida que dan sus respuestas pueden ser: 2. ¿Qué causó estos cambios y de qué modo? a. ¿Cómo te ayudó Alerta Joven a aprender algo nuevo o a hacer un cambio en tu vida? (Posibles indicaciones: con respecto a la escuela, calificaciones, asistencia, conseguir un trabajo, crear tu propio negocio, problemas legales/arrestos, diferentes patrones de relaciones sexuales, bebida o consumo de drogas, etc.) b. ¿Cómo llegaste a conocer a gente nueva a través de Alerta Joven? c. ¿Qué hizo Alerta Joven para ayudarte a ver a tu familia o comunidad de manera diferente? Haz fotos de sus dibujos para conservar como registro de recopilación de datos y deja que se queden sus dibujos si lo desean. 189 3. ¿Qué te gusta de Alerta Joven (indicar: juegos, actividades, personal, formación)? 4. ¿Qué no te gusta de Alerta Joven (indicar: no hay suficientes actividades, no hay comida o bebidas, no es en verano)? 5. ¿Qué piensan tus amigos acerca de Alerta Joven? ¿Qué piensan sobre el programa otras personas de la comunidad (padres, personas, parientes)? ¿Qué les gusta o no les gusta? 6. ¿Qué dicen tus amigos y compañeros de clase acerca de tu participación en AJ? ¿Qué dicen tus padres, profesores y parientes acerca de tu participación en AJ? ¿Cómo te hace sentir lo que dicen acerca de tu participación en AJ? 7. ¿Qué te gustaría que cambiase en el proyecto para mejorarlo? a. ¿Cuáles programas o servicios te gustaría ver para los jóvenes de tu comunidad? 8. Si tuvieras que contar a un amigo que no participa acerca de Alerta Joven, ¿cuáles son las tres palabras que utilizarías para describir el programa? 9. ¿Tiene alguien otra opinión o sentimiento que quieren compartir acerca de Alerta Joven? 190 MINI ENCUESTA SOBRE CAPACIDAD DE LAS ORGANIZACIONES SOCIAS NOTA PARA EL ENCUESTADOR (NO LEER AL PARTICIPANTE) El propósito de esta mini encuesta es proporcionar una medida cuantitativa de la capacidad de gestión técnica y financiera de las organizaciones socias que sirva como dato para responder a la pregunta de evaluación 4. Este breve cuestionario se administrará al inicio de las entrevistas de informantes claves y discusiones en grupo con las Organizaciones Socias, como una forma de introducir la conversación, y se triangulará con datos de preguntas similares recogidas de ENTRENA. EQ4: ¿En qué medida el entrenamiento en desarrollo de capacidades ofrecido por ENTRENA ayudó a la Organización Socia a mejorar capacidades de gestión técnica y financiera que puedan ayudarles a acceder a otras fuentes de financiación una vez que haya finalizado Alerta Joven? CLÁUSULA DE CONSENTIMIENTO (LEER AL PARTICIPANTE) Mi nombre es [____] y él/ella es [_____]. Representamos a Social Impact Inc., una empresa de investigación independiente contratada por la Agencia de los Estados Unidos para el Desarrollo Internacional. Estamos realizando una evaluación de desempeño del proyecto Alerta Joven para comprender mejor qué funciona y qué puede mejorarse en el proyecto, lo que será de gran ayuda para la programación y enfoques futuros. Nos gustaría hacerle algunas preguntas sobre su experiencia con Alerta Joven. La entrevista es voluntaria; puede abandonarla en cualquier momento o decidir no responder a alguna pregunta. Tenga en cuenta que no hay respuestas acertadas o equivocadas y que estamos interesados en sus opiniones sinceras. Sus comentarios se mantendrán confidenciales y no se vincularán en modo alguno con usted o con su organización en nuestro informe. Las opiniones que nos proporcione se analizarán con las respuestas de otras personas y grupos para lograr hallazgos generales. Esta encuesta no debería llevarle más de 10 minutos y la entrevista completa no debería durar más de 60 minutos. Teniendo en cuenta esta información, ¿está de acuerdo en seguir con la entrevista? En caso afirmativo, nos gustaría citarle como informante en el anexo de nuestro informe junto a los nombres de los demás informantes consultados para esta evaluación. ¿Está de acuerdo en que incluyamos su nombre en esa lista? Por cierto, puede ponerse en contacto con el equipo de evaluación después de la entrevista en caso de que tenga preguntas sobre esta evaluación o quiera proporcionar información adicional. Para tales fines, puede ponerse en contacto con REDACTED. 191 Pregunta 1: ¿Cómo ha cambiado, en su caso, la capacidad de gestión económica de su organización respecto a antes de recibir la formación de ENTRENA? 1 2 3 4 5 Ha disminuido mucho Ha disminuido algo Sin cambios Ha aumentado algo Ha aumentado mucho Pregunta 2: ¿Cómo ha cambiado, en su caso, la capacidad de gestión de personal de su organización respecto a antes de recibir la formación de ENTRENA? 1 2 3 4 5 Ha disminuido mucho Ha disminuido algo Sin cambios Ha aumentado algo Ha aumentado mucho Pregunta 3: ¿Cómo ha cambiado, en su caso, la capacidad de gestión de los datos y el mantenimiento de registros de su organización respecto a antes de recibir la formación de ENTRENA? 1 2 3 4 5 Ha disminuido mucho Ha disminuido algo Sin cambios Ha aumentado algo Ha aumentado mucho Pregunta 4: ¿Cómo ha cambiado, en su caso, la capacidad de escribir propuestas para nueva financiación respecto a antes de recibir la formación de ENTRENA? 1 2 3 4 5 Ha disminuido mucho Ha disminuido algo Sin cambios Ha aumentado algo Ha aumentado mucho Pregunta 5: ¿Cómo ha cambiado, en su caso, la capacidad de supervisión y evaluación de su organización respecto a antes de recibir la formación de ENTRENA? 1 2 3 4 5 Ha disminuido mucho Ha disminuido algo Sin cambios Ha aumentado algo Ha aumentado mucho Pregunta 6: ¿En qué medida está de acuerdo con la siguiente afirmación? "Debido a la formación en desarrollo de capacidades que mi empresa/organismo ha recibido de ENTRENA, percibo que mi empresa/organismo se encuentra en mejores condiciones para acceder a nuevas fuentes de financiación". 1 2 3 4 5 Totalmente en desacuerdo Parcialmente en desacuerdo Ni de acuerdo ni en desacuerdo Parcialmente de acuerdo Totalmente de acuerdo 192 ANNEX E: QUALITATIVE ANALYSIS CODEBOOK Code Description 1. Sector-Specific Outcomes EQ1: What is the likelihood that Alerta Joven youth beneficiaries will: return to and stay in school; reduce risky behaviors; obtain and maintain jobs; and, start their own business? 1.1 Educational outcomes Discussion of educational outcomes for AJ youth. This code should only be used if the educational outcomes are not included in the sub￾categories below. 1.1.1 Enrollment/attendance Likelihood to return to school 1.1.2 Retention/dropout rates Likelihood to stay in school 1.1.3 Academic performance Discussion of AJ academic performance 1.2 Employment outcomes 1.2.1 Obtain jobs Ability of youth to obtain jobs 1.2.2 Maintain jobs Ability of youth to maintain jobs 1.2.3 Entrepreneurship Youth starting their own business 1.2.4 Skills gaps Discussion of gap between youth skills and skills required for the job market 1.3 Reduce risky behaviors Discussion on the likelihood of youth to reduce risky behaviors. Use the sub-codes unless a different risky behavior is discussed. 1.3.1 Sexual activity 1.3.2 Drugs and alcohol 1.3.3 Violence/crime 1.4 Other outcomes Other outcomes may include leadership, improved youth network/relationships, attitudes, family relations, etc. 1.5 Comparisons to non-Alerta Joven youth 1.6 Positive effect The outcome effect is positive 1.7 Null or negative effect The outcome effect was null or negative 2. Contributing Factors EQ2 (a) Which major factors have contributed, so far, to the achievement (or failure to achieve) of the results through each one of the project interventions by the end of the third year of implementation? 2.1 Most successful elements/factors Factor contributing to the achievement of project interventions. Should co-code with specific interventions sector. 193 2.1.1 Essential components Adapted from EQ2 (c): What are the essential components for the intervention to be successful? Particularly looking for essential components of workforce development programming. 2.2 Least successful elements/factors Factor contributing to the failure of project interventions. Should co￾code with specific intervention sector. 2.3 Recommended programming improvements Co-code recommended improvements with the type of project intervention. 2.4 Public-private partnerships EQ2 (b) To what extent have PPPs contributed or not to the results achieved? Should be co-coded with the "most successful" or "least successful" factors as well as intervention sector, if applicable. 2.5 Exogenous factors influencing results Responds directly to a question in the qualitative interview guide. Refers to elements outside project control, such as policy changes, infrastructure, etc. 3. Sustainability of Services EQ3: Are the services offered to youth by Entrena and sub-awardees sustainable, in terms of financial resources and institutional capacity built under the project and potential local uptake? 3.1 Financial sustainability Comments about the services' financial sustainability 3.2 Institutional capacity sustainability Comments about the services' sustainability because of local institutional capacity 3.3 Local uptake Comments about local update of Entrena services 3.4 Recommendations for improving sustainability Recommendations to improve sustainability of services 3.5 Sustainability successes/benefits Describes successes in sustainability of services 3.6 Sustainability failures/challenges Describes drawbacks/failures in sustainability of services 4. NGO Capacity Building EQ4: To what extent has the capacity building training offered by Entrena helped sub-awardees improve their technical and financial management capabilities which could help them access other sources of funding once Alerta Joven is over? 4.1 Technical capabilities Training in the technical activity area (education, health, etc.) 4.2 Financial management capability Effect of training on financial management capabilities 4.3 Strategic/management capacity Training or efforts to improve strategic of management capacity of NGOs 4.4 M&E capabilities 4.5 Access to outside funding 4.6 Recommendations for capacity building 194 4.7 Capacity-building positive effect Training had a positive effect on technical and financial capabilities. Should be co-coded with the improved capabilities above. 4.8 Capacity-building null/negative effect Training had a negative effect on technical and financial capabilities. Should be co-coded with the improved capabilities above. 5. Perceptions EQ5: How is Alerta Joven perceived and valued by the beneficiaries, the community and other related parties including pertinent national authorities? Should only be applied when discussing group perceptions of the program, not individual feedback. 5.1 Perceptions by group 5.1.1 Youth 5.1.2 Government 5.1.3 Employers 5.1.4 School 5.1.5 CBOs/CSOs 5.1.6 Sub-awardees 5.1.7 Families 5.1.8 Other 5.2 Positive perceptions 5.3 Negative perceptions 6. Other 6.1 Intervention type/sector 6.1.1 Health 6.1.2 Education 6.1.3 Workforce development 6.1.4 Life skills 6.1.5 Crime and violence prevention 6.1.6 Other 6.2 Proportion of population as Alerta Joven youth 6.2.1 Proportion of school 6.2.2 Proportion business Proportion of employees aged 11-25 in a business 195 6.3 General recommendations Should only be used for recommendations that are not included under the specific evaluation question recommendations categories above (sustainability, program components, NGO capacity building). Co-code with appropriate intervention type/sector. 6.4 DON'T USE - Change attributed to Alerta Joven Need to clarify when this code should be used vs. the positive/negative codes within each EQ above. 6.4.1 Positive change Need to clarify when this code should be used vs. the positive/negative codes within each EQ above. 6.4.2 Negative change Need to clarify when this code should be used vs. the positive/negative codes within each EQ above. 6.5 Powerful quotes 7. Sub-awardees and Programs 7.1 Subawardee Use these codes for references to sub-awardees that are outside the scope of the original descriptors. That is, if a full transcript is only referring to one sub-awardee, these codes should not be used. However, if multiple sub-awardees are referenced, then use these codes to distinguish the effect or status of different sub-awardees. 7.1.1 Children Int – Boca Chica 7.1.2 ACOPRO 7.1.3 CAMINANTE 7.1.4 Children Int – Santo Domingo Este 7.1.5 CSA 7.1.6 MOSCHTA 7.1.7 PROJECT HOPE 7.1.8 IDDI 7.1.9 PROFAMILIA 7.1.10 FPUM 7.1.11 FURJUG 7.1.12 JADOM 7.1.13 Sur Futuro 7.2 Program 7.2.1 Entrepreneurship 7.2.2 Vocational education training 196 7.2.3 Assistance to legal documentation 7.2.4 Espacios para crecer 7.2.5 Public policy advocacy 7.2.6 School insertion 7.2.7 Youth mediation 7.2.8 Microcredit 7.2.9 HIV prevention 7.2.10 HIV testing 7.2.11 School retention 7.2.12 Health services 7.2.13 Psychological therapy 197 ANNEX F: DECOMPOSITION ANALYSES Regression analysis can assess which independent variables (drivers) are affecting the outcome and can provide an estimate of how one unit change in the independent variable changes the outcome. However, what a traditional regression analysis cannot provide is an estimate of relative share of each driver in explaining the overall change in the outcome variable. This is a particular challenge in the Alerta Joven analyses where the regression analyses are clearly showing an influence of age on key outcomes, but where it is otherwise not possible to tell the extent to which age is a key influencer vs other potential explanatory variables. Several factors (both contextual as well as individual factors) can under- or over-estimate the overall effect of the program in many ways and sometimes they are hard to account for. For example, one can expect the likely strong impact of natural age progression on key outcomes like school enrollment, engaging in sexual activity, drinking, and holding a job. As we can see in the figures, effects of age on outcomes are stark. A participant’s location may also affect outcomes. To account for these influences in total change in outcome between intake and follow up, the evaluation team embarked on an effort to decompose which variables held the strongest explanatory power and what is their relative share in total change. In light of the lack of comparison group, these decomposition analyses helped the team better assess potential explanatory variables by assessing which ones had the most explanatory power. The evaluation team conducted Fairlie Decompositions50 for four of the key outcomes: school enrollment, consumption of alcohol, having sex, and holding a job. The Panel I of Table X shows the total change over time between intake and follow up; and change (i.e., explained variation) that is attributable to the participant specific characteristics such as age, marital status and province they are from. The difference between ‘overall change’ and ‘total explained variation’ can be considered as change attributable to the program (after controlling for age, region etc.). The identifier, province is included in the model to control for other unforeseen factors that may be location specific. Table 1 provides the overall results. As can be seen in Panel II, age was statistically significant at p<.001 for all four outcome variables. And, as a proportion of the total explained variation, age was the primary explanatory factor. Even for school enrollment, where the province one lived in and one’s relationship status were also statistically significant, they only explained a very small part of the overall variation (- .0005 and -.0001 respectively), whereas age explained -.055 of the overall variation of -.067. As described under the study limitations, issues with the available data on the intensity of enrollment prevented the evaluation team from including an estimation of enrollment intensity among the explanatory variables. 50 Fairlie decomposition is a non-linear method used with binary outcomes. 198 Table 7: Fairlie Decomposition Results *= significant at p<.05; **=significant at p<.01; ***=significant at p<.001 To better visualize the age effects that are being discussed, the graphs in Figures X-Y depict the Locally Weighted Scatterplot Smoothing (LOWESS) to depict the distribution of respondents who have engaged in the given activity by age. LOWESS uses bivariate local regression analysis between the variables of interest to create a smooth line through a time plot. We plotted all four outcome indicators against age from both the intake forms (red lines) and the follow-up survey (green lines). When we compare the gap between the green and the red line at a particular age reference point, Figure 1 shows that before and after the program, the participants across age behave in a similar fashion in terms of enrollment. However, the drop is less in the follow up period for participants 20 years or older. Similarly, if we look at the drinking habits, Figure 2 shows that it was similar at intake (as high as 80%) irrespective of age, while, in Fairlie Decomposition Results Panel 1 School Enrollment Having Consumed Alcohol in the Last 12 Months Having Ever Had Sex Had a Job Within the Last Week Number of observations 1,176 825 959 446 N of obs in Follow up 606 594 394 285 N of obs in Intake 570 231 565 161 Intake Average 0.925 0.762 0.250 0.429 Follow up Average 0.861 0.513 0.442 0.505 Change Over Time (Follow up-Intake) -0.063 -0.248 0.192 0.077 Panel II: Decomposition Analysis Total Explained Variation in the Dependent Variable -0.067 0.064 0.168 0.099 Change in the Dependent Variable that is Explained by: Respondent Age -0.055*** 0.066*** 0.171*** 0.098*** Relationship Status -0.005* -0.001 -0.001 0 Province -0.001*** -0.002 -0.002 -0.001 199 the post intervention period, it is remarkably less, particularly for the participants less than 20 years old. We also find that likelihood of being employed has improved for participants above 15. These visual observations showing the significant effect of age on outcomes echo the intake vs follow-up analyses noted in the report. Figure 6: Distribution of School Enrollment and Employment by Age Figure 7: Distribution of Drinking and Sexual Habits by Age .2 .4 .6 .8 1 Portion of Respondents Currently Enrolled 5 10 15 20 25 30 Age Intake Follow-Up Distribution of Enrollment Status by Age .2 .3 .4 .5 .6 .7 Proportion of Respondents with a Current Job 0 5 10 15 20 25 30 Age Intake Follow-Up Distribution of Employment by Age 0 .2 .4 .6 .8 Proportion of Respondents Having Consumed Alcohol 0 5 10 15 20 25 30 Age Intake Follow-Up Distribution of Drinking Habits by Age 0 .2 .4 .6 .8 1 Proportion of Respondents Who Had Ever Had Sex 0 5 10 15 20 25 30 Age Intake Follow-Up Distribution of Sexual Habits by Age 200 ANNEX G: TEAM COMPOSITION AND DISCLOSURE OF COI Team Leader, Olga Nazario is a civil society expert with 30 years of experience working in international development and conducting quantitative and qualitative research focusing on accountability, access to justice, and other civil society issues. Dr. Nazario conducted the field based qualitative data collection and was instrumental in the analysis and drafting of the reports. Evaluation Specialist, Basab Dasgupta is an expert in quantitative methods at Social Impact, with more than a decade of experience in the design of quantitative instruments, analysis of large datasets, and integration of quantitative methods into a variety of evaluation designs and approaches. He was instrumental in the rigorous analysis of quantitative data from the youth outcomes survey. Qualitative Research Analyst, Jacqueline Lyon aided in the development of instruments and training materials and served as a lead data collector for the qualitative data along with Dr. Nazario. The analyst will conduct research on interviews and focus group discussions while providing oversight on the transcription process to ensure quality of the data. She is also responsible for analysis of qualitative data and development of the final report. Senior Youth Development Specialist, Christy Olenik assisted the team in the development of qualitative and quantitative data collection tools by providing technical guidance, particularly ensuring inclusion of a Positive Youth Development approach. She also provided technical guidance and revisions to the inception, draft, and final reports. Local Evaluation Specialist, Julio Canario is a seasoned M&E professional with over 10 years of demonstrated qualitative data collection, survey oversight, and data analysis experience. He provided significant oversight during the quantitative youth outcomes survey data collection. He also participated in the out brief to the Mission. Project Director, Danielle De Garcia is a deputy director at Social Impact with more than 10 years of experience in qualitative methods, human and institutional capacity development, data collection and analysis, facilitation, and strategic planning. Her experience includes work in more than 20 countries, and she has provided technical assistance to a variety of local, national, and international organizations. She provided quality oversight throughout all periods of the evaluation and on all deliverables. Project Manager and Quantitative Research Analyst, Catherine Ludwig Villada brings over four years of professional experience in M&E, offering a well-rounded skill set in survey implementation, data collection and analysis, training, data quality assurance, research, and reporting. She assisted quantitative data analysis and was the main point of contact with the Mission throughout the evaluation. Project Assistant, Sierra Frischknecht has a wide range of experience in qualitative methods for performance evaluations and with administrative and logistical support throughout the lifecycle including recruitment, consultant onboarding, and contract management. She provided assistance in the development of the qualitative survey tool, codebook, and qualitative analysis. 201 202 203 204 205 206 207 ANNEX H. DESK REVIEW OF DOCUMENTS REVIEWED Main Documents Reviewed: 1. Entrena Annual Report for Year 3 (2014-2015): Attachment 14 – Partners Profiles a. This document includes one-page summaries of the name, principal scope of activity, sub-agreement with Entrena, achievements in Year 3, and major highlights of every sub￾awardee in the Alerta Joven project. b. The information was used to determine the sample frame of implementers in the Santo Domingo area. 2. Entrena Entry Survey a. This survey was conducted by Entrena at the beginning of the Alerta Joven project to collect demographics and baseline information on risky behaviors. b. This was used to directly draw some questions to craft the Youth Outcome Survey c. Questions directly drawn from the Entry Survey include: i. Family status (marital status; whether have children) ii. Educational status (whether can read and write, the highest educational level obtained; current schooling status; frequency of missing school without an excuse/being suspended from school in the past 12 months; aspired achievement with current education iii. Employment status and history (whether have performed a job for pay or profit, whether did any work for pay or profit during the last week, weekly earning, work location, role, whether currently working without pay) iv. Risky behaviors (alcohol consumption and the frequency, drug consumption, sexual activity, protection, whether received sex education, informational source of pregnancy prevention) 3. Dominican Republic Cross-Sectoral Youth Assessment (August 2010) a. Prepared by Education Development Center, Inc. (EDC), this document analyzes the structure and characteristics of the youth cohort in the Dominican Republic b. It reviews issues facing youths and the resources available to address them in four sectors – education, economic growth, health, and democracy and governance through the results of 40 youth focus groups presenting what youth themselves see as their needs, aspirations, and challenges. It further reviews a first generation of 9 youth projects funded by USAID/DR and provided recommendations for a new generation of USAID/DR sponsored youth programming c. The information was used to gain contextual understanding about youths in the Dominican Republic for the Alerta Joven project 4. Entrena Alerta Joven Annual and Quarterly Reports a. Annual Reports (2012-2013, 2013-2014, 2014-2015) i. Annual Reports provide brief introductions to the Alerta Joven project, numerical results (total number of participating youths relative to overall target population, percentage of budget execution, number of implementing organizations, number of government agencies that develop private-public partnerships, etc.) ii. They also cover major achievements, modifications and resulting impacts and challenges during the year 208 b. Quarterly Reports (July-Oct, Nov-Dec in 2012; Jan-Mar, Apr-June, Oct-Dec in 2013; Jan-Mar, Apr-Jun, Oct-Dec in 2014; Jan-Mar, Apr-June, Oct-Dec in 2015; Jan-Mar 2016) i. Quarterly reports summarize the characteristics associated with the specific period and provided detailed information about the major achievements, highlights, and challenges during that period c. Both Annual Reports and Quarterly Reports were used to understand the development of the Alerta Joven project over time Additional Documents Reviewed: 5. Performance monitoring and evaluation: a. Dominican Republic At-Risk Youth Activity (DRARYI) Performance Monitoring and Evaluation Plan (PMEP) with baseline i. This document covers the key aspects of the design of the performance monitoring and evaluation plan (PMEP) in the Alerta Joven project and a brief discussion on the indictors selected for this PMEP. It also describes quality control process to be used for the implementation of the PMP 6. Community surveys: a. Presentation of “Perceptions of Crime, Insecurity, and Trust in the Police in the Dominican Republic in 2013” (Nov 20, 2013, American Society of Criminology) i. Prepared by RTI International, this deck summarizes findings of the Baseline Community Survey b. CBS Executive Summary of “AJ Baseline Community Security Survey: Perceptions and Victimization Experiences in 10 Provinces in the Dominican Republic” c. Informe CBS Linea Base (Online community-based survey: Perception of violence and victimization degree in 10 provinces in Dominican Republic, May 2015) by RTI International [in Spanish] d. Final report of “Interim Community Survey: Perception of Violence and Self-Reported Victimization in 10 Provinces in the Dominican Republic in 2015” i. Prepared by RTI International, the report summarizes the interim survey conducted as part of Alerta Joven’s M&E strategy in Apr 2015 ii. It captures and analyzes information regarding 1) knowledge about Alerta Joven, 2) perceptions about personal and community security, 3) victimization experiences, and 4) trust in a variety of institutions e. These documents provide detailed information about findings in baseline and interim community surveys. It helps to understand perceptions of crime and self-reported victimization in the communities where the Alerta Joven project took place 7. Work plans: a. Entrena AJ Annual Work Plan for Year 1 (July 5, 2013 – July 31, 2013) i. It covers overall objectives and results, assumptions, perceived problems and possible barriers 209 ii. It also discusses details regarding project management, M&E and information system, sustainability, identifying and locating the target population, review policy and legal protocol for ARY, sub-grants, direct field interventions – opportunities for youth, reporting and dissemination b. Entrena AJ Annual Work Plan for Year 2 (July 5, 2013 – July 4, 2014) – Narrative i. It covers overall objectives and results, assumptions, perceived problems and possible barriers. It also includes a complete month by month calendar with planned activities c. Entrena AJ Annual Work Plan Year 2 – English Version [Excel file] d. Entrena AJ Annual Plan Year 3 – English Version [Excel file] e. Entrena AJ Annual Plan Year 4 (Oct 2015 – Sept 2016) [Excel file] i. These work plans provide complementary information about how the Alerta Joven project was planned to achieve its goals and impact on an annual basis. 210 ANNEX I. PRESENTATION FOR DISSEMINATION 211 212 213 214 215 216 217 218 219 220 221 222 223