Evaluation of the Guyana Skills and Knowledge for Youth Employment Program December 2014 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Danielle de García, Sarah Edith Jones, and Mateusz Pucilowski of Social Impact, Inc., and Aisha Fraites, Roxana Kawall, Esther McIntosh, Roxanne Myers, and Renata Chuck-a-Sang of The Consultancy Group. EVALUATION OF THE GUYANA SKILLS AND KNOWLEDGE FOR YOUTH EMPLOYMENT PROGRAM MID-TERM PERFORMANCE EVALUATION December 2014 Task Order No. AID-538-TO-14-00001 Awarded through the Monitoring and Evaluation Services for the Caribbean Basin Security Initiative Indefinite Quantity Contract DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Cover Photo: SKYE Youth working in the tourism industry following her experience in the program. Photographed by Tim McCartney. Evaluation of the Guyana SKYE Program i TABLE OF CONTENTS Acronyms............................................................................................................................................................................ iii Acknowledgements .......................................................................................................................................................... iv Executive Summary ........................................................................................................................................................... 1 Project Background ........................................................................................................................................................... 9 Introduction .................................................................................................................................................................... 9 Project Background ....................................................................................................................................................... 9 SKYE Activities and Evolution .................................................................................................................................. 10 Evaluation Purpose & Evaluation Questions.............................................................................................................. 12 Evaluation Purpose ...................................................................................................................................................... 12 Evaluation Questions .................................................................................................................................................. 13 Evaluation Methods and Limitations ............................................................................................................................ 14 Data Collection ............................................................................................................................................................ 14 Data Analysis ................................................................................................................................................................ 17 Limitations ..................................................................................................................................................................... 17 Findings, Conclusions & Recommendations .............................................................................................................. 18 Question 1: Findings ................................................................................................................................................... 18 Question 1: Conclusions ........................................................................................................................................... 35 Question 2: Findings ................................................................................................................................................... 38 Question 2: Conclusions ........................................................................................................................................... 42 Question 3: Findings ................................................................................................................................................... 44 Question 3: Conclusions ........................................................................................................................................... 48 Recommendations ........................................................................................................................................................... 50 Recommendations within the Period of Performance and Budget: ................................................................ 50 Recommendations that Exceed the Period of Performance and/or Budget: ................................................ 54 Annexes ............................................................................................................................................................................. 58 Annex I: Contract SOW ............................................................................................................................................ 59 Annex II: Data Collection Instruments .................................................................................................................. 61 Annex III: Desk Review .............................................................................................................................................. 86 Annex IV: Sources of Information ........................................................................................................................... 87 Annex V: YAG Agenda ............................................................................................................................................... 90 Annex VI: Utilization Matrix ..................................................................................................................................... 91 Annex VII: Conflict of Interest Forms .................................................................................................................... 92 ii Evaluation of the Guyana SKYE Program FIGURES Figure 1. SKYE Theory of Change ............................................................................................................................... 10 Figure 2. SKYE implementation regions ..................................................................................................................... 11 Figure 3. SKYE implementation timeline .................................................................................................................... 12 Figure 4. Regions visited during evaluation ................................................................................................................ 14 Figure 5. How did participants learn about SKYE? .................................................................................................. 21 Figure 6. Asset ownership ............................................................................................................................................. 23 Figure 7. Education, by gender ..................................................................................................................................... 25 Figure 8. Youth ranking of program components .................................................................................................... 27 Figure 9. Self-reported confidence in employment skills....................................................................................... 29 Figure 10. Link to employment, by gender ................................................................................................................ 30 Figure 11. Employment, by gender and region ......................................................................................................... 31 Figure 12. Productive engagement ............................................................................................................................... 33 Figure 13. Population estimates ................................................................................................................................... 33 Figure 14. Highest education level desired ................................................................................................................ 34 Figure 15. Youth responses to survey questions on staying out of trouble and standard of living improvements ................................................................................................................................................................... 35 Figure 16. Youth responses to survey questions on satisfaction and recommending program to peers .. 45 TABLES Table 1. Regional distribution of completed surveys .............................................................................................. 15 Table 2. Survey sample and applicants, by gender ................................................................................................... 16 Table 3. Parental status of survey respondents ........................................................................................................ 22 Table 4. Education, by gender ....................................................................................................................................... 24 Table 5. Monthly salary, by gender..……………………………………………………………………………………………………………..….32 Table 6. Job benefits……………………………………………………………………………………………………………………………………….....32 Evaluation of the Guyana SKYE Program iii ACRONYMS ADS Automated Directives System AEA Adult Education Association CBO Community-based organization CBSI Caribbean Basin Security Initiative CO Contracting Officer COR Contracting Officer’s Representative CRS Catholic Relief Services CXC Caribbean Examinations Council EDC Education Development Center FGD Focus Group Discussions GCCI Georgetown Chamber of Commerce and Industry GIT Guyana Industrial Training Centre GoG Government of Guyana GYBT Guyana Youth Business Trust KII Key Informant Interviews MCYS Ministry of Culture, Youth, and Sports MLHSS Ministry of Labour, Human Services, and Social Security NOC New Opportunity Corps PRA Participatory Rural Appraisal SES Socio-economic status SI Social Impact SKYE Skills and Knowledge for Youth Employment STA Senior Technical Advisor TCG The Consulting Group TOCOR Task Order Contracting Officer Representative TPM Team Planning Meeting USAID United States Agency for International Development VYC Volunteer Youth Corps YCG Youth Challenge Guyana iv Evaluation of the Guyana SKYE Program ACKNOWLEDGEMENTS The SKYE Evaluation team would like to express our appreciation to USAID and the Education Development Center (EDC)/Guyana for their support throughout this evaluation. Ms. Chloe Noble of USAID/Guyana provided consistent support, guidance, and comments to the team. Her contributions and acumen are much appreciated. The EDC/Guyana team was patient in responding to requests and clarifications, and their involvement contributed a great deal to our understanding of SKYE and its evolution. The members of the Youth Advisory Group were particularly helpful in providing contextual knowledge and assistance revising data collection instruments. We also appreciate the insight of SKYE’s implementing partners, trainers, coaches, and Government of Guyana representatives. We are grateful for your contributions not only to the evaluation, but for your work with and commitment to improving the situation of at-risk youth in Guyana. Finally, special thanks to the many youth who spoke with us and provided us with vital insights and answers to our many questions. Your candor, time, and trust in our team were invaluable, and we hope this report accurately reflects your experiences and recommendations. Evaluation of the Guyana SKYE Program 1 EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS Social Impact (SI) was contracted to undertake a mid-term performance evaluation of the USAID-funded Skills and Knowledge for Youth Employment (SKYE) Project in Guyana. SKYE is being implemented by EDC as a part of a larger USAID initiative to improve security in the Caribbean Basin. The mid-term evaluation came at the mid-way point for the project, and was designed to assess whether or not the SKYE project was on target to meet their primary objectives as well as the successes and challenges that the project faced along the way. In doing so, SI and their subcontractor, The Consultancy Group (TCG), were asked to provide recommendations to help improve programming and ensure that the positive outcomes of SKYE be sustained over time. Specifically, the Mid-Term evaluation was designed to answer the following evaluation questions: 1. Are activities sufficiently coordinated and building towards the achievement of strategic objectives? 2. How effectively has the project engaged stakeholders at the local and national level? 3. What are the prospects for the sustainability of the results produced/approaches being utilized by this project? 4. Are there any recommendations to enhance project results within the project’s current period of performance and budget? 5. Are there any recommendations to expand or enhance project results should additional resources become available? PROJECT BACKGROUND The SKYE project was developed with the primary objective to reduce violence in Guyana through strengthened economic participation and civic engagement of at-risk youth. In order to meet this goal, the SKYE project aims to meet three intermediate objectives:  The expansion of employment, education and skill building opportunities for youth at risk  The strengthening of reintegration of youth offenders into society  The strengthening of an enabling environment for youth development. The SKYE project aims to meet these objectives both by working with youth directly as well as by building structural support for youth offenders. The work that occurs directly with youth comes in the form of a formal training and mentorship program designed to enhance life and employment skills. Youth are engaged throughout the process with a coach who mentors and guides them through the development of a livelihood plan. The youth are enrolled in literacy or soft skills training, designed to build on their ability to apply for a job, construct a resume, prepare for an interview, and interact with managers and supervisors in an appropriate manner. If youth in the project are struggling with their literacy level, they have the opportunity to take an eight week training course to help improve their literacy skills prior to enrolling in the skills training course. To help establish structural support for youth, the SKYE team is working with the judicial system to build SKYE into a viable sentencing alternative for youth, and with employers, SKYE is working to create employment opportunities for youth upon completion of the program. 2 Evaluation of the Guyana SKYE Program EVALUATION METHODS In order to assess the project’s progress and answer the evaluation questions, the team employed a mixed methods approach that included a close-ended survey of a sample of 397 SKYE participants, qualitative interviews with forty of those 397 youths, nine focus group discussion with youth, as well as 69 key informant interviews across Regions 4, 6, 9 and 10. This approach allowed the evaluators to obtain and document training results and programmatic processes from a multiplicity of perspectives. In order to maximize the fidelity of data, and ultimately the utility of the evaluation, the team convened a group of youth participants (the youth advisory group, or YAG) to provide feedback on the appropriateness of data collection protocols and instruments. The team ensured anonymity of responses throughout data collection and analysis. Quantitative data gathered from the survey responses were analyzed using Stata, whereas the qualitative data collected from the interviews and focus groups were transcribed and coded using the Atlas.ti software package. Once coded, the team used complex queries to identify emerging themes and triangulate data with the quantitative findings. The results of these queries then helped inform the findings, conclusions and recommendations for each of the evaluation questions. EVIDENCE AND FINDINGS Question 1: Are activities sufficiently coordinated and building towards the achievement of strategic objectives? In order to answer this question, the team evaluated SKYE’s targeting and recruitment strategies, as well as whether SKYE has equipped youth with the skills they need to gain employment, re-enter school, or start their own businesses. Finally, the team explored the extent to which the program has been successful in achieving its direct objectives. Targeting and Recruitment The features that were most commonly identified by key informants that characterize youth as “at-risk” included: school-drop outs, low literacy levels, and criminal behavior or interfacing with the juvenile justice system. While these features were commonly named, key informants also acknowledged that there was no hard and fast rule for determining whether or not a youth would qualify as “at-risk.” For example, key informants stated that youth from single parent homes, those with mental health issues, victims of physical or sexual abuse and individuals who use or abuse drugs or alcohol are also “at-risk.” Drawing from the qualitative interviews, the team attempted to gauge the types of youth that were being targeted for the program and determine whether they aligned with the broader understanding of risk. Broadly, the information provided indicated that youth that were being targeted did have a number of the features identified as risk factors. Data also indicated that the target has broadened over time and with more recent enrollees facing significant challenges such as extremely low levels of literacy and exposure to violence, illicit drugs or other criminal activities. Expanding the target has presented challenges for those who train, coach, and employ youth. To understand how targeting of youth played out operationally, the process by which they were recruited was examined. The survey data indicated that youth most commonly heard of the program through their coach followed by other participants. Key informant interviews revealed that recruitment is a large task that requires a variety of methods to locate youth for the program. Evaluation of the Guyana SKYE Program 3 In examining the risk profile of survey respondents the evaluation team found that the majority of youth displayed at least one of five risk factors: broken household (83%), teenage pregnancy (8%), self￾reported arrests (18%), low socio-economic status, and low levels of educational achievement. Program Delivery The evaluation team identified four primary conduits through which beneficiaries experience the program: coach, trainer, training (activities, materials, etc.) and other beneficiaries. Of these four, youth most frequently named the training as their favorite component of the SKYE project. Interview data revealed that youth particularly liked the trainings because they were “fun” and gave them “real life” practice. Diving a little deeper, data indicated that while training was their favorite activity, the coaches were a key component in the structuring of SKYE but one that was not easily defined. Self-report data indicated that coaches are responsible for recruiting youth, providing linkages to future employment or education, assisting trainers, developing and implementing livelihood plans with the youth, and monitoring youth’s progress towards these goals. Trainers have a slightly more clearly defined set of roles and responsibilities. Specifically, they are expected to assess the skill levels of the youths, facilitate trainings, and counsel youth. Intended Outcomes While the objectives of the SKYE project are multifaceted, the evaluation team decided to focus on beneficiary skills to measure intermediate outcomes. Data indicate that the youth feel more confident in their abilities, due to the skills they acquired through SKYE (undertaking a job search, writing a CV, career planning, interacting with superiors, interviewing for a job, financial planning, and starting a business). Youth also reported high aspirations for continuing their education, with three out of 4 reporting a desire to return to school, the majority of whom were aiming for either tertiary education or technical/vocational trade school. However, their confidence did not necessarily translate into positive outcomes in the sense that fewer than 40% of youth received a job due to their involvement in SKYE. Similarly, two in five youth were connected to professional trainings related to their desired job sector. Data showed, however, that for those youth who were employed, there were positive trends in the reported quality of those jobs, as indicated by salary, access to benefits, and hours worked. Additionally, when asked if they had met the goals established in their livelihood plans, more than half reported being successful (score of 4 or 5 out of 5) in meeting their goals. Question 2: How effectively has the project engaged stakeholders at the local and national level? In order to better understand the extent of SKYE engagement with stakeholders external to the project, the evaluation team broke up regional- and national-level stakeholders into seven main groups:  National government  Regional and local government  Private sector  Educational and vocational institutions  Legal system  Families and communities  Local and regional NGOs and CBOs 4 Evaluation of the Guyana SKYE Program National Government National-level government stakeholders included those from a wide variety of ministries and agencies, including from the Ministry of Culture, Youth, and Sports; the Ministry of Labour, Human Services, and Social Security; the Ministry of Education; and others. Data indicated that key stakeholders from these ministries and other national agencies were generally aware of the SKYE program, however, engagement tended to occur at the individual level rather than the institutional level. Interview data from key informants demonstrated competition for resources and overlap in the roles and responsibilities of the different ministries. That stated, stakeholders from the national government reported high levels of satisfaction and interest in both continuing and increasing their level of engagement with SKYE. Regional and Local Government At the regional and local levels, in comparison to the national levels, stakeholder engagement was less consistent and even more heavily centered on individual networks and contacts.. A number of key stakeholders at the regional and local levels reported never having heard of SKYE, including from central offices and ministries (such as from offices of the Regional Chairman, educational officers, regional social workers, and Village Councils). Private Sector While SKYE headquarters has been responsible for some element of private sector engagement, many key informants noted that the interaction has been limited to a few companies and institutions. Coaches and youth also reported that the majority of connections made with employers were due to relationships that had been established between the coaches and the employers. According to key informants and youth, there was a disconnect between what youth desired and were eligible for and the jobs to which they were linked through the SKYE project. A number of employers and agencies noted that they would be interested in hiring more SKYE youth, but did not know who to contact beyond one coach or individual. Others noted that many organizations and employers were not aware of the SKYE program or what the program teaches, so that the lack of awareness prohibited further alignment of opportunities. Educational and Vocational Institutions Interviews with key informants showed that coordination between educational institutions and SKYE largely depended on the extent and type of interaction coaches had with individuals at the training organizations. There was no SKYE project approach, understood by institutions and coaches, to be the ideal link between these two entities. As such, some coaches worked with vocational institutions to enroll unaccepted students in SKYE, while other coaches would reach out to institutions to try to access recent graduates of vocational programs and recruit them into the project. Legal System National-level stakeholders, including probation officers, magistrates, representatives from Child Services, and SKYE, reported high levels of satisfaction and coordination with one another, asserting that the program has helped decrease the number of youth in prison. While most stakeholders stated that they believed diversion and alternative sentencing were fundamental to curbing youth violence, the majority of respondents did not believe it was being implemented consistently or broadly enough throughout the country. At the regional and local levels, the team received mixed reactions to the SKYE Evaluation of the Guyana SKYE Program 5 approach to reducing youth criminal activity, with some stating that SKYE’s approach was successful, and others believing it to be an ineffective or incomplete approach. Families and Communities Interview data indicated that there was a high level of variability with respect to interaction between SKYE and participants’ families and communities. Some coaches reported that meeting with families was a fundamental part of their role, while others saw it as unnecessary or outside their scope of work. It is important to note that trainers, coaches, and youth from all regions reported pressures on youth from family members to leave the training to earn money; or a lack of positive encouragement at home to continue working in their place of employment. Local and Regional NGOs and CBOs As with other types of engagement, interaction with local and regional NGOs and CBOs varied depending on the region and individuals involved. Many SKYE stakeholders, including trainers, coaches, government representatives, partners, and educators, have well-established links and connections within the youth ambit. Most were able to identify a number of organizations doing complementary or relevant youth work, many of which knew of SKYE or had some link to SKYE due to those individual connections. Church organizations were mentioned by a wide variety of individuals as an opportunity for further engagement, as were graduate placements with NGOs and CBOs in a volunteer or internship capacity. Question 3: What are the prospects for the sustainability of the results produced/approaches being utilized by this project? The evaluation team examined four major factors as indicators of sustainability: stakeholder buy-in, financial sustainability, policy developments/institutionalization of programming, and the development of regional and local capacity Stakeholder Buy-In To assess overall buy-in, the evaluation team used two proxies: satisfaction with SKYE and a willingness to participate in or engage with SKYE. Survey findings indicate that overall, youth were very satisfied with the SKYE project and were even more supportive of recommending it to their peers. Qualitative findings from youth interviews correspond with these quantitative findings, as an overwhelming majority of youth were pleased with their experience with SKYE. The majority of key informants with whom the evaluation team spoke had positive things to say about SKYE. Many stated that SKYE was the first program of its kind regarding development of youths’ soft skills and encouragement of pro-social behaviors. Specifically, many pointed to the important role that the coaches played. Key informants at the National, Regional and Local levels all indicated a high level of interest in SKYE, though stakeholders at the National level tended to be more familiar with the program and more actively engaged than those at the local level. Financial Sustainability Currently, SKYE is 100 percent donor-dependent, with all funding provided through USAID’s Caribbean Basin Security Initiative. During interviews with several representatives from key Ministries, respondents stated that they would be willing to commit financial resources to the continuation of SKYE. When asked why they weren’t currently funding the effort, they responded that it takes time to secure funding. 6 Evaluation of the Guyana SKYE Program Policy Developments and Institutionalization of Programming Another crucial component of activity and outcome sustainability is the development of policies and structures to provide for the institutionalization of activities. Interviews with key informants pointed to three major initiatives with the most potential in integrating elements of SKYE into existing systems: the implementation of SKYE at the NOC, the use of SKYE as an alternative sentencing mechanism, and the integration of SKYE training into secondary school curricula. Each of these initiatives is in a different stage of being operationalized, has different prospects for institutionalization, and would require different levels of resources to ensure sustainability. Development of Regional and Local Capacity While the evaluation team did not include a question in its protocol regarding the types of capacity building that SKYE was undertaking at the local and regional levels, on the whole, very few key informants discussed the development of Regional and Local Capacity for SKYE. When it was mentioned, it was done so in the context of increasing resources within their respective organizations, as was the case in Region 9. While key informants on the whole had a basic understanding of the SKYE objectives and activities, they did not seem to be versed in the specifics. . CONCLUSIONS Question 1: Are activities sufficiently coordinated and building towards the achievement of strategic objectives? While SKYE should consider tightening eligibility criteria, data indicate that on the whole, and given the programmatic context, the project was reaching an appropriate population of youth. In terms of those aspects of programming that targeted youth most directly, the project is making consistent progress in reaching their objective of preparing youth for employment. As expressed by youth and triangulated by employers, SKYE participants, had, on average, better soft employment skills (interviewing, communication, leadership, etc.) than comparable youth without SKYE training. While SKYE youth had not gained employment nor enrolled in school to further their education at the rate desired by the program, participation in the program was associated with increased life skills: youth reported being more confident, more comfortable with the opposite sex, more helpful, and better able to plan for the future. Question 2: How effectively has the project engaged stakeholders at the local and national level? With the exception of national government engagement, the successes in engagement have largely been due to individual SKYE coaches and their approaches. Centralized coordination and engagement from SKYE seemed to be strong at project start, but has since focused inwardly rather than externally to the stakeholders: the national government, regional and local government, private sector, educational and vocational institutions, legal system, families and communities, and local and regional NGOs and CBOs. Data indicate that key stakeholders were not only satisfied with the program, but keen to engage with it; even those who were not aware of SKYE, demonstrated an interest in getting involved. However, data also indicated that, while individuals on the national level seemed to play an active role in SKYE, individuals at the regional and local level were less likely to be directly involved, and often knew less Evaluation of the Guyana SKYE Program 7 about the program than their counterparts at the national level. Given these findings, one may conclude that the buy-in of SKYE is fairly high, though it atrophied in relation to distance from the capital. Question 3: What are the prospects for the sustainability of the results produced/approaches being utilized by this project? Overall, the SKYE program has begun to take steps toward ensuring sustainability of programming and outcomes. However, further work needs to be undertaken now that the program has reached its midpoint and an endpoint is in view. Overall there were high levels of satisfaction with the SKYE project. Furthermore, when dissatisfaction occurred or a weakness was identified, many argued that the project should be improved rather than discontinued. While there are promising indications that the SKYE project has some important champions, the project is now at a phase where it must move beyond the engagement of individuals and increase institutional engagement at the national, regional and local levels. Additionally, while steps have been taken to build SKYE into institutional structures such as the NOC and secondary schools, integration has met with varied levels of success. In terms of financial sustainability, officials at the national level did express a commitment to provide resources to ensure the continuation of various aspects of the SKYE project. However, their ability to do so remains uncertain. As such, additional research needs to be undertaken in order to understand how the Ministries are building SKYE activities into their budgets. Currently there is also no financial link between the private sector and SKYE, despite the vested interest employers have in making the project work. The possibility of engagement of the private sector in sustaining SKYE activities is something that warrants further exploration. Survey data indicated that youth would be willing to pay for their participation in SKYE. However, this too needs to be explored further, as it is unclear as to the amount youth would be willing to contribute and if youth would still be attracted to SKYE were a payment to be required. Generally, the vast majority of work being undertaken for the SKYE project is being carried out by SKYE staff (those employed by EDC) and a limited number of partner organizations. In order for the project to achieve sustainability, greater efforts need to be invested in building the capacity of individuals in formal structures that already exist within Guyana, including at the NOC and within secondary schools. Additionally, while employers reported that the soft skills of SKYE youth stood out when comparing them to their counterparts who had not gone through training, no efforts have been made to formally train individuals from the technical institutes in the SKYE training. RECOMMENDATIONS Evaluation questions 4 and 5 focus on recommendations that the evaluation team has developed, based on the findings and conclusions. The questions state the following:  Are there any recommendations to enhance project results within the project’s current period of performance and budget?  Are there any recommendations to expand or enhance project results should additional resources become available? Recommendations within the Period of Performance and Budget:  Actively engage alumni in the recruitment of new participants.  Provide gender and LGBT sensitization training to trainers and coaches.  Customize, where possible, materials to speak to the specific context in which the program is 8 Evaluation of the Guyana SKYE Program operating.  Improve coordination between coaches internal and external to the NOC.  Bolster the program monitoring and evaluation system.  Improve performance monitoring of coaches and trainers and provide a feedback loop.  Further engage the private sector and increase the level of coordination between coaches and the SKYE staff responsible for private sector engagement.  Work with current SKYE employers and GCC members to better integrate employers into the job board.  Increase coordination, communication, and engagement with school welfare officers, counselors, probation officers, MCYS reintegration officers, and social service workers.  Start a SKYE alumni group.  Focus yearly targets on ways to integrate SKYE into existing institutions. Recommendations that Exceed the Period of Performance and/or Budget:  Hold formal discussions to determine which of two objectives, reduction in crime versus increased labor market participation, should take precedence with regard to a targeting strategy.  Provide coaches and trainers with additional training in counseling and identify ways in which the program may integrate with counseling support systems that are already in place.  Provide coaches and trainers with a formal support system to help relieve stresses of working with at-risk youth.  Create a formalized feedback loop from SKYE employers.  Determine a SKYE approach to interaction with families and communities and make that approach and associated roles clear to trainers and coaches.  Capitalize on high levels of interest in SKYE displayed by youth and stakeholders at the national, regional and local levels by providing them guidance on specific roles that they can play to increase sustainability. Evaluation of the Guyana SKYE Program 9 PROJECT BACKGROUND Introduction This report presents the findings of the mid-term evaluation of the USAID Skills and Knowledge for Youth Employment (SKYE) project. The evaluation, commissioned by USAID for USD $277,954, was conducted between August and November of 2014 by Social Impact, Inc. (SI) and their subcontractor, The Consultancy Group (TCG). The task order (No. AID-538-TO-14-00001) was awarded through the Monitoring and Evaluation Services for the Caribbean Basin Security Initiative Indefinite Quantity Contract, and the contract Scope of Work (SOW) can be found in Annex 1. Project Background Despite national-level improvements in Guyana over the past decade, at-risk youth in the country continue to face significant challenges in overcoming high levels of unemployment, stigma, and violence. While the country has demonstrated marked improvements in health, wellbeing, and civic participation,1 criminal activity, including homicides and crimes related to drug trafficking, has increased—owing, in part, to the large proportion of unemployed, young males in the country.2 Increased youth involvement in criminal activity has also become linked to other developmental issues for youth, including reduced educational opportunities and marginalization from civic participation processes.3 A number of Government of Guyana (GoG) ministries, international donors, civil society organizations, and other stakeholders have recognized the need to address youth-centered development issues. These organizations and others provide children and youth with a variety of literacy, informal education, and skills training opportunities. In order to further mitigate the incidence of crime and violence, USAID, through the Caribbean Basin Security Initiative (CBSI), identified the need for further engagement of ‘high-risk’ youth between the ages of 15 and 24. In response to this need for support and resources, USAID partnered with the Education Development Center (EDC) to establish the Skills and Knowledge for Youth Employment (SKYE) Program. SKYE’s objective is to reduce violence in Guyana through strengthened economic participation and civic engagement of at-risk youth. Given that young men between 15 and 35 years of age commit roughly 80 percent of the crime in the Latin American and Caribbean Region4 and that there is a pronounced geographic disparity in terms of criminal activity, young men in particular have been targeted to participate in SKYE throughout six of the ten regions of Guyana. EDC’s initial assessment pointed to a prevalence of trainings for hard skills but a lack of the soft skills needed to access and maintain job opportunities.5 In working to enhance these skills and increase economic opportunities, SKYE has three objectives and four intermediate results, as shown in the results framework below. 1 The Commonwealth. September 2013. Youth Development Index. Accessed online on May 16, 2014 at http://www.youthdevelopmentindex.org/cms/cms-youth/_images/197918019952385f3219c75.pdf. 2 United Nations Development Programme (UNDP). 2012. Caribbean Human Development Report. 3 Caribbean Human Development Report. 2012. Human Development and the Shift to Better Citizen Security. 4 January 2010. Report to the CARICOM Commission on Youth Development. 5 Education Development Center (EDC) SKYE Assessment Report and Project Design. 2011. 10 Evaluation of the Guyana SKYE Program Figure 1. SKYE Theory of Change Within this scope, SKYE has defined at-risk youth as those who:6  Have dropped out of school;  Have completed formal education, but who did not achieve minimum academic requirements to gain further access to higher education or jobs;  Have insufficient levels of education, skills, and/or behaviors for workforce integration (which includes those youth who have been in the juvenile justice system); and/or  Who demonstrate need based on other characteristics, including those coming from broken families, low socioeconomic status, or neighborhoods with a high incidence of violence. In order to reach the appropriate youth and achieve the stated objectives, and in agreement with the Educational Quality Improvement Program (EQUIP 3) approach, SKYE interacts with a variety of additional stakeholders at the national and regional levels. These include the Government of Guyana’s Ministries of Culture, Youth, and Sports (MCYS); Education; Home Affairs; and Human Services and Social Security; as well as local non-profits and NGOs that target programming toward the youth population. SKYE Activities and Evolution SKYE’s programming consists of two primary elements, which work to aid this population. Most prominently, the program provides training in work skills and life skills. The training is paired with a mentorship component, which works to link youth to educational and/or employment opportunities. The second program component focuses on the institutional system surrounding youth offenders and their reintegration. Working through the judicial system, the program advances alternative sentencing as a legal option for non-violent youth. 6 Education Development Center (EDC). November 2012. Skills and Knowledge for Youth Employment (SKYE) Program Description. Goal: To reduce crime by enhancing skills and increasing opportunities for at-risk youth. Objective 1: Expanded employment, education, and skill-building opportunities for youth at risk. Result 1.1: Improved capacity of youth at risk to pursue employment or livelihoods. Objective 2: Strengthened reintegration of youth offenders into society. Results 2.1: Increased judicial use of alternative sentencing and diversion for minor offenders. Result 2.2: Improved capacity of youth offenders to re-enter society. Objective 3: Strengthened enabling environment for youth development. Result 3.1: Improved capacity of youth serving organizations, institutions, and justice systems. Evaluation of the Guyana SKYE Program 11 The training component has evolved gradually throughout the implementation process, having started with a six-week Work Readiness training in Regions 4 (Demerara – Mahaica), 6 (East Berbice – Corentyne), 9 (Upper Takatu – Upper Essequibo), and 10 (Upper Demerara – Berbice). Youth Challenge Guyana (YCG) and Volunteer Youth Corps (VYC) partnered with EDC to implement this training from May 2012 through September 2013.EDC and its partners found that the curriculum level was too high for their intended population and that many at-risk youth had lower levels of literacy than those needed to adequately access the program and the workforce. Similarly, they found that six weeks was too much time for some of the youth, who wanted to access a job more quickly. As a result of these findings, SKYE altered its model to provide three different trainings suited to differing literacy levels in the target population. Work Ready Now, a four￾week training, is aimed at those youth with a Grade 7–11 literacy level (regardless of age). Work Ready Plus was designed as a five-week training course infused with literacy skills as well as the topics covered in the four-week course, to suit youth with a Grade 4–6 literacy level. This training has been implemented by VYC since January 2014. When it became apparent that there were youth, particularly those coming out of the juvenile justice system, with even lower levels of literacy, the project expanded to include an eight￾week course aimed at enhancing literacy levels, titled Pre￾Work Ready, which would prepare them to take either Work Ready Now or Work Ready Plus upon completion. The Adult Education Association (AEA) has implemented this training since February 2014. In late 2013, SKYE expanded to Region 3 (Essequibo Islands – West Demerara) and Region 5 (Mahaica – Berbice). Altogether, SKYE has trained approximately 1,300 youth since project inception in at least one of these three courses. In addition to the core training classes, SKYE worked with the Guyana Youth Business Trust (GYBT) to offer a six-day entrepreneurship training in 2012 and 2013, training 37 youth, primarily in Region 9. Given that many youth were unable or unwilling to access loans following this training and uncertainty regarding the level of the curriculum, this training was discontinued in 2013. 9 6 10 3 5 4 Figure 2. SKYE implementation regions 12 Evaluation of the Guyana SKYE Program Figure 3. SKYE implementation timeline The roster of trainings is considered a component of a broader implementation package. All youth participating in SKYE are paired with a coach charged with mentoring and helping to link beneficiaries with job opportunities or further education. For those in the juvenile justice system, and for some youth in Region 9, the trainers and the coach are the same individual, because of the small number of participants and environmental constraints. For the rest of the program participants, these roles are distinct and performed by separate individuals. All program participants, whether recruited from the juvenile justice system or not, undergo training and coaching as part of the SKYE intervention. Independently of the trainings, SKYE has also advocated for alternative sentencing options for minor offending youth. As part of this programmatic focus on youth involved in legal troubles, SKYE has placed two full-time staff at the New Opportunities Corps (NOC). Lastly, in an attempt to promote the sustainability of the programmatic benefits, SKYE has begun working to institutionalize elements of the program within the existing secondary school curriculum. At the time of the evaluation, these efforts had been limited to Region 9. EVALUATION PURPOSE & EVALUATION QUESTIONS Evaluation Purpose Now more than halfway through implementation, SKYE is at a critical juncture to examine the overarching objectives, successes, and challenges related to implementation, and to develop insights on steps toward sustainability and improvements for the remainder of the project. This evaluation is expected to aid USAID, EDC, and other partners in better understanding the initial results and contributions of the project in order to help it re-focus and strengthen where necessary. Given ongoing programmatic and contractual modifications, the evaluation is intended to support USAID/Guyana in its decision on whether and how to extend, expand, or modify the interventions. EDC and its implementing partners, AEA, VYC, and YCG, will use this evaluation in conjunction with USAID to determine whether changes are needed to increase the chances that the intended outcomes will be reached. These November 2011: SKYE assessment completed April 2012: Entrepreneurship training introduced May 2012- September 2013: Work Readiness training implemented October 2013: Work Ready Now training introduced January 2014: Work Ready Plus training introduced February 2014: Pre-Work Ready training introduced Evaluation of the Guyana SKYE Program 13 decisions might include modifying program activities or foci, creating or strengthening partnerships, or altering the timeline or budget of the project. In order to feed into these decisions, the evaluation examines whether program activities are creating the anticipated impact, the validity and completeness of the program logic, and recommended changes.7 Furthermore, some of the challenges youth face in Guyana are similar to those challenges faced in other parts of the Caribbean and Caribbean Basin. As such, this evaluation may assist USAID’s Barbados and Eastern Caribbean (USAID/BEC) Regional Mission or other CBSI stakeholders in determining whether and how the SKYE approach might be useful in other countries in its current or modified form. Evaluation Questions The evaluation team focused their data collection and analysis on the following SOW questions: 1. Are activities sufficiently coordinated and building toward the achievement of strategic objectives? 2. How effectively has the project engaged stakeholders at the local and national level? 3. What are the prospects for the sustainability of the results produced/approaches being utilized by this project? 4. Are there any recommendations to enhance project results within the project’s current period of performance and budget? 5. Are there any recommendations to expand or enhance project results should additional resources become available? Given the complex nature of the first evaluation question, the team divided this question into three components: targeting, implementation, and outcomes. The first item explores the extent to which SKYE is reaching the intended population. The second item investigates programmatic components and their interrelationship. The third assesses programmatic success in meeting intended outcomes. For question two, the team examines the engagement of a variety of stakeholders in the SKYE project. These stakeholders included the national government, regional and local governments, the private sector, educational and vocational institutions, the legal system, families and communities, and local and regional NGOs and CBOs. For the third question, the team assesses the potential sustainability of SKYE project activities and outcomes through an examination of buy-in, financial sustainability, policy developments and institutionalization of programming, and the development of regional and local capacity. Upon presentation of the findings and conclusions of the first three evaluation questions, the team then responds to questions four and five by providing detailed recommendations of how the project may be enhanced or expanded within the current period of performance and budget as well as if additional resources were to become available. 7 USAID Statement of Work. 2014. RFTOP Number SOL-538-14-000002 Mid Term Program Evaluation of USAID Skills and Knowledge for Youth Employment Project. 14 Evaluation of the Guyana SKYE Program EVALUATION METHODS AND LIMITATIONS Data Collection A seven-person team of specialists from SI and TCG conducted fieldwork over a three-and-a-half week period in September 2014. The evaluation team reviewed background documents and held a series of planning calls before beginning fieldwork. Upon arrival in Guyana, the expatriate members met local counterparts in Georgetown to conduct a team planning meeting, solicit youth input on data collection instruments, and pilot test data collection protocols and instruments. Preparatory activities were completed and all sampling plans, instruments, and protocols were finalized and approved by the Contracting Officer’s Representative (COR). Select team members met with the COR for a formal inbrief to ensure consistent understanding of USAID’s needs as well as the evaluation’s users, uses, and questions. The whole team began data collection in Region 4 and divided into sub-teams for data collection in Regions 6, 9, and 10. This approach enabled the team to ensure systematic application of methods (during their time in Region 4) and a cost-effective approach to maximize data collection across a wide geographical area (splitting into sub-teams). The team had a number of check￾ins throughout this process and reconvened following completion of fieldwork to conduct initial data analysis. While the team initially planned on working in Regions 3, 4, 9, and 10,8 Region 6 replaced Region 3 at the request of USAID. As such, data collection was not undertaken in Regions 3 and 5, though the team did survey some youth from those regions through their visits with employers, the New Opportunity Corps (NOC), and BK International (a construction and quarrying company). Given that more than 90 percent of program participants came from the four regions visited by the team, and because these regions represented a diversity of geographic and ethnic contexts, the team felt comfortable with the final regional sampling strategy. The team utilized a mixed-methodology approach to this evaluation, combining administration of a large-scale survey (quantitative) with in-depth interviews, focus groups, program observation, and a literature review (qualitative). This approach allowed the evaluators to obtain and document training results and programmatic processes from multiple perspectives. In order to maximize the fidelity of data and, ultimately, the utility of the evaluation, the team convened a group of youth participants (the Youth Advisory Group, or YAG) to provide feedback on the appropriateness of data collection protocols and instruments. The YAG representatives were selected such that each sampled region would have one high-performing youth and one youth that had struggled with the program. Recommendations were made by EDC and vetted by the evaluation team. The YAG provided critical 8 The technical proposal envisioned a bracketed, purposive sampling approach, with selection based on geographic, ethnic, economic, and programmatic elements. 9 6 10 4 Figure 4. Regions visited during evaluation Evaluation of the Guyana SKYE Program 15 feedback (particularly around phrasing of questions), significantly improving both the quantitative and qualitative instruments. Data collection methodologies utilized in this evaluation included:  Desk Review: A literature review began prior to field work and continued throughout the assessment. In addition to specific literature referenced in the SOW, the evaluation team referred to the following types of documents (see Annex III for a full bibliography): o Project design documents o Quarterly and annual reports o Monitoring data o Relevant assessments and evaluations o Secondary data on economic and contextual factors  Youth Survey: The evaluation team designed the sampling approach to provide findings representative of the SKYE population as a whole as well as within Region 4. The former was important to assess programmatic experiences across a broad geographic range, while the latter was necessary because of the importance of the region (highest degree of programmatic saturation and containing the national capital, largest population center, and primary economic hub). Following minimum sample size calculations9 and a purposive oversampling in non-capital sampled regions, the evaluation team aimed for a total of 400 interviews across the four sampled regions and 240 interviews in Region 4. While the team did not intend to survey youth outside of the target regions (4, 6, 9, and 10), enumeration at the NOC and BK International sites resulted in limited numbers of youth from Regions 3 and 5. The evaluation team administered a total of 397 surveys over three and a half weeks of field work. The distribution of the completed surveys can be seen in Table 1. Table 1. Regional distribution of completed surveys Region 4 10 6 9 3 5 TOTA L n 233 70 70 13 7 4 397 % 59% 18% 18% 3% 2% 1% 100% 9 Sample size was calculated to yield a representative sample within a 95% confidence level and within ±5% precision. Following Cochran, W. G. (1963. Sampling Techniques, 2nd Ed., New York: John Wiley and Sons, Inc.), two equations were used to calculate the necessary sample. Equation 1 calculates the sample size needed for a large population, while equation 2 applies a finite population correction to adjust for the relatively small number of SKYE beneficiaries. In equation 1, N0 is the calculated sample size, Z is the z-score (1.96 for a 95% confidence level), p is the estimated population proportion (0.5 for maximum conservativeness), and e is the precision level (.05 for ±5%). The first equation yields a sample size of 385. In equation 2, N is the population size, or total number of trained beneficiaries (1283 for total SKYE population and 649 for Region 4, as per SKYE monitoring records). Adjusting for finite population yields a calculated sample size of 296 for entire population and 241 for Region 4. Equation 1 Equation 2 16 Evaluation of the Guyana SKYE Program The sampling frame was derived from program monitoring data consisting of youth applicants (current as of August 27, 2014). The evaluation team called randomly selected phone numbers and invited respondents to central locations for enumeration, while those unwilling or unable to come were administered a modified version of the survey via phone. The survey was voluntary, and all youth were informed of their right to refuse to participate in the survey as a whole or any specific items therein.10 All travel costs were reimbursed by the evaluation team, and a small snack was provided to youth who participated in person. Table 2. Survey sample and applicants, by gender Sample Applicants Male 72% 78% Female 28% 22% Given the elapsed time, the voluntary nature of the survey, and the opportunity costs of participation, the evaluation team was concerned about selection bias. Comparisons between population parameters and sample statistics provide support that, at least on the indicators for which SKYE had youth data, the two groups were broadly similar. Reflecting programmatic targeting, the majority (72%) of survey respondents were male. Table 2 presents the comparison between the sample and population.11 Respondent ages ranged from 15 to 31, with mean age of 19. The average respondent was one year older than the average applicant, with the difference stemming from the time elapsed between application and participation in the study. The instrument was designed to gauge respondent demographics, experiences with the program, perceptions of the program, a comparison of programmatic components, and evidence of programmatic outcomes. A copy of the survey is available in Annex II, and the final, cleaned and anonymized dataset was made available to USAID.  Key Informant Interviews (KIIs): The team conducted key informant interviews with representatives of EDC/SKYE and GoG (Ministries of Culture, Youth, and Sports; Education; Labour; Legal Affairs), as well as with regional government ministries, educators, trainers, coaches, implementing partners, employers, judicial representatives, private sector affiliates, and others. The semi-structured key informant interview guide included a number of questions aimed at all key informants as well as tailored questions for specific stakeholder groups. Sixty￾nine KIIs were conducted over the four targeted regions. KIIs were recorded and transcribed for qualitative coding and analysis. The KII guide can be found in Annex II.  Youth Interviews: A subset of youth were selected during the survey to participate in qualitative interview that built of their survey responses. The youth interview protocol, available in Annex II, probed further into youth experiences and their recommendations for the SKYE project. Forty youth were interviewed, and, as with the KIIs above, interviews were recorded and transcribed for coding and analysis.  Focus Group Discussions (FGDs): Nine focus group discussions with youth participants were held to better understand youths’ self-identified goals, challenges, and motivators as well as how SKYE did or did not fit into those areas. From there, youth discussed aspects of SKYE they felt were working well and not working well to determine recommendations for the program 10 All youth were read an informed consent statement (see Annex II) and all survey items included “don’t know” and “no response” options. 11 The proportion of males was 6 percentage points lower than the total number for SKYE applicants, perhaps reflecting selection bias. However, the sampling frame was derived from program applicants, not graduates. To the extent that sex mediated completion rates, the sample may better reflect graduates than applicants. Regional sex composition ranged from 77 percent (Region 6) to 62 percent (Region 9). Evaluation of the Guyana SKYE Program 17 moving forward. The focus groups included mixed-gender FGDs as well as some with only young men or only young women to ensure participants were comfortable discussing their perspectives. The FGD guide is available in Annex II.  Direct Observation: Team members sat in on portions of live trainings. The trainings observed included Pre-Work Ready and Work Ready Plus, and provided an opportunity for the team to learn more about youth prior to graduation as well as the organization and dynamics of the trainings themselves. Data Analysis The team employed double data entry and paper-based reconciliation of discrepancies to ensure that survey data was digitized accurately. Identifying information (e.g., name, sex, and home town) were entered separately from survey data to ensure anonymity of responses. The two datasets were merged, cleaned, and analyzed using the Stata software package. Quantitative analyses included summary statistics, correlational analysis, and multiple regression analysis.12 Data gathered as part of the qualitative data collection were transcribed and coded using the Atlas.ti software package. Once coded, the team used complex queries to identify emerging themes and triangulate data with the quantitative findings. Limitations As mentioned above, the team collected data in four of the six SKYE regions. The two regions not included in this study were the two regions where the program has most recently expanded. Furthermore, Region 3 has a different economic context than all other sampled regions, focusing on industry and commerce. As such, it is possible that conclusions drawn from data in Regions 4, 6, 9, and 10 may not be generalizable to realities in the omitted regions. Difficulties in accessing youth beneficiaries present a significant limitation to this evaluation. In order to reach out to youth participants (including graduates and dropouts), the team used records from EDC containing participant contact information. A significant portion of the phone numbers were out of service or no longer in use. While the team was able to use alternate means to contact some of these individuals, it is possible that those youth who were unable to be contacted are different from those youth who were reachable by phone. Therefore, the evaluation findings are not necessarily generalizable to the population of youth who were not contacted. Similarly, those youth who were willing to participate in the evaluation were either more likely to have closer ties to the SKYE program or had more free time on their hands, than those who were unwilling or unable to participate. While every effort was made to ensure participation by youth from various age groups, geographic locations, literacy levels, and socioeconomic strata, the potential biases inherent in those responding make it difficult to confidently state that the experiences and perspectives of the evaluation respondents are representative of the views of their peers who were not included. Sample size calculations were predicated on ability to generalize to the entire SKYE population, with a Region 4 disaggregation. Given the smaller number of completed surveys in the other three sampled regions, it is not possible to statistically infer findings from this report to the total population of youth trained in Regions 6, 9 and 10. 12 The regression model used in this report is: Yi = 0 + 1X1 + 2X2i + … + kXk + i Where Yirepresents the outcome (dependent variable), 0 is the constant (y-intercept), 1–k capture the effect of controls X1–Xk (independent variables), and i is the error term. Unless otherwise noted in the body of the report, all regressions use a standard set of control variables consisting of: age, sex, education level (highest level reached), region, parents (do you live with either parent), and household wealth (see page 24 for discussion of this variable). 18 Evaluation of the Guyana SKYE Program Similarly, youth interviews were typically administered to those youth whose responses in the survey spurred further questioning. As such, many of the youth interviews were given to those who the team felt truly met the “at risk” criteria. These responses proved invaluable to providing enhanced understanding around the target population and their outcomes, but again are not representative of all SKYE youth. A final limitation to note is the lack of a comparison group or strong baseline data against which to measure the post-SKYE experiences and achievements of participants. With regard to the former, individuals who participated in SKYE may have internal characteristics that distinguish them from their peers who decided not to participate or who were not reached through SKYE recruitment. As such, extrapolating the experiences of SKYE participants to all at-risk youth would not be advisable. With regard to the latter, without an ability to measure change over time (compounded with lack of a counterfactual), it is not possible to definitively attribute any outcomes solely to the SKYE program. Given that the program is designed to effect changes that are dependent on the macroeconomic context, programmatic performance on job placement (as one example) is intertwined with the performance of the Guyanese labor market. As such, any generalizations about success should be taken in the context of the environment. FINDINGS, CONCLUSIONS & RECOMMENDATIONS The sections below have been divided by the first three evaluation questions, which focus on the achievement of strategic objectives, internal coordination, stakeholder engagement, and sustainability. The final two evaluation questions focus on recommendations for project enhancement within and outside the resources that are currently available to the implementing partner, EDC. As such, the evaluation team has structured the report such that recommendations are listed after all findings and conclusions from the first three evaluation questions. Question 1: Are activities sufficiently coordinated and building toward the achievement of strategic objectives? In order to answer this evaluation question, it is necessary to unpack the SKYE theory of change and explore how the program has functioned at each level of the causal hierarchy. In doing so, the evaluation team presents a segmented version of the logical framework, using three parts. The first section analyzes program targeting and recruitment strategies. The second section examines the degree to which SKYE has equipped youth with the skills they need to gain employment, re-enter school, or start their own businesses. Finally, the third section explores the extent to which the program has been successful in achieving its intended, final objectives of increasing employment and reducing criminality. Question 1: Findings I. Targeting and Recruitment Defining Risk Funded under the Caribbean Basin Security Initiative, SKYE’s stated goal is “to reduce crime by enhancing skills and increasing economic opportunities for at-risk youth.” In order for the program theory of change to take hold, it is a necessary precondition for SKYE to reach at risk youth (i.e., those Evaluation of the Guyana SKYE Program 19 youth who would have been perpetrators of crime absent the intervention). According to SKYE documents, the program targeting strategy is as follows: Project activities have focused on supporting youth aged 15 to 24 who are: i) school dropouts; ii) youth who have completed formal education, but did not achieve minimum academic requirements to gain a certificate to access higher education or employment; and iii) those with education, skills, and behaviors that are insufficient for integration into the workforce, including being in the juvenile justice system.13 When asked what constitutes “at risk” during key informant interviews, respondents, including both those with a direct role in SKYE, such as coaches and trainers, and external stakeholders, such as representatives from the juvenile justice system and the Ministry of Labour, Human Services, and Social Security, did indicate that there was some alignment with the definition of “at risk” outlined above. The most commonly identified features of a youth who can be categorized as at-risk included: school drop￾outs, low literacy levels, and criminal behavior or interfacing with the juvenile justice system. While the preceding would seem to provide clear guidance on how to screen for eligibility criteria, discussions with key informants indicated that there are multiple other categories that would predispose a youth to qualify for the program on the basis of being ‘at risk’ and acknowledged that there was no hard-and-fast rule for determining whether or not a youth would qualify as “at risk.” For example, additional features that were frequently mentioned were youth who came from impoverished backgrounds or geographic regions with a high prevalence of criminal activity or negative influences such as drug use and/or gangs. Furthermore, while mentioned on a less frequent basis, respondents also stated that youth from single parent homes, those with mental health issues, victims of physical or sexual abuse, and individuals who use or abuse drugs and alcohol are also ‘at risk’. During key informant interviews with the trainers and coaches, respondents also described the recruitment process and modifications to that process over time. The majority used a broader definition of ‘at risk’, stating that this was guidance that they had received from the head office. As one respondent stated: We work with at-risk youths, or youths they think are at-risk. We have a profile we got from head office that would outline what a youth has for him to be a SKYE youth, so if the youth is from a high-risk community, a lot of crime, violence or what not between the ages of 15 to 24, well we mostly target males, with less than five CXC (Caribbean Examinations Council) subjects.14 What else? Oh, they may have skills, some amount of skills but no work experience, or sometimes you may find a youth with 5–6 subjects the subject they may have are grade 3 or grade 4 in it, those are not really what employers look for. So most of the time when I’m out recruiting, I would go into areas that I know, like they have certain areas where you think you may find 1–2 youth or if not you may be able to share flyers to persons who may know or youths, all like (name of location), that is a high risk community. (Coach) In sum, data gathered through key informant interviews indicated the following qualities to characterize youth at risk:  School drop-outs  Low literacy  Criminal behavior  Low socio-economic status 13 Education Development Center (EDC). November 2012. Skills and Knowledge for Youth Employment (SKYE) Program Description. 14 Coaches and trainers mentioned that the guidance regarding the number of CXCs in the recruitment process was not implemented from project inception but was introduced as a means to better target at-risk youth. 20 Evaluation of the Guyana SKYE Program  “Bad” neighborhood/community  Family structure  Mental health issues  Drug and alcohol use/abuse  Physically or sexually abused Program Target Drawing from the qualitative interviews, the team attempted to gauge the types of youth that were targeted for the program and determine whether they aligned with the broader understanding of risk. During qualitative interviews, respondents were asked either who the most appropriate target of the program should be or whether or not those who had been enrolled were the correct target. Broadly, the information provided indicated that youth that were being targeted did meet a number of the criteria defined above. However, a number of respondents expressed concern that the youth being enrolled were not as at risk as others. The data also revealed, however, that the youth targeted have shifted over the course of SKYE. In the earlier days of programming, youth who were recruited may have been school dropouts or had not passed a substantial number of CXCs. More recent enrollees include youth who have extremely low levels of literacy, come from more challenging communities, and have been exposed to violence, illegal drugs, or other criminal activities in addition to those youth who have dropped out of school or passed a very limited number of CXCs, if any at all. While broadening the scope may have increased the numbers of eligible youth, it has presented challenges for those who are training and coaching youth as well as those who employ youth. Key informants expressed concern that they may not be able to meet the needs of youth who have substantial challenges in the time frame allotted and with the current resources. As one trainer said, “But by the end of 8 weeks, I try to get them to read and that’s a tall order. I know you can understand that especially with some…that are brought in knowing nothing. Not being able to identify some of the letters, not being able to identify words, not even able to understand words or anything, to take them in two months to the level where they can read.” The challenge of literacy and numeracy described above was also described by several employers. Some reported that those skills remained a substantial barrier for the youth they employed. Recruitment In order to understand how the targeting of youth played out operationally, the evaluation team turned to recruitment. When asked how they first heard about the SKYE program, the most common answer was from their coach (37 percent). At 23 percent, “other participants” was the second most common answer (Figure 5). “And to a larger extent, they were looking at the recruitment process that they had were taking youth who could easily find employment in my estimation or my officers’ estimation. These youths were not at high risk of violence or being engaged in violence so to speak.” -Trainer “I think a lot would’ve been a little easier to get the ones who are just got caught up in the system and work with them. Um, dealing with at risk is an entirely different larger effort than a program to catch because there are more things to do at that level. You know, because one is preventative and one is rehabilitative.” -Coach Evaluation of the Guyana SKYE Program 21 When all non-institutional responses were tallied, a total of 226 referrals came from community members (participants, relatives, friends, churches, other community members).15 This number surpasses all the other sources of recruitment, indicating the important role the community plays as a recruitment conduit. Half of the respondents reported knowing other youth who were interested in the program but did not apply, indicating a large potential pool of additional program participants. According to interview data, the manner by which coaches recruited youth to the SKYE program somewhat varied. Methods that the coaches used to identify potential program participants included: recommendations from members of the community or community-based organizations (CBOs) such as churches or youth groups, referrals from the juvenile justice system or probation officers, community walks (walking around the community, making door-to-door visits to homes and organizations), and referrals from SKYE headquarters. During qualitative interviews, when asked what was discussed with coaches when they stayed in touch after training, some youth responded that coaches asked youth for referrals of other youth who may be interested in the program, though the data do not indicate whether the youth were given guidance on recruitment guidelines or if it was believed that youth would be engaged with other youth who had similar at-risk characteristics as themselves. This is demonstrated in the following from a young man in Region 10, when asked if he was still in touch with his coach: Well, she be calling me to see if I could get friends, youths, since the program is now starting. So, they’re recruiting and she wants to know if I have any friends or youths who’re interested. And then, yeah well, I could probably get a few. And then she could be calling to find out when they gonna be coming for the interview, if they have time, if they have a job and still want to do the program. So, she be calling. Coaches reported that recruitment consumes a substantial amount of their time. In some areas, they worked with trainers to help reduce the burden. The trainers also indicated that they were held to a quota in order to have sufficient numbers of youth to train. One coach suggested that partnering with a community liaison would help reduce burden and provide greater access to youth in need. In the next 15 As respondents could select multiple responses, this metric includes some double-counting. Figure 6. How did participants learn about SKYE? “That’s what you have to get in now to the communities, or see if you can find a mobilizer for it, mobilize or you do your own mobilizing, which when you do your own, you observe that it’s challenging, especially in new communities” -Coach 0 25 50 75 100 125 150 Community Institutional Figure 5. How did participants learn about SKYE? 22 Evaluation of the Guyana SKYE Program section, the team explores whether the population surveyed matches with the intended program target of at-risk youth. Risk Profile of Survey Respondents Table 3. Parental status of survey respondents Given the complex nature of the notion of risk, it was important to use several constituent factors in order to understand the risk profile of survey respondents. The evaluation team deconstructed risk into five key components: broken households, teenage pregnancy, low socio-economic status, history of criminal behavior, and educational underperformance. Broken households (Table 3) Nearly a quarter of the surveyed youth lived without either parent, though only 5 percent listed themselves as household head. Fathers were less 27 percentage points less common as mothers, and at 4 percent of the sample, father-only households were a rarity. Teenage pregnancy Thirty respondents (8 percent) reported having children at the time of the interview, with females more than four times as likely to have children (p=0.00).16 Teenage pregnancy itself limits the range of options available to parents. This effect can be mollified with a strong social support network and a supportive partner. However, the majority of respondents with children (60 percent) reported being single and fewer than half were living with a partner at the time of the interview. The age at which these youth had their first child ranged from 14–24, with females starting, on average, more than 2 years earlier than males.17 Criminality Perhaps the best predictor of future criminal behavior is a history of transgressions with the law. When asked whether they had ever been arrested, 18 percent of the youth responded in the affirmative. While it is difficult to deduce the true extent of criminality within the sample,18 it is clear that a significant portion of the respondents (and by extension SKYE participants) have a history with the legal system. Qualitative data gathered indicates that SKYE has made strides in advocating for alternative sentencing among members of the judiciary. However, only five youth (1 percent) reported having received alternative sentencing. During key informant interviews, a number of representatives from the judiciary system did see SKYE as a viable alternative to the NOC or time spent in prison. However, while SKYE 16 Throughout the report, p-values are presented to notify significance level of regression analyses. P-values are the estimated probability that there is no statistical difference between two groups (i.e., probability that the null hypothesis is correct). The smaller the p-value, the more likely that two groups have truly different average values for a given variable. The following table presents p-values and the associated significance levels, following the formula [significance level = (1 – p-value) * 100]: P-value Significance Level 0.1 90% 0.05 95% 0.01 99% 17 Average age of females = 18.2. Average age of males = 20.3. 18 Given the direct nature of the question, it is likely that the evaluation team encountered response bias and the true population parameter is even higher. On the other hand, 14 of the 397 interviews were conducted at the NOC, where—by definition—all youth have criminal records. This sampling approach introduced some modicum of selection bias into the sample statistics. Parental Status % Both Parents 44% Mother Only 31% Neither Parent 22% Father Only 4% Evaluation of the Guyana SKYE Program 23 had its own criteria, representatives from the justice system said there were additional criteria they used when considering whether or not to refer youth to the SKYE program as a form of alternative sentencing. They stated that they tended to look for youth who were not currently employed, who had committed lesser offences such as wandering or petty theft, who were willing to participate, as well as those with a sufficient safety net to support them while they attend SKYE. There was no way to measure the severity of criminal offenses directly, and only seven (2 percent) admitted to having been to prison. Criminality (defined as having been arrested) was found to be significantly associated with a number of factors: males were more than twice as likely to have been arrested (p=0.05), while more education and higher SES were associated with lower arrest rates (p=0.09 and p=0.04, respectively). While in an absolute sense the numbers were low (0.5 percent), 20 youth reported having dropped out of school due to a range of criminal behaviors, including “stealing,” “violence in school led to suspension,” “sent to NOC,” “got in trouble with the law through stealing and gambling,” and “followed bad company.” Socio-Economic Status One quarter of respondents reported being financially independent at the time of the interview (i.e., receiving no support from family or friends) and 39 percent supported at least one person financially (median = 1.5). Of the youth who were financially supported, parents were the most common source of support (62 percent). Aside from direct economic support, one third of respondents reported receiving remittances, though the magnitude of the transfer was limited (77 percent of recipients receiving between 1–20,000 GYD per month).19 Youth that received direct financial support were more than two times as likely to receive remittances (p=0.01) and 70 percent less likely to provide financial support to others (p=0.00). Figure 7. Asset ownership 19 At the time of writing, the GYD to USD exchange rate was 0.0049 ($1 USD = 205 GYD). Accordingly, GYD 20,000 corresponds to USD $97. 0% 20% 40% 60% 80% 100% Electricity Cellphone TV Refrigerator Water Indoor Toilet Computer “So far since I have been here persons that I would have sent to the SKYE project, I think they were all…under sixteen. The persons who are usually above that age they are usually employed. So, when it comes to alternative sentencing maybe something like the community service would work and they ask for a particular time.” -Representative from the Judicial System 24 Evaluation of the Guyana SKYE Program Household size ranged from Household size ranged from 1–20 people, with the average containing 5 individuals (including the respondent). When compared with the number of rooms (mean = 5.5), the average household had one person per room (mean = 1.02). Given that the “rooms” metric captured all rooms in a house (bedrooms, bathrooms, kitchens, living rooms, etc.), the majority of youth lived in houses with shared bedrooms. The evaluation team used ownership of household assets and access to household services as a proxy for socio-economic status (SES). Figure 6 presents average asset ownership (columns) and the largest regional differences in ownership (error bars). Access to electricity and cellphones was very common across all the regions, while luxury goods (e.g., computers) and piped water were both less common and exhibited much more variability with regard to ownership. To make asset data more useful as a predictor of SES, the evaluation team pooled the seven items and calculated an asset index using principal components analysis (PCA).20 The index yields valid estimates of respondent asset ownership, though the measure is an imperfect proxy for SES because of limited variability in ownership of some assets (e.g., electricity) and non-inclusion of other important predictors of household wealth (e.g., income). 21 Education Eighty-two percent of respondents were not attending any formal schooling at the time of the survey. One in three were secondary school dropouts, with the dropout rate 11 percentage points higher for males than females (35 and 24 percent, respectively). Of the youth not attending secondary school at the time of the survey (i.e., those who have not yet had an opportunity to take the exams), 45 percent did not pass a single CXC, and more than two-thirds had fewer than 5 subjects. Table 4. Education, by gender Three in four youth reported reaching secondary school as the highest level of education. As evidenced in Table 4, females were twelve percentage points more likely to have concluded schooling at the secondary level, while males were thirteen percentage points more likely to have enrolled in technical/vocational institutes. While the table and associated graph in Figure 7 indicate differential education outcomes between males and females, multiple 20 In order to construct a single metric, we aggregated ownership information and attached weights to differentiate common and luxury items. In lieu of assigning these weights judgmentally, the evaluation team used the PCA method to calculate the one factor score that describes the most variability in asset ownership. This measure, while devoid of intrinsic meaning, allows for sub-group analysis and acts as a key control variable in multiple regression analysis. Following Filmer and Pritchett (“Estimating Wealth Effect Without Expenditure Data or Tears: An Application to Educational Enrollments in States of India” Demography 38, no. 115-32 [2001]), we construct a wealth index A for each individual i: Where aik is the value for asset k for individual i, ak is the sample mean, sk is the sample standard deviation and fk are the weights associated with the first principal component. 21 It would be much more informative to use continuous measures (e.g., household income) and/or additional indicators beyond access to assets/services (e.g., household conditions/construction). However, given survey length, respect for respondent time, and the prioritization of other survey questions, the team opted to utilize a limited approach to estimating SES. Male Female Primary 1% 0% Secondary 71% 83% Tech/Voc 25% 11% Tertiary 3% 5% Evaluation of the Guyana SKYE Program 25 regression analysis does not substantiate any statistically significant differences. 22 Lack of gender￾disaggregated differences with regard to grades notwithstanding, females did have an average of 1.1 additional CXCs as compared to males (p=0.00). This latter phenomenon was largely driven by the relatively high number of males who had not passed a single subject (47 versus 35 percent for males and females, respectively). Figure 8. Education, by gender II. Program Delivery The evaluation team identified four primary conduits through which beneficiaries experience the program: coach, trainer, training (activities, materials, etc.), and other beneficiaries. When asked to rank the programmatic component in order of preference, the training was overwhelmingly the most popular (47 percent of youth ranked it as their favorite). By comparison, the second most popular component was trainers (22 percent). The frequency with which the training was chosen as the favorite indicates that youth value the skills derived from the SKYE program more so than the mentoring, socialization, or job-placement functions. Role of Coach Qualitative data indicate that the role of the coach is not something that can easily be defined. According to self￾reports, the coaches have definite roles and responsibilities including recruiting youth to the program, providing youth with potential linkages to jobs or further education, assisting trainers with particular activities, developing livelihood plans with youth, and monitoring youths’ progress toward their goals. While coaches have established roles and responsibilities, many coaches say that their jobs often exceed these tasks. Coaches spoke of developing strong relationships with youth, playing a substantial role in their lives. On the other hand, coach involvement with youth varied 22 While there was a strong, statistically significant relationship between years of education and respondent sex, incorporating controls removed any significance between the two variables. Respondent age (males in the sample were, on average, older) was the key driver of differential educational attainment between the sexes. 0% 10% 20% 30% 40% 50% 60% 6 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 27 Primary Secondary Tech/Voc Tertiary Female Male I’m their mother. I’m their friend. I’m their sister, you know? So I’m their coach, and they can always feel free and come and talk. Or they could come and hug you up and say “Miss, how was your day? You know you miss us.’ Get a kiss on the cheek and an ‘I love you’ and they gone. It’s a joy for me, really.” -Coach 26 Evaluation of the Guyana SKYE Program significantly across the program, with some coaches serving in a much more active capacity than others. A number of youth mentioned that their coach had little interaction with them after the program and only responded when they reached out. Coaches’ own descriptions of their roles and responsibilities also differed significantly with respect to the level of involvement with families, places of employment, and mentorship. Some coaches also indicated that, given the challenges the youth face, they would like additional training in how to counsel youth. When asked if they served as counselors, they said they did not but that they often had to provide guidance, and, as one coach phrased it, “a listening ear.” They also reported not having received central guidance on counseling services to recommend to their youth. While youth ranked coaches as their third-favorite program component, qualitative data showed that a majority of youth had a positive experience with their coach. For example, one youth reported that her coach was an inspiration to her: She’s always on the go, she’s always busy, it’s like, she takes care of all her personal business and then finds time to be with us as youths and coach. So I see that as inspirational because someday I like to divide my time and be dedicated to two things at the same time, which is a job and spending time with something else, or somebody else. Many youth reported maintaining a connection with their coach after the end of the program. It should be noted that while the majority of youth interviewed had positive things to report regarding their coach, there were a number of youth who did say that they didn’t have a relationship with their coach, that they had lost contact with their coach, or that their involvement in the program had ended. Role of Trainer According to qualitative interviews with the coaches and trainers, the role of the trainer has slightly more clearly defined parameters versus those of the role of the coach. Specifically, trainers are expected to assess the skill level of the youth and facilitate the trainings. However, the majority, similar to the coaches, stated that they also played a role in counseling the youth. Two trainers indicated that by counseling youth, they were better able to meet their needs in terms of training. One trainer said that by counseling youth, they were able to become more familiar with their literacy and comprehension, which helped them target the training to the needs of the youth who were enrolled. Another trainer shared that by counseling one of the participants, she was better able to understand youth misbehaviors and address them during the training. She stated: There are a lot of the boys here that is considered to have misbehaviors that might scare you, but then you know one day one of them pushed his head and said, miss can I speak with you, I never spoken to him before because I had a slight dislike for his behavior, it wasn’t him, it was his behavior. So I had never spoken to him. And one day he pushed his head and said, Miss can I please speak with you, so I said, what you would like to speak to me about, he said I see everyone coming in and talking to you and I would like to talk to you. So he came in and he talked for like two hours, I said nothing, and I then understood the reason why he was behaving the way he was. “And so, our role as Pre-work Ready trainers is to ensure that they develop those skills. We have to ensure that we look at the phonics, the phonology, morphology, orthography – everything, everything. We have to develop all those skills because they come with practically nothing at that stage.” -Trainer Evaluation of the Guyana SKYE Program 27 Like coaches, the trainers also indicated that some additional preparation in counseling would help them to improve their skills and meet the needs of the youth who partake in the training. Additionally, one Pre-Work Ready trainer said that it would be helpful to learn these skills, since the majority of the curriculum was not focused on the development of soft skills, as is the case in the other types of training. Most coaches and trainers indicated that they collaborated with one another in terms of counseling youth and offering guidance and support, as well as supporting one another with their roles and responsibilities. Youth seemed to confirm what the trainers and coaches had shared. The qualitative interviews with youth indicated that trainers played a larger role than simply training. In fact, several youth stated that they had remained in touch with their trainers. As one female from Region 6 stated, “And even up to now my trainers will be like if you need any help, you call us.” In addition to training and counseling, some trainers also stated that they were involved in recruitment of youth to the program. At least two trainers stated that their implementing organization was asking their trainers to recruit youth, though it appears as though only one organization was carrying this out as a mandate through which a certain number of students needed to be recruited by trainers. Role of Training As demonstrated in Figure 8, youth consistently identified the trainings as their favorite part of the SKYE program. Several key informants mentioned the rigorous vetting that the curriculum had undergone. The result was a set of trainings that youth said were engaging and fun. During the interviews, the evaluation team asked youth what it was about the trainings that they liked. Most pointed to the activities. The youth liked that the trainings were interactive and gave them “real life” practice. Additionally, youth found the content of the trainings helpful. Many said they provided information that they hadn’t thought of before or been able to practice before, though the favorite/most useful topic varied from youth to youth. Figure 9. Youth ranking of program components (1 = least favorite, 4 = most favorite) 54 59 98 138 67 114 85 89 64 105 102 174 88 81 74 44 0 50 100 150 200 250 300 350 Training Trainer Coach Peers 1 2 3 4 “The modules are designed basically for work ethics. That’s ugh, what it’s designed to do, um. But these are at risk youths. Youths that are easy prey to criminal activity and so forth. So in the dispensing of the modules, you cannot just teach a curriculum without, um, you know, applying bits of psycho-social skills as you interact.” -Trainer 28 Evaluation of the Guyana SKYE Program Because at-risk males are the primary target of SKYE, the evaluation team sought to understand of how youth perceived the uneven distribution of males and females in the trainings. The majority of both male and female youth stated that they preferred that the trainings be a mix of male and female, though some females said that they wished it had been more evenly balanced. Trainers and coaches consistently stated that having female youth in the classroom served as a motivator for the male youth. When asked what impact this had on female students, the most common response was that it was good for the female participants as well, as it taught them self-confidence. While most female youth echoed this, a few did say that they had felt uncomfortable in the presence of their male counterparts. It should also be mentioned that a few male respondents stated that they had been encouraged to engage in particular type of careers as they were masculine choices. One youth said that he was actually interested in becoming a fashion designer, but feared being “called gay.” III. Intended Outcomes While the objectives of the SKYE program are multifaceted, the evaluation team decided to focus on beneficiary skills to measure intermediate outcomes. These intermediate outcomes include employment, education, crime reduction, and the development of livelihood plans. Employment Outcomes The features that the evaluation team examined in relation to employment outcomes include employment skills, linkages to employment, employment rates, job seeking/unemployment, and job quality. Employment Skills In order to assess the skills youth had gained through SKYE, the team examined three data sources: survey data on respondent perceptions, qualitative interviews with youth, and qualitative interviews with employers. The team examined self-reported confidence on a set of seven skills that were addressed through the SKYE training (Figure 9). Large majorities of youth said they were confident (score of 4 or 5) with their ability to use all seven skills. With the exception of entrepreneurship, three quarters of youth said they were confident in all the skill areas.23 The skills that solicited the greatest amount of self-reported confidence were the least technically demanding (e.g., interviewing for a job and interacting with superiors). Entrepreneurship was the skill respondents felt least comfortable with. It should be noted, however, that the majority of youth with whom the team spoke had not undergone training in entrepreneurship, nor did they attempt to start their own businesses. One in five youth reported using this skill since completion of the SKYE training. As such, it is not surprising that this was the skill with which they felt least comfortable. Financial planning, writing a CV, job searching, and career planning were ranked in the middle. Female respondents were, on average, more likely to report being confident in their employability skills (especially writing professional CVs, where they were over 14 percentage points more likely to report being “very confident”). Males were slightly more confident in their ability to interview for jobs and practice financial management. 23 61 percent of respondents reported being confident in their ability to start a business “Yeah, there were things that it was very important, that I find very interesting: money management, how to manage time, money, how to go about interacting with employers/managers, preparing for interviews…” -Male Youth, Region 10 Evaluation of the Guyana SKYE Program 29 Figure 10. Self-reported confidence in employment skills (1 = not confident, 5 = very confident) Qualitative interviews confirmed that youth have a fairly high level of confidence in the soft skills described in the survey. For example, one male from Region 4 explained that his newfound confidence helped him successfully complete an interview: But our trainers teach us about it so when I come, there I was, like proud about myself, doing an interview. . . . When I went home in the afternoon, I didn’t expecting a call because I told myself I did bad, but I still proud. After I get the call I was, like, thank god. I get through with this job and I go and get police clearance and my documents together. And I get them in-within one week time. I get them in and I started out to work. However, it is worth noting that when confidence was discussed, it exceeded the boundaries of employment. Instead, youth spoke of general increases in their confidence. While many youth were pleased with their progress, it was critical to triangulate self-assessments with representatives from employers of SKYE youth. These interviews yielded more mixed feedback. While some reported being pleased with the youth they had employed through the SKYE program, others reported facing some challenges. Those individuals who reported challenges tended to be those who interfaced with youth most directly, such as their on-site supervisors. For example, at one organization, while the HR department voiced their confidence in the group they were currently working with, the individuals who were working most directly with the youth reported facing some issues with them. It was stated that the youth took additional time and effort as new employees. It should be noted, however, that these youth were working and living on-site, away from home. Generally, the most frequently cited challenges employers reported facing included a lack of hard skills needed to complete the tasks at hand, low levels of literacy or numeracy, a lack of discipline both in terms of timeliness and behavior, and a lack of overall commitment.24 While these employers stated that they faced challenges, 24 It should be noted that the sample size of employers with whom the team spoke was fairly small, and that the data come from qualitative interviews. As such, it is difficult to say with certainty whether the challenges listed 18% 13% 14% 18% 19% 15% 26% 31% 40% 39% 37% 35% 42% 32% 45% 43% 43% 43% 41% 38% 29% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Financial Planning Interview Interact with Superiors Career Planning Write CV Job Search Start a Business 1 2 3 4 5 30 Evaluation of the Guyana SKYE Program the majority did state that they are still interested in engaging SKYE youth and are committed to the program. Link to Employment The evaluation team designed the survey instrument to include items corresponding to each stage in the SKYE theory of change, with questions corresponding to each of the programmatic logic model steps. Youth were first asked about what types of jobs they desire and whether SKYE (1) taught them about their preferred work sector, (2) connected them to trainings in their desired work sector, (3) connected them to employers in their desired sector, and/or (4) provided them a job in their desired sector. Recognizing that due to constraints beyond the scope of the program (labor markets, beneficiary competencies, etc.), the team also asked whether (5) SKYE provided a job in any sector. Summary findings from this module are presented in Figure 10 below. Males were 2.6 times more likely to still be employed at the SKYE-provided job (p=0.04). Interestingly, while fewer than 40 percent of youth reported receiving a job through the SKYE program, nearly 70 percent had learned about jobs in their desired sector. Figure 11. Link to employment, by gender Interviews revealed that youth as well as their families were disappointed that they had not been employed through SKYE. While the trainers and coaches with whom the evaluation team spoke stated that they did not promise the youth employment, many youth and their families believed that the SKYE program would secure them a job, and some did state that they were explicitly told that SKYE is able to find a job for the majority of youth graduates. The survey findings indicated that fewer than 40 percent of youth were connected to trainings and employers in their desired job sector, and fewer than 20 percent were linked to jobs in their desired sector. However, qualitative interviews revealed there may have been flaws in the questioning, as many youth subsequently reported that they had not told their coach about the type of job they most desired. Additionally, qualitative data indicated that they may have been connected with jobs in sectors other than the ones they desired, but that the youth did not accept the here are representative of all employers or if the challenges vary by type of employer. This would warrant further exploration. The team encourages SKYE to follow up systematically with employers to assess strengths and weaknesses of the program through a simple survey. This is described further in the recommendations section. 0% 10% 20% 30% 40% 50% 60% 70% 80% "SKYE taught me about my desired job sector" "SKYE connected me to trainings in my desired job sector" "SKYE connected me to employers in my desired job sector" "SKYE provided me a job in my desired job sector" "SKYE provided me a job in ANY sector" Females Males “We heard that before we finish the training, we going to get a job. As soon as we graduate, they going to help us look for the job. But it was nothing like that. As soon as we graduate is like they don’t check up on us, or nothing like that. So that’s what I didn’t like about it.” -Female Youth, Region 4 Evaluation of the Guyana SKYE Program 31 employment because it was not the type of job they wanted or would require them to work under conditions that they did not want. It should also be noted that the coaches reported being largely responsible for job placement on their own, with little support in introductions from SKYE headquarters. The link between SKYE headquarters and coaches was not clearly or uniformly understood, and the majority of coaches reported that they had limited or no interaction with other coaches to understand job opportunities and match youth with those opportunities. Employment Building on SKYE linkages to jobs and training, the team assessed youth experiences with employment. Eighty percent of respondents held a paid job at some point in their lives, with half that number employed at the time of the survey (38 percent). This is a higher unemployment rate than the national average for youth. According to the World Bank, information gathered from the 2012 Census indicates that 42 percent of Guyanese youth were unemployed at the time of data collection.25 The team did not gather information on the duration, quality or type of employment youth engaged in prior to SKYE. As demonstrated in Figure 11, males tended to be employed at higher rates both historically as well as at the time of the interview. Using multiple regression analysis, males were 2.7 times more likely to have had work experience (p=0.00) and were 2.4 times more likely to have been employed at the time of the survey (p=0.00).26 Employment was most common in Region 9, though the relatively small sample size should not inform any generalizations beyond the evaluation sample. Regions 10, 3, and 5 had the lowest rates of employment. 25 Information derived from: http://www.theglobaleconomy.com/Guyana/Youth_unemployment/. 26 For both analyses, regression models included receipt of financial support (binary) as additional controls. Both employment outcomes had significant (p=0.00) associations with age, with each additional year associated with a 5% increase in likelihood of employment. Youth that received financial support were almost 20 percent less likely to be working at the time of the survey (p=0.00) and 10 percent less likely to have worked at some point in the past (p=0.03). 80 83 73 80 81 100 77 64 38 42 29 41 41 46 26 27 0% 20% 40% 60% 80% 100% TOTAL Male Female 4 6 9 10 OTHER Employed Ever Employed Now Figure 12. Employment, by gender and region 32 Evaluation of the Guyana SKYE Program Job Quality In seeking to understand low workforce participation rates among respondents, the team examined the quality of jobs that youth currently had as well as their qualitative reports about some of the challenges they faced at previous places of employment. Through the survey, the team examined the number of jobs youth held, the salary, and average hours worked as well as their access to benefits. For those employed at the time of the interview, the vast majority (88 percent) had a single job, and the average youth worked 46 hours per week. While females were 14 percentage points less likely to be employed than males and worked, on average, seven fewer hours per week, they tended to have higher paying jobs (Table 5). This discrepancy is mostly explained by the relatively high number of males earning minimum wage (0–20,000 GYD/month). Job benefits were relatively common: 81 percent of employed youth reported at least one benefit, while 61 percent reported three or more (see Table 6). Half of working youth reported being satisfied (score of 4 or 5) with their job, compared to one in five expressing dissatisfaction (score of 1 or 2). Job satisfaction was found to be a complex construct, with no significant associations between self-reported scores and a wide range of explanatory variables.27 The one indicator that had a statistically significantly relationship with job quality was the number of benefits, where each additional benefit was associated with an increase of 0.3 on the 5 point satisfaction scale (p=0.00). Quantitative analysis of job quality is biased toward higher quality jobs. As evidenced by the rate of youth not employed at their SKYE-provided jobs and qualitative interviews, it was common for youth to have left their jobs. Of those who had left positions, the most common reasons included inconvenient shifts (early in morning/late at night, or shift work), high degree of job stress, and the physically challenging nature of work. One male youth described the long days he was required to work: They was saying 8 hour, but the place I was working was not giving 8 hour. [I was] starting at 6. Sometimes, I wake up and not feel like to go, and had to go again. So I go and did. My aunt said go for the week and quit cause it was very hard. When I come in I don’t feel like to eat, I just run in my bed and sleep. When I get up back, bathe and ready to get to work—it was very hard on me, I can’t able to help them do nothing inside the house, only bathing and eating cause work was that much, from six coming off at 6:30. When I come sometime I don’t feel like bathe I just lay in bed. Youth expressed concerns about having to work or commute late at night and working outside of the home at remote sites. One of the employers with whom we spoke suggested that perhaps their site was better suited for older youth given that they would need to live on site away from friends and family.28 It 27 Standard controls plus salary, number of hours worked, job sector, and benefits. 28 This is not something the evaluation team explored in-depth. However, it may be an item that could be included on a survey of possible employers in the future. Monthly Salary Male Female 0–20,000 22% 9% 20,001–30,000 9% 9% 30,001–50,000 34% 39% 50,001–70,000 28% 30% 70,000+ 6% 13% n 119 32 Benefit Insurance 64% Bonus 55% Overtime 54% Meals 49% Vacation 44% Risk Allowance 17% Pension 14% Housing 13% Table 6. Job Benefits Table 5: Monthly Salary, by Gender Evaluation of the Guyana SKYE Program 33 should be noted that there were a few cases during which youth described challenging work environments, but that they could also see the next step in their career. For example, one male from Region 4 explained that he has remained in his current position due to the opportunities for promotion. Unemployment/Job-Seeking Given the low employment rate, it is important to understand job-seeking behavior and experiences. Fifty-eight percent of respondents reported looking for a job at the time of the survey. Corresponding with the lower rates of employment, females were 70 percent more likely than males to have been searching (p=0.04). It is true that employment is not the sole intent of the program (SKYE has more than one intended outcome). However, of the 236 unemployed youth, only forty-six (19 percent) were attending school at the time of the survey, and eight (3 percent) owned their own business. Figure 12 provides an overview of productive engagement across employment, education, and entrepreneurship. Figure 13 provides population-based estimates for the entire cohort of trained SKYE youth. Figure 13. Productive engagement When asked about their experiences looking for a job, many youth stated that they were unable to secure employment primarily due to an overall lack of job opportunities. Less frequently, youth stated that their lack of education had prevented them from securing a job. This aligns with information collected from the Guyana Chamber of Commerce, which stated that the greatest needs their members have is for highly skilled workers. As such, many of the youth did not qualify for the jobs that were available. Figure 14. Population estimates (95% confidence) 0% 10% 20% 30% 40% 50% 60% None Employed School Business Employed and School Employed and Business School and Business 0% 10% 20% 30% 40% 50% 60% MaleFemaleTotal MaleFemaleTotal MaleFemaleTotal Employed Attending School Own Business 34 Evaluation of the Guyana SKYE Program What remains unclear is whether or not the youth were only seeking employment in their desired field or if they were seeking employment generally. In some instances, youth talked about possible linkages to jobs that they were not interested in taking on. For example, in the following exchange with a male youth from Region 4, he reported not wanting the types of jobs he was being connected to through SKYE: Interviewer: Your coach didn’t help you identify a job? And SKYE didn’t help? Youth: No . . . Well they called me . . . but the type of job they giving to me I won’t do it Interviewer: So what were some of the types of jobs? Youth: They was getting like pump attendants in the night, like working at a gas stations and national hardware, I won’t go to them places and work. Youth with a prior work history but unemployed at the time of the survey reported being out of the labor force for an average of 2.5 months (median). One in ten had been out of work for a year or more. Of the youth not looking for work at the time of the study, the two most common reasons provided were currently working and currently studying (44 and 40 percent, respectively). Of the 130 youth reporting having received a job from SKYE, half (51 percent) were no longer employed at the same job at the time of the interview. A slight minority of the youth no longer employed at SKYE-provided employment (46 percent) were employed at the time of the study, indicating that they moved on to other jobs. Entrepreneurship 41 youth (10 percent) reported owning a business at some point in their lives, with 23 owning a business at the time of the survey (6 percent). There was no significant difference in entrepreneurship between males and females. Businesses were predominantly small-scale ventures, with the average business being in operation 2.8 months, earning 5,000–20,000 GYD a week. Businesses were largely informal, with the most common including cosmetology, vending, transportation, and mechanical services. Education Of the youth not attending school at the time of the survey, three in four expressed a desire to return to school. As demonstrated in Figure 14, educational aspirations ranged from completion of secondary school to tertiary education. Males tended to prioritize work training, whereas females expressed more desire for formal education (secondary, CXCs, and tertiary). Youth attending school at the time of the survey had, on average, higher educational aspirations. There was a statistically significant association between the highest educational level desired and school attendance as well as socio-economic status (attending school at the time of the survey and having a higher SES score were correlated with higher educational aspirations). Figure 15. Highest education level desired 0% 20% 40% 60% 80% Secondary Write CXCs Tech/Voc Tertiary Male Female Evaluation of the Guyana SKYE Program 35 In most regions, youth did not discuss the opportunities for continuing education. Region 10 was the exception. During interviews, youth from Region 10 listed continuing education for the reason why they are not currently seeking employment. Both youth and coaches and trainers from Region 10 said that youth had clear career paths, but often that path would warrant additional education. Coaches and trainers reported forging relationships with the local and regional technical institutes so that youth could pursue additional education. There was at least one case in which the Linden Technical Institute had reached capacity, but, through the advocacy work of a coach, a SKYE youth was accepted into his desired program. Other youth, while stating a desire to return to school, were unable to lay out a plan to move in that direction; or were not aware of the steps needed to attain additional academic credentials. Program Goal: Crime Reduction Through Economic Growth Youth were asked whether participation in the program led to an improved standard of living and improved ability to stay out of trouble. As presented in Figure 15, youth reported a high degree of agreement with both statements. Figure 16. Youth responses to survey questions on staying out of trouble and standard of living improvements Individual Livelihood Plans While an analysis of intended programmatic outcomes is important in an evaluation, the goals of youth should also be considered. Youth were asked whether they completed an Individual Livelihood Plan. Of the 333 that remembered the activity, more than half reported being successful (score of 4 or 5) in meeting their goals. While only three percent of respondents ranked their success as a 1, females were more than 10 times as likely as males to report failure on the plan. Question 1: Conclusions Targeting Across a broad range of metrics, SKYE beneficiaries exhibited signs of being at risk. Of particular note were that nearly one in five had a criminal history, nearly one in three dropped out of secondary school, and almost one in two did not pass a single CXC. That being said, the evaluation team found evidence of 0 2 4 6 8 10 12 D i s a g r e e A g r e e Trouble Living 36 Evaluation of the Guyana SKYE Program leakage, with tertiary students, youth outside the stated eligibility range, and clearly affluent and well￾connected youth having passed the eligibility screening process. It appears SKYE implementers also noticed this leakage, as is evidenced by the program issuing additional guidance to coaches and trainers regarding CXC requirements for youth. Targeting for programs working with at-risk youth is always challenging. This is particularly acute for SKYE, a program that is tasked with reducing criminal behavior through economic growth. This mandate presents an inherent targeting dilemma: should EDC target those youth that are most likely to engage in criminal behavior (and thus most likely to contribute to the programmatic goal of crime reduction), or should EDC target youth that are the most likely to be employed (therefore meeting the intermediate objective of increased employment)? Recognizing that being “at risk” is less a binary distinction than a multifaceted spectrum, it becomes necessary to identify what segment of the “at-risk” population presents the most appropriate program beneficiaries. Through three weeks in the field, it is clear to the evaluation team that SKYE, in its present construct, is not the most appropriate program for youth on the extreme risk end of the spectrum. Given its limited scope, particularly the lack of trained counselors, SKYE is an insufficiently robust program to meet the needs of youth engaged in gangs, dependent on drugs, suffering from mental illness, engaged in abusive relationships, etc. This is witnessed by the lack of counseling services, the amount of time necessary to truly address remedial literacy and numeracy skills, and the lack of social support services available to youth outside of the training and coaches. While the program could be reformulated to meet these needs, the additional resources necessary to provide the necessary suite of services to such youth would detract from the program’s ability to offer workforce development training to youth able to put the knowledge to practice without large-scale outlays in support. It seems clear, therefore, that the ideal beneficiary is one that is both at risk and employable. While SKYE should tighten eligibility screening, given the programmatic context, the evaluation team feels that, on balance, the program reached an appropriate population. Program Youth were very satisfied with the program, particularly what they learned in the training. Lessons were not only practical (most youth reported having applied concrete skills after the training), but the method of delivery was participatory and effective. As expressed by youth and triangulated by employers, SKYE participants were, on average, better equipped at key employment skills (interviewing, communication, leadership, etc.) than comparable youth without the training.29 Irrespective of employment, participation in the program was also associated with increased life skills: youth reported being more confident, more comfortable with the opposite sex, more hopeful, and better able to plan for the future. The vast majority of youth reported positive experiences with their coach, a role that commonly expanded beyond what was initially envisioned for the position (counseling, long-term life-coaching, etc.). Trainers were also well liked, though relationships tended to be shorter-term than with coaches. Outcomes Due to its reliance on the economy, SKYE cannot promise to provide youth with employment. The labor market is beyond the control of the program, as are the individual decisions that alumni make post-graduation. That being said, the ultimate barometer of programmatic success is the extent to which youth are able to find work (and ultimately abstain from criminal behavior). The program achieved mixed success in provision of employment: fewer than two in five youth were matched with a job in their preferred sector, while a little more than one in four were placed in a job in any sector. Short- 29 Unit of comparison for youth are their pre-training selves. Unit of comparison for employers are non-SKYE applicants/employees. Evaluation of the Guyana SKYE Program 37 term employment gains were noted, though it is understood that many youth will have to engage in prolonged job-seeking after program completion to gain employment (partially due to the fact that they have ‘risk factors’ like limited schooling, and partly due to a depressed labor market). Unfortunately, at the time of the survey, only two in five respondents were employed, slightly below the national average for youth employment. Coaches had difficulty placing youth in jobs; and youth often did not stay in the jobs they were placed in. While this statistic in and of itself is telling, more confounding is the fact that a substantial number of SKYE youth who had participated in the program had employment at some point in their lives prior to SKYE involvement than they did at the date of surveying. The data is limited in that it does not permit us to say with certainty why youth were not employed, nor did it provide details into the type, quality and duration of employment youth had prior to their participation in the SKYE program. As such, it is unclear as to whether or not the youth had been engaged in long term meaningful employment prior to their participation in SKYE, which would be troubling. Data do suggest, however, that youth had turned down some opportunities for employment upon completion of the program citing a desire for more meaningful substantive work. We may conclude, therefore, that further work needs to be done to help youth understand the advantages of accepting employment that may not immediately meet their goals, as it may be the case that SKYE raised their expectations such that they now turn down work that may be a good building block. Entrepreneurship was also fairly limited, with six percent of respondents owning a business at the time of the study. It is difficult to say whether this is a viable solution to the lack of current employment opportunities in Guyana. However, what is clear is that youth will need some sort of employment experience in order to run their own businesses. The evaluation team could not identify a linkage between youths’ employment experiences and their desire to start a business, nor the success of that business in relationship to employment experience. Question 2: How effectively has the project engaged stakeholders at the local and national level? In order to better understand the extent of SKYE engagement with stakeholders external to the project, the evaluation team broke up regional- and national-level stakeholders into seven main groups:  National government  Regional and local government  Private sector  Educational and vocational institutions  Legal system  Families and communities  Local and regional NGOs and CBOs The team recognizes that these groups are not mutually exclusive, but felt that the distinctions above were necessary to gauge the range and types of engagement between SKYE and these groups. As such, the Chamber of Commerce has been included with private sector stakeholders, and the Magistrate has been included among the legal system stakeholders. Interaction with representatives from each of these groups varied over time and geography as detailed below. The team coded engagement within qualitative analysis as including those instances where collaboration, coordination, and/or engagement was discussed, capturing individuals or groups working together. It encompasses active involvement between the program, participants, and/or EDC as reported by both youth and key informants. 38 Evaluation of the Guyana SKYE Program Question 2: Findings National Government National-level government stakeholders included those from a wide variety of ministries and agencies, including from the Ministry of Culture, Youth, and Sports; the Ministry of Labour, Human Services, and Social Security; the Ministry of Education; and others. Data indicated that key stakeholders from these ministries and other national agencies were generally aware of the SKYE program. Similarly, interview data demonstrate that individuals agreed with the need to work with at-risk youth and increase the employability of that population. They also reported having been involved in meetings at the program design stage. Due to that involvement, individuals largely bought into SKYE’s methodology and felt engaged with the project. That stated, the engagement of stakeholders was largely at the individual level rather than at the institutional level, with only a few members of each ministry or agency reporting awareness and interaction with SKYE. This led some newer individuals to be less aware, or not aware at all, of the SKYE project; and reported having a lack of documentation about the historic involvement or interaction between SKYE and their department. Engagement with the national government was also complicated by political dynamics among various stakeholders. Interview data from key informants demonstrated competition for resources and overlap in the roles and responsibilities of the different ministries. That stated, stakeholders from the national government reported high levels of satisfaction and interest in continuing and increasing their level of engagement with SKYE. Regional and Local Government At the regional and local levels, stakeholder engagement was less consistent and more heavily centered on individual networks and contacts than at the national level, as some regional government officials noted that specific coaches and trainers had involved their agency in discussions. That said, interviewees noted that this was primarily done on an as-needed basis, drawing from personal connections. A number of key stakeholders at the regional and local levels reported never having heard of SKYE, including from central offices and ministries (such as from offices of the Regional Chairman, educational officers, regional social workers, and Village Councils). As one regional respondent stated: Well we haven’t had too much of interaction with the SKYE program because it seems, and I want to be very blatant. It seems as though, you yourself seem to be targeting groups of people. You know, oblivious of what the administration thinks and so on. This was particularly true of Regions 6 and 9, though nearly all respondents from the regional and local governments in all regions expressed high levels of willingness to improve engagement and contribute to SKYE’s objectives. Region 4, due to its interaction with national-level government stakeholders, was more coordinated than others; and Region 10’s successful engagement with regional and local-level individuals was largely due to initiatives by the trainers and coaches in that region. As with national-level government agencies, most engagement existed at the individual level rather than the institutional level and was, therefore, dependent on a lack of turnover within the relevant ministry or agency. Many officials who reported a lack of interaction with SKYE identified ways in which engagement would have benefited youth outcomes or the likelihood of sustainability, including helping to match youth with the SKYE program, or linking graduates to job opportunities within the public sector. “The Ministry of Youth, many times, they had involved in their meetings….I know partnership meetings would happen. Used to happen on a monthly basis. Sometimes more often than that. And, we would be updated overall on what’s happening on the project. I know they are continuing to work with the government of Guyana overall.” -National Level Stakeholder Evaluation of the Guyana SKYE Program 39 Private Sector SKYE’s work engaging the private sector has been mixed and largely contingent on individual coaches and trainers reaching out to individual employers, HR departments, and their own personal contacts. While SKYE headquarters has been responsible for some element of private sector engagement, many key informants noted that the interaction has been limited to a few companies and institutions. These stakeholders also noted that they were unclear as to how coordination between SKYE headquarters and coaches should occur. As such, coaches and youth reported that the alignment of jobs with youth needs depends largely on individual networks and contacts. Despite the presence of an EDC staff member dedicated to private sector engagement, coaches reported having varying degrees of success, highlighting during interviews the need for further support in this area. Similarly, coaches and employers described varying degrees of interaction with employers, such as when seeking out job opportunities, or when meeting with employers to check in on youths’ progress in their place of employment. A number of employers and agencies noted that they would be interested in hiring more SKYE youth, but did not know who to contact beyond one coach or individual. Others noted that many organizations and employers were not aware of the SKYE program or what the program teaches and that the lack of awareness prohibited further alignment of opportunities. Furthermore, while the Georgetown Chamber of Commerce and Industry (GCCI) was involved in discussions with SKYE officials, there was mixed feedback regarding its success. The job board was reported to be a good idea that lacked adequate implementation, and some members were unclear as to how to post jobs, screen participants, or contact youth they were interested in interviewing. One respondent mentioned that a recent GCCI survey asked whether their members were satisfied with the job bank, and only 5 percent responded affirmatively. While some members of the GCCI believe the job board to be a success, others believe it has lacked stability, feedback, updating, and sustained coordination. Regional Chambers of Commerce, meanwhile, reported a lack of interaction more broadly, with many respondents never having heard of SKYE. Employers reported varying degrees of success with engagement following the initial job placement, with some employers successfully hiring and promoting SKYE participants, and others having tried placing numerous graduates, only to have them all underperform or leave due to dissatisfaction with the job. One private sector representative stated that: The participants from the SKYE project are very mannerly . . . you know, they are very polite . . . is not like sometimes . . . you know sometimes somebody new coming in for an interview and would walk into the interview with a jeans and dress you know . . . What I’ve noticed is that persons from the SKYE project and stuff like that, when they come in for an interview they are very professional, I don’t know if that is something you guys discuss, with work ethics and stuff like that but they will be very professional when they come. Despite the mixed experiences, most employers the evaluation team spoke with said they were interested in further engagement with SKYE and would be willing to work with the SKYE team to enhance youth employment outcomes. “What they did was they decided to work with certain organizations like the Chamber of Commerce, the Private Sector Commission, and some other bodies, and they picked a few large companies and I know they created a private sector body. Umm and they put all their eggs in that basket and basically outsourced or hoped that they would’ve contacted their other private sector folks…The program has not resonated with the private sector.” - Employer and former member of the Georgetown Chamber of Commerce 40 Evaluation of the Guyana SKYE Program Educational and Vocational Institutions As with the private sector, engagement with technical, vocational, and educational institutes was inconsistent across regions and coaches. Regions 4 and 10 again demonstrated higher levels of interaction with these organizations than did their counterparts in Regions 6 and 9, both due to the prevalence of available institutions and the coaches’ and trainers’ approach to engagement. Coordination between educational institutions and SKYE largely depended on the extent and type of interaction coaches had with individuals at the training organizations. As such, this coordination was done differently by different people. There was no SKYE program approach, understood by institutions and coaches, to be the ideal link between these two entities. As such, some coaches worked with vocational institutions to enroll unaccepted students in SKYE, while other coaches would reach out to institutions to try to access recent graduates of vocational programs and recruit them into the program. While other regions experienced difficulties in linking SKYE graduates to further technical training opportunities, Region 10 stakeholders expressed a good deal of satisfaction and engagement at this level, further detailed in the textbox above. More broadly, youth expressed a desire to further their education, particularly with respect to obtaining additional CXCs. Despite this desire, many youth were unaware how to access these types of opportunities or what the next steps would be in doing so. Legal System At the national level, stakeholders involved in the legal system were both well aware of the SKYE program and had bought into its approach. National-level stakeholders, including probation officers, magistrates, representatives from Child Services, and SKYE, reported high levels of satisfaction and coordination with one another, asserting that the program has helped decrease the number of youth in prison. While most stakeholders stated that they believed diversion and alternative sentencing were fundamental to curbing youth violence, the majority of respondents did not believe it was being implemented consistently or broadly enough throughout the country. This sentiment was echoed at the regional levels, where a number of magistrates, probation officers, police commissioners, and others were unaware of the SKYE program and the option to divert into SKYE. At the regional and local levels, the team received mixed reactions to the SKYE approach to reducing youth criminal activity, with some stating that SKYE’s approach was successful, and others believing it to be an ineffective or incomplete approach. One magistrate in Region 6 reported having sent 4–5 youth to SKYE with none of them returning to the legal system for offenses, while another in the same region said that the various youth she sent to SKYE were back in trouble within the year. As with private sector engagement, stakeholders in the legal system noted individual connections with specific coaches. Coaches who interacted with specific probation officers and magistrates often were more likely to access youth and divert them from the formal legal system. As one magistrate put it: When you don’t see somebody you tend to forget that they are around as I said the probation officers refers the juveniles to SKYE from time to time but sometimes because of the fact that I don’t see them in court, I forget…that the program is existing. Paralleling the findings surrounding broader youth issues, different stakeholders acknowledged internal competition for resources with respect to legal issues and youth, as different Ministries and staff face “I normally communicate via the coaches, whenever their session their session nearly over. Or, when it’s close to graduation, they would bring me a list of their trainees that would be perspective trainees. Bring them, fill out the form. I think they would even help them, assist them with filling out the form. Then I would check them, review them and then I would call them for the interview. I haven’t at any time rejected any SKYE trainee because they all meet our qualification level.” -Technical Institute Representative Evaluation of the Guyana SKYE Program 41 overlapping roles regarding youth justice and reintegration. This was particularly evident in discussions surrounding NOC, with a number of different individuals assigned to help re-integrate youth leaving the facility. Despite the lack of agreement regarding roles, as well as the coach changes inside and outside of the NOC, the SKYE coaches were viewed by many as the most effective resource for those youth needing assistance reintegrating into society. Families and Communities Echoing data gathered regarding the engagement of other stakeholders, interview data indicate that there was a high level of variability with respect to interaction between SKYE and participants’ families and communities. This engagement depended primarily on the coach and the coach’s individual view on the coach’s role vis-à-vis families and communities. While some coaches felt that meeting with families was a fundamental part of their role, others saw it as unnecessary or outside their scope of work. The trainers and coaches who actively engaged families tended to be the most active and outwardly passionate individuals in the program, and youth appreciated their actions and would actively seek out pairings with those individuals. Trainers, coaches, and youth from all regions reported pressures on youth from family members to leave the training to earn money; or a lack of positive encouragement at home to continue working in their place of employment. As one youth participant who had to leave SKYE put it, “When I joined . . . it was going good until I had some family problems at home. I had to quit and get a job to help support them, the family.” A trainer put it similarly, stating: The homes, the influences in their environment, I think that plays a great part in determining whether this child will be prepared or not. Also, well, sometimes parents come in and they wouldn’t want to take the blame, so they will sit there and say like it’s a result of influence of their peers in these depressed communities. . . . Those children from those communities you find that they come in not so prepared because I think they’re focused maybe in a totally different direction. Let’s say the parents aren’t working. At a young age these children have to play a role in bringing in necessary bread and butter for the family and so school, education, literacy, those things suffer. They—and—and the primary thing for the home, money getting, finances, that plays—that takes priority instead of the child’s education. Many parents reported not knowing about SKYE other than from what their child told them. Furthermore, youth, trainers, and coaches reported disappointing parental turnout at SKYE graduations for all regions except for Region 10. Given these factors, as well as the lack of adequate counseling and social services available for many of the targeted youth, most stakeholders believed that increased engagement with families and friends of youth participants would lead to enhanced outcomes for the SKYE program. Local and Regional NGOs and CBOs As with other types of engagement, interaction with local and regional NGOs and CBOs varied depending on the region and individuals involved. Many SKYE stakeholders, including trainers, coaches, government representatives, partners, and educators, have well-established links and connections within the youth ambit. Most were able to identify a number of organizations doing complementary or relevant youth work, many of which knew of SKYE or had some link to SKYE due to those individual connections. Church organizations were mentioned by a wide variety of individuals as an opportunity for “When you interact with parents, you’re making sure that what you are doing will tie in with what is happening at home. So, you’re helping the child, and the parent is also helping the child at the same time. And I think that helps a lot.” -Trainer, Region 4 42 Evaluation of the Guyana SKYE Program further engagement, as were graduate placements with NGOs and CBOs in a volunteer or internship capacity. Question 2: Conclusions With the exception of national government engagement, the successes in engagement have largely been due to individual SKYE coaches and their approaches. Centralized coordination and engagement from SKYE seemed to be strong at project start, but has since focused inwardly rather than externally to the stakeholders analyzed here. The coaches deserve praise for the variety of roles they are playing in the SKYE project, but, because of the variety of stakeholders and stakeholder needs, as well as the inconsistencies regarding how interactions occurred between these stakeholders, there have been a number of missed opportunities for effective involvement with these actors. Many opportunities exist in engaging these stakeholders further, and in enhancing the probability of success for youth as well as the sustainability of outcomes. This is particularly true given the high levels of enthusiasm and willingness by individuals in each of these spheres to increase engagement. National Government SKYE has, for the most part, involved the national government effectively in the design and implementation of the program. National-level stakeholders were content with SKYE’s management of the political dynamic and satisfied with the level of coordination established and maintained between USAID, EDC, and their various ministries. That stated, internal competition between government agencies for resources combined with continued levels of turnover make it difficult to institutionalize awareness and buy-in of the SKYE approach and programming. With a few key exceptions, the SKYE team has been able to engage a number of ministries and individuals throughout the implementation process, but maintaining and institutionalizing that involvement throughout the second half of the project will prove crucial for the sustainability and effectiveness of programming. Regional and Local Government Differing levels of engagement led to inconsistent implementation of SKYE at the regional level. While some key individuals, particularly coaches, worked to engage local and regional government officials, the lack of consistent interaction by SKYE headquarters resulted in some missed opportunities, particularly as implementation often happens at the local levels. As with engagement at the national level, SKYE will need to remedy the lack of institutionalized engagement between SKYE and regional and local governments in the coming years. These stakeholders are willing and eager to engage with SKYE to identify opportunities for collaboration. Increased involvement with the regional chairs, education officers, social welfare officers, probation officers, and others will increase the ability of SKYE to achieve their stated outcomes. Private Sector Despite some success stories linking youth with jobs and engaging the private sector, a lack of sufficient central coordination with the private sector has led to lesser outcomes with respect to both job placement and job retention. Success in private sector engagement has, to date, largely happened at the project design stage and subsequently has been based on individual coaches and their networks. Furthermore, because the alignment of jobs and youth is primarily placed on the coach, opportunities for youth are largely dependent on the employers and networks that their coach is connected to, rather than on the best fit between job and individual. For example, if a coach has a connection to a large business, the company will contact that coach with openings. The coach will then find a match with one of the SKYE graduates assigned to him or her, but does not circulate the opening or employer contact with other coaches. Evaluation of the Guyana SKYE Program 43 Although a number of medium- and large-scale employers are looking to hire additional staff and are open to hiring SKYE youth, a lack of central coordination and strategy from SKYE headquarters has stymied progress. This is evidenced by the GCC job board and was echoed by a number of employers. Private-sector experiences with youth graduates in jobs have been mixed, but there is an openness by employers to work more collaboratively with SKYE to enhance the chances of success. Given the centrality of private sector engagement to SKYE’s theory of change, increased coordination with this sphere is one of the most important areas to focus on throughout the remainder of the program. Educational and Vocational Institutions: SKYE coaches have had mixed success in engaging technical, vocational, and other educational institutions. As most SKYE graduates demonstrated interest in furthering their education, and because employers often look for employees with technical skills, the link here is not only important, but a relatively easy one to make. Successes in Region 10 point to ways in which the links can be established between different SKYE stakeholders to encourage the development of both hard and soft skills for youth participants. Additional work needs to be done to clarify SKYE’s vision of that link, for example, whether SKYE is best suited to technical graduates or those prior to enrollment in these institutions, in order to set the stage for future work. As with engagement with other stakeholders, the links made thus far are not institutionalized and are therefore at risk of being unsustainable. This engagement should likely not come solely from individual coaches, but rather with some central SKYE coordination to ensure that all SKYE graduates have access to similar educational opportunities, regardless of their coach’s individual contacts or their geographic location. Similarly, while technical and vocational institutes are fundamental, there is inadequate engagement with other educational institutions to provide youth the opportunity to gain additional CXCs. Legal System Stakeholders recognized the legal element of SKYE’s work as one of the most important in stemming the tide of youth crime and violence. This was true both of the work with youth in the NOC as well as with the push toward using SKYE as an alternative sentencing opportunity. Youth, legal workers, and implementers all saw some of the highest value or return on this element of programming. At the national level, legal stakeholders are well aware of SKYE and buy into its approach. This type of engagement and awareness is not consistent at the regional and local levels, however, where the project has a great opportunity for impact. Without stakeholders’ consistent understanding of the diversion opportunities and the different roles within the juvenile justice sphere, SKYE risks missing out on opportunities to institutionalize this type of alternative sentencing structure. Families and Communities Coaches varied widely with respect to their engagement with families and communities, though the most popular coaches among youth were also those who were active with families and more broadly with the general community. The family context that most SKYE youth come from cannot be ignored when discussing outcomes and engagement, and the fact that there is no standard SKYE approach to engagement at this level was viewed by many as problematic. Most key stakeholders felt that increased engagement at this level would lead to enhanced outcomes, particularly with respect to reducing drop￾out rates and increasing job retention. Although many recognized that filling the gap in family and social services might be outside the scope of SKYE, leaving this unfilled causes difficulty in achieving SKYE’s desired outcomes. Local and Regional NGOs and CBOs The youth-centered community in Guyana is relatively small and well connected, with individual networks and links being utilized by SKYE regularly. That stated, coordination and engagement was again 44 Evaluation of the Guyana SKYE Program inconsistent across the regions. Involvement with NGOs and CBOs was largely dependent on contacts made by individual coaches rather than guided by a central strategic approach. Engagement with these organizations included utilizing them as sources for recruitment or places to engage graduates, demonstrating the variety of opportunities engagement at this level can create. Question 3: What are the prospects for the sustainability of the results produced/approaches being utilized by this project? USAID defines sustainability as “the capacity of a host country entity to achieve long-term success and stability and to serve its clients and consumers without interruption and without reducing the quality of services after external assistance ends.”30 It is with this definition in mind that the evaluation team assessed SKYE’s progress toward achieving long-term success in terms of sustainable outcomes, as well as the development of host-country capacity to carry out programming to maintain positive outcomes. To do so, the evaluation team examined four major factors as indicators of sustainability: stakeholder buy-in, financial sustainability, policy developments/institutionalization of programming, and the development of regional and local capacity. Question 3: Findings Stakeholder Buy-in The notion of engagement is complex and plays an important role in the sustainability of programming. If key stakeholders do not have confidence in a program, are not satisfied with implementation, or do not believe that it can yield positive outcomes, research shows that the likelihood of future investment of time and resources is limited. Additionally, for any program to be sustained, there need to be both sufficient supply (program services) and demand (youth who want/need to participate). To assess overall buy-in, the evaluation team used two proxies: satisfaction with SKYE and a willingness to participate in or engage with SKYE. Given the important role that youth play in the recruitment of other participants, their satisfaction with the program is important in sustaining SKYE. As a part of the survey, the evaluation team asked youth to score the extent to which they agreed/disagreed with a number of statements reflecting dimensions of programmatic success. The first analytical grouping was designed to solicit the extent to which participants were satisfied with their experience in the program and would be willing to recommend the experience to their peers. As evidenced in Figure 16, not only did youth report large-scale satisfaction with SKYE, they were even more supportive of recommending the program to peers.31 30 From USAID’s Global Health website. 31 Of the five items in this section, the statement “I would recommend SKYE to youth like myself” solicited the highest average score, the least dispersion, most maximum scores, and the highest minimum score. Evaluation of the Guyana SKYE Program 45 Figure 17. Youth responses to survey questions on satisfaction, recommending program to peers Qualitative findings from youth interviews correspond with quantitative findings: an overwhelming majority of youth were pleased with their experience with SKYE. Frequently during interviews, youth asked the evaluation team members if additional trainings would be coming online. A number of youth also stated that they would be interested in serving as mentors or trainers themselves. However, according to coaches, trainers, and youth, currently, there is not a formal way for youth to remain actively involved. Many of the youth who have completed the program reported that they have kept in touch with their coaches or trainers, and in one case a former participant now serves as a training assistant in Region 10. The majority of key informants with whom the evaluation team spoke had positive things to say about SKYE. Many stated that SKYE was the first program of its kind regarding development of youths’ soft skills and encouragement of pro-social behaviors. Specifically, many pointed to the important role that the coaches played. In fact, some representatives from Ministries said that SKYE was filling a gap by providing services that their Ministry could not currently provide in terms of coaching and mentorship. As described in the section on engagement, overall the level of interest in SKYE is quite high. This was true at the national, regional, and local levels. For those stakeholders that did not have an awareness of SKYE at the local level, most expressed interest in finding ways to collaborate with SKYE and saw the program as an overall benefit to the youth as well as to the community. At the national level, key informant interviews revealed that numerous stakeholders had played a role in the design, rollout, and uptake of SKYE. For example, representatives from the juvenile justice system and judiciary have provided guidance on the needs of youth and are actively working with SKYE to establish it as an alternative to prison or the NOC. Representatives from the Ministry of Education and The Ministry of Labour, Human Services, and Social Security also said they were looking for ways for organizations or individuals under their purview may interface with SKYE. 0 2 4 6 8 10 12 D i s a g r e e A g r e e Recommend Satisfied “I think about being a trainer, but I never really followed it up because I never really know where to apply or how to go about doing it.” -Male Youth, Region 10 46 Evaluation of the Guyana SKYE Program While many stakeholders expressed an interest in and engaged in the design of SKYE, at the regional and local level there seemed to be a lower level of engagement. This was evidenced when the evaluation team sought to set up interviews with individuals and organizations that had been identified as having a link to youth within their respective regions. Stakeholders reluctantly agreed to participate in interviews but reported that they were unaware of the SKYE program. Many asked the evaluation team for an overview. Once they had learned about the program, the immediate response from most was that the thought SKYE sounded like a good program and they wanted to know how they might engage. Financial Sustainability Currently, SKYE is 100 percent donor-dependent, with all funding provided through USAID’s Caribbean Basin Security Initiative. There are some signs, however, that elements of the program may persist beyond the current period of performance. During interviews with several representatives from key Ministries, respondents stated that they would be willing to commit financial resources to the continuation of SKYE. When asked why they weren’t currently funding the effort, they responded that it takes time to secure funding. Now that there is some evidence of programmatic success, they would have the leverage they would need to request future support. Whether or not financial resources were forthcoming from the government, nine out of ten youth surveyed reported that they would be willing to pay for the SKYE training if it were required.32 Policy Developments and Institutionalization of Programming Another crucial component of activity and outcome sustainability is the development of policies and structures to provide for the institutionalization of activities. Interviews with key informants pointed to three major initiatives with the most potential in integrating elements of SKYE into existing systems: the implementation of SKYE at the NOC, the use of SKYE as an alternative sentencing mechanism, and the integration of SKYE training into secondary schools’ curricula. Each of these initiatives is in a different stage of being operationalized, has different prospects for institutionalization, and would require different levels of resources to ensure sustainability. SKYE at the NOC The SKYE program has been deeply integrated within the institutional structure of the NOC. As noted previously, two full-time SKYE staff live at the NOC and provide training and coaching support to residents. SKYE has given thought on institutionalizing the training within the NOC and has trained MCYS staff on the SKYE approach. It is reasonable to expect NOC staff to continue implementation of the Pre-Work Ready and Work Ready Plus trainings after the program concludes. Staff currently offer a wide variety of courses, and there seems to be support among administration for the program. SKYE as a Diversion or Alternative Sentence Key Informants (representatives from the Ministry of Labour, Ministry of Human Services and Social Security, as well as magistrates and probation officers) described formalized and informal referral 32 Amounts were not discussed by the evaluation team. The question was intended to solicit willingness to pay as a general construct. “I remember we had a girl on SKYE, on the SKYE project who the magistrate – she wasn’t a girl who could live at home because she was sexually abused at home. So we had to – our thing is looking at the alternative care part. She could benefit (from the program), so she was referred to the (SKYE) program. But, the care part – the alternative care part, the alternative care part…is strictly-strictly the responsibility of the child care agency.” -Representative from the Ministry of Labor, Human Services and Social Security Evaluation of the Guyana SKYE Program 47 systems by which SKYE is being used as an alternate sentencing option to internment at NOC or prison. Evidence of SKYE’s influence in the legal system was found in 3 out of the 4 regions visited (4, 6, and 10). The program was active in socializing alternative sentencing as a judicial option among magistrates and probation officers, including having arranged meetings, meeting privately with representatives of the judicial system, and making personal appearances at court cases to advocate for usage of SKYE as a judicial option. At least one high-level representative from the Ministry of Labour, Human Services, and Social Security reported being actively involved with SKYE in drafting legislation that would allow SKYE to be used as a diversion program. As a part of the program, youth would go before a community board to determine if enrollment in SKYE was a possibility. This youth would then participate in SKYE rather than being formally entered into the judicial system. While this initiative has not yet passed, using SKYE as an alternate sentencing option has been institutionalized by several probation offices as the regional level. A representative from Probation Services in Region 4 described the process: When the system would kick in is when the magistrate makes contact with the SKYE coach and/or the probation officer. So, the magistrate looks at the case. Once it’s not a serious case and they think that this person can benefit from this SKYE program then they ask whether or not he SKYE coaches are going to accept this youth on the program. And the SKYE coaches based on the information given to them will make that determination. Once that determination is made then the magistrate prepares the court order and embodies the court order where it would stipulate the conditions of being placed on probation and being committed to the SKYE program. This being said, the evaluation team found limited utilization of SKYE outside of Georgetown. One regional magistrate communicated only having referred 5 youth to SKYE in their tenure, while a regional probation officer noted that they do not see SKYE as an appropriate sentencing option. SKYE and Educational Institutions Interview data indicated that there are two ways in which SKYE is being integrated into educational institutions. First, some coaches, trainers, youth, and representatives from Technical Institutes reported that informal connections had been made between SKYE and several Technical Institutes. However, while these training institutes have been identified by SKYE participants, coaches and trainers, representatives from these organizations in Regions 4, 6, and 10 reported that they were not fully aware of and engaged with SKYE. During a visit to one of the training institutes, the evaluation team learned that the faculty were not aware of SKYE and that they had a new school director who had not yet learned of the program, despite having former SKYE participants enrolled. A representative from another training institute said that they had been invited to a SKYE graduation and became familiar with the program and some of the youth through that route. Second, SKYE is in the beginning stages of integrating elements of the program into the secondary school system in Region 9. Program staff have conducted one training session with eight teachers from four communities. While there were issues in the manner in which the training was organized (respondents indicated that it was marketed as a math training, that they learned of the session a matter of days beforehand, and that the pace of the training was very strenuous), the sessions seem to have been well received. Teachers were instructed to provide the training after the completion of CXCs “I was invited to a meeting in Georgetown…That was two years ago. I can’t recall exactly what it was about, I think the nature. I think they were about to form or commence the activity. So it was more about the plans of the criminal aspects basically of the (indistinct). The youths were more or less involved in it and they were looking at that context basically (indistinct)…We weren’t really involved in it thereafter.” -Representative from a Technical Institute 48 Evaluation of the Guyana SKYE Program while youth were waiting for their results. The teachers interviewed indicated that it would be possible to achieve attendance, particularly if parents were informed of the activity. On the basis of limited interviews, teachers seemed eager to teach the SKYE training and had spoken to the Head Teacher about the plan. In contrast to the outreach done at the grassroots level, key regional informants indicated that there was insufficient coordination. First, Regional Education staff were waiting on approval from the central ministry before providing direction to teachers. Additionally, they argued that additional resources would be needed in order to fully integrate the SKYE program into the school system: I still feel to have it properly into the curriculum would be, would mean more classrooms. More classrooms would mean more teachers, would mean more subjects also. As it is now if we just come and say ministry say you have to put this on timetable, it’s gonna be tough. If we had got the population that we were supposed to, it would mean more, well even the infrastructure in itself is not suitable for the kind of numbers we were expecting this academic year which would have been way over three hundred and because of no space in the dormitory, no space in the classrooms . . . It should also be noted that during an interview with a high-level key informant from the National Ministry of Education, they expressed a lack of awareness of the intentions to implement the SKYE program into the secondary schools in Region 9. While the respondent did believe it was a good idea, they also expressed concern that the Ministry had not been consulted. However, according to EDC and USAID, the Ministry was engaged in this conversation and knew about the intention to integrate into the secondary schools in Region 9. Development of Regional and Local Capacity On the whole, very few key informants discussed the development of Regional and Local Capacity for SKYE. When it was mentioned, it was done so in the context of increasing resources within their respective organizations, as was the case in Region 9. While key informants on the whole had a basic understanding of the SKYE objectives and activities, they did not seem to be versed in the specifics. However, the evaluation team did not include a question in its protocol regarding the types of capacity building that SKYE was undertaking at the local and regional levels. Question 3: Conclusions Data from the youth surveys and interviews with key informants indicated that those individuals and organizations that were familiar with SKYE were, on the whole, satisfied with the program. When individuals identified programmatic weaknesses, for the most part, they did not express that the program should be discontinued, but rather modified to strengthen it. Data indicate that key stakeholders were not only satisfied with the program, but keen to engage with it; even those who were not aware of SKYE demonstrated an interest in getting involved. However, data also indicated that, while individuals on the national level seemed to play an active role in SKYE, individuals at the regional and local levels were less likely to be directly involved and often knew less about the program than their counterparts at the national level. Given these findings, one may conclude that the buy-in of SKYE is fairly high, though it atrophied in relation to distance from the capital. This engagement is a promising indicator of program sustainability. Adding to this promise is the evidence that efforts are being made to incorporate SKYE into institutional structures that already exist, including the juvenile justice system, schools, and the NOC. Substantial efforts have been made to integrate SKYE into each of these systems with varying degrees of success. Evaluation of the Guyana SKYE Program 49 While SKYE has enjoyed the most integration within the NOC, there are two concerns from a sustainability perspective. First, SKYE is not synonymous with the training curriculum. The method of training delivery is a key component of programmatic success, with facilitation done in a participatory, youth￾friendly manner. As with all integration activities, if continuation of SKYE is limited to materials and does not encompass pedagogical techniques, beneficiary satisfaction and outcomes cannot be expected to mirror those of the EDC-led effort. Second, while NOC staff have been trained, sustainability will not be achieved without high-level buy-in from MCYS officials, including building program implementation into the annual budget. With regard to the legal system, SKYE has done much to advocate for alternative sentencing. While the judicial architecture for alternative sentencing has been on the books for a long time, uptake on the part of the judiciary has been limited. SKYE has helped reinvigorate the usage of alternative sentencing, though uptake has varied widely across the regions. It is likely that momentum for alternative sentencing will continue, particularly with other donors continuing to work on the issue. However, unless there is an institutional arrangement with a training institute to continue providing SKYE courses, the ability of magistrates to use SKYE as an option will disappear with the end of the program’s period of performance. However, additional work needs to be undertaken to engage additional partners and formalize relationships and activities. This was especially noticeable when speaking with representatives from technical institutes in Regions 4, 6, and 10. SKYE fills a gap that is not currently offered by the technical institutes such as mentorship and the development of life skills that will help aid youth in securing a job. As descried earlier in this report, some employers reported a certain level of dissatisfaction with the SKYE youth they had employed, citing a lack of hard skills and low levels of literacy and/or numeracy. Therefore, it may be concluded that a combination training of soft skills from SKYE trainers, mentorship from their coaches, and the training of hard skills that institutions such as Board of Industrial Training (BIT), Linden Technical Institute (LTI) and Kuru Kuru offer could prove to be a winning combination. While further work needs to take place, the incorporation of aspects of SKYE into Guyanese institutions shows promise for the sustainability of programming. At this point, additional research would need to be undertaken to gain a better understanding of the financial sustainability of the program as well as capacity building. Preliminary data shows that there is some level of commitment by Guyana’s Government to continue aspects of SKYE. However, it is unclear as to whether or not the country has the financial resources to do so. Furthermore, how the private sector may become engaged and provide financial support also warrants exploration, especially as it is the private sector who could also benefit from the outcomes of the SKYE program. While some GCC stakeholders and other private sector representatives stated willingness to engage with SKYE to determine which components they might be able to absorb, additional data needs to be gathered to more fully understand the level of capacity building that is taking place by SKYE aside from the formal training of youth and information relationships built between SKYE partners and staff and potential employers and technical institutes. “If I could make a recommendation…would say, yes, continue the program. And, secondly, build on it. And the third thing is to establish your tentacles so to speak. There’s more organizations and agencies in (name of community) that would allow, or groups, to be involved. I mean, we can have an attachment aspect as part of our recommendation. After so many weeks of training, find particular areas to which you…any carpentry or joinery…any business sector allow them to go there for a week, two weeks, three weeks. You know, just to have a hands on of what they would have done. That is going to enable them more to be convinced that they should come to the (name of school). -School Principal, Technical Institute 50 Evaluation of the Guyana SKYE Program Overall, the SKYE program has begun to take steps toward ensuring sustainability of programming and outcomes. However, further work needs to be undertaken now that the program has reached its midpoint and an endpoint is in view. RECOMMENDATIONS Evaluation questions 4 and 5 focus on recommendations that the evaluation team has, based on the findings and conclusions. The questions state the following:  Are there any recommendations to enhance project results within the project’s current period of performance and budget?  Are there any recommendations to expand or enhance project results should additional resources become available? In this section, the evaluation team provides a set of recommendations to enhance SKYE programming and build on initial successes. The first half focuses on those activities that EDC and their partners may undertake under the current contract scope. The second half focuses on those that might exceed the current project parameters. During the initial outbrief that occurred prior to the evaluation team departing Guyana following fieldwork, both the COR and the Contracting Officer (CO) encouraged the team to create recommendations that were as specific as possible and that provided actionable steps that could be taken to actualize the work. The evaluation team constructed the recommendations for program design, stakeholder engagement, and sustainability. It is with this in mind that the team puts forward the recommendations below. Recommendations within the Period of Performance and Budget: Recruitment Actively engage alumni in the recruitment of new participants. Current recruitment practices leverage coaches, trainers, and community outreach. However, given the extent of success of community-based recruitment efforts, it is recommended that SKYE more actively engage program alumni and the community to solicit applicants. While it would be helpful for program staff to more actively socialize the program through non-governmental organizations, churches, and other community fora, the program may explore institutionalizing a system that rewards and/or recognizes beneficiaries and/or community members for bringing eligible applicants into the program. On one end of the resource spectrum, this could include providing rewards to youth that bring the most eligible applicants into the program (e.g., paper certificates, trophies, etc.). On the other end, the program may consider paying a recruitment fee for successful referrals. Both of these should be done in concert with an effort to improve levels of program awareness in communities and target populations. In addition to boosting program ranks, this approach would reduce the workload on both trainers and coaches, freeing them up to provide more support to their portfolio of youth. Program Implementation Provide gender and LGBT sensitization training to trainers and coaches. The evaluation team found evidence of gender stereotyping from both program staff (coaches/trainers) and youth beneficiaries. Most saliently, opinions about what constitute male versus female jobs were Evaluation of the Guyana SKYE Program 51 shared openly by representatives of the program. It is recommended that frontline staff be provided formal training so as to be sensitized to gender norms and to facilitate group discussions that do not discourage either sex from aspiring to jobs they desire. Customize, where possible, materials to speak to the specific context in which the program is operating. To the extent the program chooses to continue operating in predominantly Amerindian communities, it is the recommendation of the evaluation team that efforts be made to contextualize the intervention to the country’s interior. Specific improvements include reviewing training materials to ensure that examples are context appropriate (e.g., not using hospitality services as an example) and addressing cultural practices surrounding concepts like borrowing money for businesses. Improve coordination between coaches internal and external to the NOC. For beneficiaries receiving services at NOC, it is recommended that the program improve coordination between coaches resident at the NOC and coaches that will be supporting youth in their home communities. In particular, the following should be considered:  Enhance coordination between NOC coaches and new coaches assigned to NOC participants. This would include ensuring communication via phone or in person.  New coaches should meet youth at the NOC and travel with them back to their communities (this is particularly important if youth do not have any family). The new coaches should also coordinate with officers working on behalf of the government to ensure understanding by participants of who they can go to with different needs. To the extent possible, and with participant approval, coaches could also work with the participants’ families or communities to aid in the reintegration process. Monitoring and Evaluation Bolster the program monitoring and evaluation system. Of particular importance would be a tracer study to gauge what happens to graduates after completing the program. Current indicators capture the number of youth employed after the program. This metric, however, does not capture youth that quit their jobs (overestimate of potential impact) or those youth that found jobs after a prolonged search (underestimate of potential impact). Developing a short telephone questionnaire and calling sampled youth on a predetermined timeline could shed light on long￾term experiences of youth and sustainability of programmatic outcomes. In bolstering the M&E system, the team recommends:  Strengthening the participant intake form to better determine and track participants who meet specific at-risk criteria. Also add to this a section about their previous employment experience. This will help understand why youth may have had employment in the past but are not employed at higher rates at program completion. This will also provide EDC with an opportunity to align new employment opportunities with previous work experience.  Incorporate indicators in the project’s activity M&E plan that track participants’ job retention, criminal activity, education, and other outcomes. This would entail having a survey tool or other mechanism by which SKYE would follow up with participants 3, 6, or 12 months after training completion. This tool might also incorporate questions to track continued interaction with coaches, participants, and/or other actors.  Streamline and enhance monitoring trackers. These trackers should include, at a minimum, columns dedicated to tracking participant drop-out rates, trainings attended by region, information from the intake form, updated participant contact information, and participant outcomes. 52 Evaluation of the Guyana SKYE Program  Specifically track NOC participants following program completion to determine the extent to which their behaviors and choices changed over time and any recidivism rates. This data may prove useful in demonstrating impact to enhance the likelihood of sustainability by government or other actors. Improve performance monitoring of coaches and trainers and provide a feedback loop. While the majority of coaches were found to be effective and hard-working advocates for their youth, the team uncovered evidence of significant variations in effort. It should come as no surprise that, for a program implemented across a wide geographic swath of the country, there were differences in staff engagement. However, the lack of active participation of frontline staff was found to have adverse effects on the success of youth beneficiaries. In order to better monitor performance of coaches and trainers, it is recommended that SKYE headquarters reach out directly to participants (either on a random basis, or through an anonymous call-in number) to solicit beneficiary feedback on frontline staff. This process should be done in a transparent and consistent way, and communicated to trainers and coaches. During meetings with and between these staff, SKYE should share best practices and lessons learned as a result of this feedback. For a coach with regularly positive feedback, for example, project staff might examine what factors or activities are well received and share that insight with other coaches. Stakeholder Engagement Further engage the private sector and increase the level of coordination between coaches and the SKYE staff responsible for private sector engagement. Data indicated that the private sector is interested in continuing to engage with SKYE. However, the data also indicated that several employers faced challenges with youth they employed. There are several ways that SKYE could consider engaging the private sector that could benefit the program in terms of sustainability both of programming and of outcomes:  Work with local employers to identify their greatest needs and also where SKYE youth (depending on the target) could fit into their organization.  Provide employers with a training on what youth are taught during SKYE and educate them on SKYE youth. Help employers understand how the youth were selected for the program. Just as external stakeholders could provide guest trainings during SKYE, SKYE may consider doing a guest training with a local employer to teach them how to work with at-risk youth.  Work with employers who have work training programs to incorporate some of the SKYE materials in the trainings they offer.  Expand private sector engagement in SKYE implementation through the use of marketing materials, job fairs for SKYE participants, internships, and/or presence of private sector representatives during the training sessions. Increased involvement during trainings will improve private sector awareness of graduates and the skill sets SKYE develops, build bonds between potential employers and SKYE, and improve the relevance and utility of training sessions for youth. Similarly, recognition of employers who have successfully supported the SKYE program and SKYE youth could incentivize continued private sector engagement.  Improve the GCC job board process and functions. Dedicate a staff member to update job listings and graduate profiles on a regular basis. This staff member, potentially a former SKYE participant, would be responsible for creating profiles for graduates (ideally with tags) and aid in matching candidates with job opportunities. Once improved, expand the job board to additional regions where SKYE is implemented. This should be done through discussions with the regional GCC. To further enhance the engagement with the private sector, SKYE should define and distinguish roles and responsibilities for coaches and SKYE staff, including details regarding: Evaluation of the Guyana SKYE Program 53  Who reaches out initially to make employer connections;  How connections with employers and private sector actors should be made;  How employment opportunities are shared among members of the SKYE team, including coaches; and  Who is responsible for checking in with existing contacts to gauge future vacancies. If feasible, USAID and SKYE should consider including a staff member to increase private sector engagement. This individual, potentially a former SKYE youth, would work to purposively identify employers with entry-level needs. This could be done through a mapping exercise or some other mechanism. Reach out directly to these employers with face-to-face meetings from SKYE headquarters. Work with current SKYE employers and GCC members to better integrate employers into the job board. Host a focus group or one-on-one discussions to better understand private sector needs from the board and priorities to improve the utility of the site both for employers and youth graduates. If possible, host these discussions every 6–12 months to ensure continued engagement and improvements. Increase coordination, communication, and engagement with school welfare officers, counselors, probation officers, MCYS reintegration officers, and social service workers. This would include creating and maintaining a list of counseling and other support services that are available for youth, potentially by undertaking a mapping exercise of existing services. This list, including contact information, should then be distributed to SKYE trainers and coaches so that these individuals may refer youth to appropriate services where applicable. Further opportunities for engagement include holding focus groups or meetings with representatives from these areas to increase awareness regarding the availability of SKYE as an option for youth, and to better understand how these programs and institutions can support one another. Sustainability Start a SKYE alumni group. Data indicate that SKYE youth have a strong desire to remain engaged with the program. By creating an alumni group, SKYE will be able to keep youth engaged and interacting with youth who have been taught a similar skill set. Through this network, SKYE may consider having youth volunteer as peer mentors or as training assistants. Both of these scenarios would allow youth to teach what they have learned and help junior SKYE participants put their new skills into practice. By keeping youth engaged with SKYE once they have completed the program, the evaluation team believes that SKYE will help sustain the positive outcomes that the program accomplished. Additionally, to the extent there are resources to support such an activity, it may be worthwhile to organize graduates for volunteer activities. Leadership of the organization could be shared between program staff and an elected body of youth leaders. If resources are constrained, it is recommended that program staff build relationships with NGOs and work in collaboration to engage youth in productive activities in the absence of employment or formal education. Concrete examples of how to do this would include inviting local NGOs into the training to present on their programs, participate in question and answer sessions, and recruit youth on-site. If training venues are not appropriate places for such an activity, a “volunteer fair” could be organized outside training hours. While SKYE headquarters would need to initiate this group and preliminary activities, the alumni group can and should transform into a youth-led initiative. Focus yearly targets on ways to integrate SKYE into existing institutions. Rather than spread the organization too thin, identify one specific sector SKYE wishes to build over the course of a year or six months. For example, if the objective is to integrate SKYE training into secondary 54 Evaluation of the Guyana SKYE Program schools in Region 9 as well as other regions, have school integration be a targeted objective to meet over the year. Set up an action committee to sort through the steps toward realizing this goal. This concentrated effort will increase the likelihood that specific SKYE activities will be sustained. Recommendations that Exceed the Period of Performance and/or Budget: Targeting Hold formal discussions to determine which of two objectives, reduction in crime versus increased labor market participation, should take precedence with regard to a targeting strategy. If portfolio-level strategic plans necessitate a focus on crime reduction, the evaluation team recommends that SKYE be more diligent in screening “overachieving” youth. The simplest method to accomplish this task would be to include a series of filtering questions on the applicant intake forms. For example, if program leadership decides that youth enrolled in tertiary education should not be eligible for inclusion, any youth reporting that they are attending university should be deemed ineligible. Realizing that applicants may have an incentive to misrepresent information, SKYE could either (1) inform applicants that any falsified statements may present reason for removal of youth from a program, or (2) validate information by having the recruiting agent (coach/trainer) make contact with the youth’s guardian or other informed individual. The latter approach could have the added benefit of involving the family at an early stage of the program, thus better explaining the program and allowing key stakeholders a chance to ask questions.33 If the primary intent of the program is to increase economic and educational benefits, the evaluation team recommends that the program bolster the applicant intake form to screen for youth in need of targeted programming in excess of what SKYE can provide. There exist a number of tools for determining whether or not respondents are at risk; however, given the deep contextuality of ‘riskiness’ it will be necessary to adapt (or even develop) a custom instrument for use by program staff. Suggested themes for inclusion in the instrument include learning disabilities, significant substance abuse, and acute anti-social tendencies. If this recommendation is acted upon, the team recommends that youth screened as being too “at risk” are provided a direct link to existing services that can provide support and that some flexibility be built into the selection mechanism. As discussed in preceding sections, risk is a complex construct not easily amenable to quantification. While there should exist some triggers for which youth do not make a programmatic ‘good fit,’ front line staff should have some ability to advocate on behalf of certain individuals. If a decision is made to prioritize employment outcomes, the gender quota currently in place should be revisited. It is the assessment of the evaluation team that the current system is appropriate for a program designed to reduce criminal behavior, but is not appropriate for one that intends to improve educational and employment outcomes. Counseling Provide coaches and trainers with additional training in counseling and identify ways in which the program may integrate with counseling support systems that are already in place. While it may not be in the formal job description of either coaches or trainers, most frontline staff engage in de facto counseling without proper preparation. In order to equip these staff with the 33 If at-risk youth are chosen as the programmatic focus, the team recommends that SKYE update the program to make it more relevant to the target population. Specific items for this are interspersed throughout this section of the report. Evaluation of the Guyana SKYE Program 55 resources necessary to answer questions brought forth by their clients, the evaluation team recommends three programmatic changes, presented in order of increasing resource demands:  SKYE should create a roster of all counseling support systems available to program applicants, regardless of inclusion in the program. This resource should include contact information and be kept current at least quarterly. In the event a youth requires support outside the ability of SKYE staff to provide, they should be referred to the list.  A full-time, on-call counseling resource should be made available to all coaches and trainers for support in difficult situations. This could be done in the form of a staff member that sits at SKYE headquarters or through a contracted service that is engaged on an as-needed basis. Having a professional resource would do much to help reassure frontline staff of correct courses of action in times of pressing need.  All coaches and trainers should be provided with formal training on counseling as part of their mandatory training. While these individuals will not become qualified social workers, the program needs to recognize the role that these staff have been playing and better equip them to handle difficult situations. For any future recruitment of trainers and coaches, the team should continue to seek out those with existing training or qualifications in social work and counselling.  Develop programmatic guidelines or a SKYE protocol for staff dealing with youth who have demonstrated high levels of psycho-social issues. This could include the resources and contacts mentioned above and could also include partnership with other organizations working in this field across SKYE regions. Provide coaches and trainers with a formal support system to help relieve stresses of working with at-risk youth. Coaching, training, and mentoring are emotionally draining jobs. In multiple instances, the evaluation team heard pleas that “coaches need coaches too.” The evaluation team recommends the creation of a formal system to provide support to frontline staff. SKYE could facilitate a community of practice and leverage the experiences of existing staff to help one another cope and recharge from the demands of their job. Alternately, this service could come in the form of a counseling “hotline” as discussed in the recommendation above, whereby the professional social worker could support staff with their personal needs. Monitoring and Evaluation Create a formalized feedback loop from SKYE employers. Given that employability and employment of youth are central to the theory of change under which SKYE is operating, it is crucial that further linkages to employers be built. However, it is also crucial to understand the needs SKYE youth are meeting and those skills they are lacking as described by current SKYE employers. Therefore, SI recommends implementing a simple survey that can be regularly administered to employers affiliated with the program. To increase response rates and ensure the accuracy of the data collected, the evaluation team recommends administering an SMS survey to those individuals who are most closely working with the SKYE youth at the various employers. The survey should consist of only 4 to 5 questions. Sample questions include the following:  On a scale of 1 to 5 (with 5 being very positive, and 1 very negative), what rating would you give the soft skills demonstrated by the SKYE youth with whom you are working?  On a scale of 1 to 5, what rating would you give the hard skills demonstrated by the SKYE youth with whom you are working?  Indicate which of the following behaviors youth are struggling with most: o Timeliness o Attitude o Confidence 56 Evaluation of the Guyana SKYE Program o Ability to communicate effectively o Numeracy skills o Literacy skills o Technical abilities o General aptitude  What are the major strengths you have identified in SKYE youth? o Timeliness o Attitude o Confidence o Ability to communicate effectively o Numeracy skills o Literacy skills o Technical abilities o General aptitude The team also recommends follow up the survey with brief phone interviews with employers who are willing to participate. Determine a SKYE approach to interaction with families and communities and make that approach and associated roles clear to trainers and coaches. At a minimum, this approach should include some level of familial engagement at the beginning of SKYE and identification of ways to enhance family and community participation in graduation ceremonies. Involve coaches and those who have been successful in this type of interaction, including those from Region 10, to identify best practices in effectively involving these stakeholders. Sustainability Capitalize on high levels of interest in SKYE displayed by youth and stakeholders at the national, regional and local levels by providing them guidance on specific roles that they can play to increase sustainability. One way to do this is to have representatives from government agencies, CBOs, and the private sector conduct one session from the trainings. To ensure that this is an effective process, the evaluation team recommends that SKYE take the following steps: 1. Identify one or two modules that may be of interest to external stakeholders. For example, how to interact with managers and supervisors may be of interest to individuals from the private sector who may employ youth, while a module on the educational steps needed to reach their goals may be of interest to individuals from the Technical Institutes or the Ministry of Education. 2. Reach out to the various stakeholders to identify an individual to give the guest training. Bring those stakeholders together for a training of trainers on the specific topics that they have committed to teach. 3. Pair the guest trainers with a youth representative from the training they will be attending. Have this youth describe what they like about the training and have them brainstorm with the stakeholder to come up with ideas of what activities to undertake in class. 4. Ask the coaches to attend the guest trainings. 5. Have the youth introduce the guest trainer to the group, and then have the trainer explain what role they play in the community. 6. Ask for feedback from the guest trainers on ways that the training may be improved or if there are additional ways in which their organization or they individually would like to get involved. There are several benefits to engaging stakeholders in this way. First, it activates the stakeholders’ interest while keeping their commitment to participation manageable. This also gives the stakeholders an Evaluation of the Guyana SKYE Program 57 opportunity to connect with the youth and get to know them on a face-to-face basis. While their presence at graduations is meaningful, this interaction is more likely to encourage on-going relationships between the youth and the stakeholders as well as SKYE and the stakeholders. Second, this approach not only empowers youth by having them teach someone who is in a more senior position than themselves, but it also provides youth with the social capital of the beginning stages of a professional network to which many at risk do not have access. Third, this will introduce the private sector and potential employers as well as representatives from Technical Institutions to the details of the training curriculum. This will help build their capacity and give them an opportunity to give SKYE feedback on how their curriculum aligns with their needs with respect to employees, or how it may overlap or align with curricula in the Technical Institutes. By building capacity at these other institutions, while SKYE may not continue as a program, specific activities may be able to be undertaken by these other organizations. Fourth, by training the guest trainers together by topic, rather than by the organization they represent, it gives them the opportunity to work together and coalesce around SKYE. In other words, it is an opportunity to set politics aside and focus on the pragmatic side of SKYE. Finally, this approach also gives the coaches and trainers an opportunity to interface with individuals from the community in a new way, with the possibility of building new relationships that could benefit SKYE in the future. 58 Evaluation of the Guyana SKYE Program ANNEXES Evaluation of the Guyana SKYE Program 59 Annex I: Contract SOW SECTION C- STATEMENT OF WORK Performance Evaluation of USAID Skills and Knowledge for Youth Employment (SKYE) C.1 PURPOSE OF THE EVALUATION This will be a mid-term program evaluation focused on: examining project approaches, assessing the quality of implementation and the effectiveness of approaches used; assessing information on performance and providing recommendations on how the project can be improved. In particular, the program will be analyzed to determine a) whether program activities are creating anticipated impact b) the validity and completeness of the program logic c) changes that may be recommended. The evaluation should provide pertinent information, statistics, and judgments that assist USAID, EDC and other partners to better understand the initial results and contributions of the project, and help re-focus and strengthen it where necessary. C.2 SUMMARY OF PROJECT TO BE EVALUATED Activity name: USAID Skills and Knowledge for Youth Employment (SKYE) Period of Performance: August 8, 2011 to August 7, 2015 Associate Award No.: AID-504-LA-11-00001 Recipient: Education Development Center (EDC) Total Estimated Cost: $5,000,000 Agreement Officer’s Representative (AOR): Chloe Noble C.3 BACKGROUND “Youth violence has become an important development challenge worldwide. Declining social and economic development conditions have increased the risk that youth will become violent offenders and victims of violence. Increased youth involvement in violence has contributed to popular perceptions of growing insecurity in the Caribbean. At the same time, the experiences of and involvement in crime and violence among Caribbean youth have become linked to other developmental issues, including high levels of youth unemployment, poor educational opportunities, and feelings of voicelessness and exclusion from national and regional governance processes.”34 In addressing youth-centered development issues, the Government of Guyana has established a number of training programs generally targeted to serve youth through various ministries and donor funded projects. Some civil society organizations also provide ‘informal’ education and skills training opportunities for children, youth and adults. 34 Caribbean Human Development Report. 2012. Human Development and the Shift to Better Citizen Security. 60 Evaluation of the Guyana SKYE Program The Skills and Knowledge for Youth Employment (SKYE) project is focused on reducing youth violence by strengthening economic participation and civic engagement of youth, with an emphasis on high-risk youth, aged 15 to 24. Project activities are designed to expand employment, education and skill-building opportunities for at-risk youth; re-integrate youth offenders into society and the workplace and; strengthen the enabling environment for youth development. From its inception, the project has been implemented in geographic regions 4 (Demerara￾Mahaica), 6 (East Berbice- Corentyne), 9 (Upper Takatu- Upper Essequibo) and 10 (Upper Demerara- Berbice) in Guyana, as represented in the map below. Project activities have focused on supporting youth aged 15 to 24 who are: i) school dropouts; ii) youth who have completed formal education, but did not achieve minimum academic requirements to gain a certificate to access higher education or employment; and iii) those with education, skills and behaviors that are insufficient for integration into the workforce, including being in the juvenile justice system. Youth are also selected based on demonstrated need. The project has anticipated that more young men than young women will be engaged in the juvenile justice-focused aspects of the program. Beginning late 2013 the project expanded to Region 3 (Essequibo Islands-West Demerara) and Region 5 (Mahaica- Berbice). SKYE has focused on providing services to more males than females. This focus is grounded in data showing young men are more likely to be perpetrators of crime than young women, and that young men between 15-35 years commit 80 per cent of the crime in the Latin American and the Caribbean Region35. The Project also recognizes that best practices in gender programming have highlighted the importance of working with both men and women to affect attitudes about gender, and improve social issues that affect the well-being of women, men, youth, and children in Guyana. While men generally experience violence outside their domestic environment, women are more likely to experience violence and abuse inside the home, including sexual assaults. 35 January 2010. Report to the CARICOM Commission on Youth Development. Evaluation of the Guyana SKYE Program 61 Annex II: Data Collection Instruments Youth Participant Survey ID: _______ / _______ / _______ Region: (2 / 3 / 4 / 6 / 9 / 10) (Enumerator) (Date) (Respondent) Municipality: _____________________________________ Informed Consent My name is _______________ I am a researcher conducting an evaluation of the SKYE Program for USAID. The purpose of this survey is to learn about your experience in the SKYE Program, your life, your community. As a participant of the program your responses could be helpful to improve the SKYE program for other youth like yourself. This interview should take about half an hour. Your participation in this study is completely voluntary. You have the right to stop the interview at any time or say ‘no response’ or ‘don’t know’ to any questions that you don’t want to answer. Only the research team will know your name and responses and the information provided will not be linked to you. Do you have any questions? ● Pause, wait for response. Do you agree to be interviewed? ● NO - Thank participant for his or her time. ● YES - Thank you! Please mark in the box that you understand the information and agree to take part in the survey. 1. IDENTIFICATION First Name(s): _____________________________________________________________ Last Name(s): _____________________________________________________________ Sex: M / F Age: ______________________________ Ethnicity: Indo-Guyanese / Afro-Guyanese / Mixed / Amerindian / Other / NR What city/town do you live in: _______________________________________________________ Region: _____ 62 Evaluation of the Guyana SKYE Program ID: _______ / _______ / _______ Region: (2 / 3 / 4 / 6 / 9 / 10) (Enumerator) (Date) (Respondent) Municipality: _____________________________________ 1. DEMOGRAPHICS Q # Question Response Code Skip 1.1 What is your marital status? Single Married Living home but unmarried Separated Divorced Widowed Don’t Know No Response 1 2 3 4 5 6 -99 -88 1.2 Do you have any children? Yes No Don’t Know No Response 1 0 -99 -88 ➔ 1.5 ➔ 1.5 ➔ 1.5 1.3 How many children do you have? Children Don’t Know No Response _____ -99 -88 1.4 At what age did you have your first child? Age Don’t Know No Response _____ -99 -88 1.5 Are you the head of your household? Yes No Don’t Know No Response 1 0 -99 -88 ➔ 1.7 1.6 Who is the head of your household? (OTHER) _____________________________ Parent Partner Non-parent relative Other Don’t Know No Response 1 2 3 4 -99 -88 1.7 Not including yourself, how many people do you support financially? Number Don’t Know No Response ____ -99 -88 1.8 Do you receive financial support from anyone? Yes No Don’t Know No Response 1 0 -99 -88 ➔ 1.10 ➔ 1.10 ➔ 1.10 Evaluation of the Guyana SKYE Program 63 1.9 Who do you receive financial support from? (Multiple Responses) (OTHER) _____________________________ Parent(s) Partner Sibling Non-parent relative Other Don’t Know No Response 1 2 3 4 5 -99 -88 1.10 Do you receive money transfers? Yes No Don’t Know No Response 1 0 -99 -88 ➔ 1.12 ➔ 1.12 ➔ 1.12 1.11 How much, on average, do you receive in money transfers per month (Guyana Dollars) 1-20,000 20,001-40,000 40,001-60,000 60,001+ Don’t Know No Response 1 2 3 4 -99 -88 1.12 Including yourself, how many people live in your home? People Don’t Know No Response _____ -99 -88 1.13 Which of the following people live in your home most of the time? Brother Sister Mother Father Partner Yes/No Yes/No Yes/No Yes/No Yes/No 1.14 How many rooms does your home have? (including Living, Kitchen, Dining, Bedroom, & Bathroom) Rooms Don’t Know No Response _____ -99 -88 1.15 Which of these items do you have in your home? Electricity Running water in your home Indoor Toilet/Bathroom Television Computer Cell Refrigerator Yes / No / DK / NR Yes / No / DK / NR Yes / No / DK / NR Yes / No / DK / NR Yes / No / DK / NR Yes / No / DK / NR Yes / No / DK / NR Yes / No / DK / NR 2. EDUCATION Q # Question Response Code Skip 2.1 What is the highest level of school you have attended? None Nursery Primary (1-6) Secondary (7-13) Technical/Vocational College/University Don’t Know 0 1 2 3 4 5 -99 ➔ 2.3 ➔ 2.3 ➔ 2.3 64 Evaluation of the Guyana SKYE Program No Response -88 ➔ 2.3 2.2 What is the highest (standard/form/year) you have completed in ___? (see response above) Answer Don’t Know No Response _____ -99 -88 2.3 How many CXCs have you passed? CXCs Don’t Know No Response _____ -99 -88 2.4 Are you currently attending school? Yes No Don’t Know No Response 1 0 -99 -88 ➔ 2.8 2.5 Why are you not attending school? (OTHER) _________________________ Finished Studies Working/economic necessity Can’t Afford It Pregnancy/care own children Care for other children Health Problems Family pressure Other Don’t Know No Response 1 2 3 4 5 6 7 8 -99 -88 2.6 How many years ago did you stop attending school? (<1 year = 0) Year Don’t Know No Response -99 -88 2.7 Do you plan on returning to school? Yes No Don’t Know No Response 1 0 -99 -88 ➔ 2.9 ➔ 2.9 ➔ 2.9 2.8 What is the highest level of education you plan on completing? Secondary (7-13) Write CXCs Technical/Vocational College/University Don’t Know No Response 1 3 4 5 -99 -88 2.9 Besides SKYE, have you ever taken courses/trainings outside of school before? Yes No Don’t Know No Response 1 0 -99 -88 2.10 Would you be willing to pay for the SKYE program if you had to? Yes No Don’t Know No Response 1 0 -99 -88 Evaluation of the Guyana SKYE Program 65 3. Training Roster Which of these SKYE courses did you sign up for? Did you finish the _______ training? (SKIP NEXT IF YES) Why did you not complete the training? 1 - Found work 2 - Pregnancy/own children 3 - Care of other children 4 - Started/returned school 5 - Health Problems 6 - Family pressure 7 - Time conflict 8 – Travel commitment 9 - Other 10 - Don’t Know 11 - No Response How long ago did you finish the training? 1 – Weeks 2 - Months 3 – Years Where did you go for the training? How did you travel to the training? 1 – Public transportation 2 – Training￾provided transportation (w/ chaperone) 3 – Walking 4 – Own transportation 5 - Other Did you ever feel unsafe travelling to or from the training? How long did it take you to get to the training (one￾way)? 1 - Min. 2 - Hours Pre-Work Ready Y / N Y / N 1 / 2 / 3 / 4 / 5 / 6 7 / 8 / 9 / 10 / 11 1 / 2 / 3 ______________ 1 / 2 / 3 / 4 / 5 Y / N 1 / 2 Work Readiness Y / N Y / N 1 / 2 / 3 / 4 / 5 / 6 7 / 8 / 9 / 10 / 11 1 / 2 / 3 ______________ 1 / 2 / 3 / 4 / 5 Y / N 1 / 2 Work Ready Now Y / N Y / N 1 / 2 / 3 / 4 / 5 / 6 7 / 8 / 9 / 10 / 11 1 / 2 / 3 ______________ 1 / 2 / 3 / 4 / 5 Y / N 1 / 2 Work Ready Plus Y / N Y / N 1 / 2 / 3 / 4 / 5 / 6 7 / 8 / 9 / 10 / 11 1 / 2 / 3 ______________ 1 / 2 / 3 / 4 / 5 Y / N 1 / 2 Entrepre. Y / N Y / N 1 / 2 / 3 / 4 / 5 / 6 7 / 8 / 9 / 10 / 11 1 / 2 / 3 ______________ 1 / 2 / 3 / 4 / 5 Y / N 1 / 2 66 Evaluation of the Guyana SKYE Program 4. ACCESSIBILITY Q# Question Response Code Skip 4.1 How did you hear of the SKYE program? (Multiple Responses) (OTHER) _________________________ Coach New Opportunity Corps Alternative sentencing/Court Training Center Participant Friend (non-participant) Relative Advertisements Other (state) Don’t Know No Response 1 2 3 4 5 6 7 8 9 -99 -88 4.2 Rank the following four things, where 4 is your most favorite and 1 is your least favorite. Coach Trainer Training (materials/activities) Peers 1 / 2 / 3 / 4 1 / 2 / 3 / 4 1 / 2 / 3 / 4 1 / 2 / 3 / 4 4.3 Do you know other youth that were interested in SKYE but did not sign up? Yes No Don’t Know No Response 1 0 -99 -88  5.1  5.1  5.1 4.4 What are the most important reasons why youth do not apply for SKYE? (Multiple Responses) (OTHER) _________________________ Working School Program Content Link to employment Facilitators/Coach Peers No payment/stipend Length Required Certificate Time conflict Other (state) Don’t Know No Response 1 2 4 5 6 7 8 9 10 11 12 13 -99 -88 Evaluation of the Guyana SKYE Program 67 5. Relevance On a scale of 1-5, how important is the following skill in getting a job in your community? 1 (not important) -> 5 (very important) Did SKYE provide you with training in _______? On a scale of 1-5, how confident are you in this skill? 1 (not confident) -> 5 (very confident) Have you used this skill since the training? Doing a job search 1 / 2 / 3 / 4 / 5 Y / N 1 / 2 / 3 / 4 / 5 Y / N Interviewing for a job 1 / 2 / 3 / 4 / 5 Y / N 1 / 2 / 3 / 4 / 5 Y / N Writing a professional CV 1 / 2 / 3 / 4 / 5 Y / N 1 / 2 / 3 / 4 / 5 Y / N Interacting with managers/supervisors 1 / 2 / 3 / 4 / 5 Y / N 1 / 2 / 3 / 4 / 5 Y / N Career planning 1 / 2 / 3 / 4 / 5 Y / N 1 / 2 / 3 / 4 / 5 Y / N Financial management 1 / 2 / 3 / 4 / 5 Y / N 1 / 2 / 3 / 4 / 5 Y / N Starting your own business Y / N 1 / 2 / 3 / 4 / 5 Y / N 6. OUTCOMES Q # Question Response Code Skip 6.1 Have you ever had a paid job? Yes No Don’t Know No Response 1 0 -99 -88 ➔ 6.11 6.2 Did you ever have a paid job before training? Yes No Don’t Know No Response 1 0 -99 -88 6.3 Do you currently have a paid job? Yes No Don’t Know No Response 1 0 -99 -88 ➔ 6.10 ➔ 6.10 ➔ 6.10 68 Evaluation of the Guyana SKYE Program 6.4 How many paid jobs do you currently have? Jobs Don’t Know No Response ________ -99 -88 6.5 How many hours, on average, do you work a week? Hours Don’t Know No Response ________ -99 -88 6.6 What industry/sector is your primary job in? (OTHER) _____________________________ Services Mining Farming Manufacturing Logging Other Don’t Know No Response 1 2 3 4 5 6 -99 -88 6.7 How much do you get paid (GYD) per month in your primary job? 0-20,000 20,001-30,000 30,001-50,000 50,001-70,000 70,000+ Don’t Know No Response 1 2 3 4 5 -99 -88 6.8 Which of the following benefits do you receive in your primary job? (Multiple Responses) (OTHER) _________________________ Private Pension Paid Vacation Overtime Private health insurance Bonus/Gratuity National Insurance (NIS) Housing Meals Risk allowance Other Yes / No / DK / NR Yes / No / DK / NR Yes / No / DK / NR Yes / No / DK / NR Yes / No / DK / NR Yes / No / DK / NR Yes / No / DK / NR Yes / No / DK / NR Yes / No / DK / NR Yes / No / DK / NR 6.9 On a scale of 1-5 (where 1 is not satisfied and 5 is very satisfied), how satisfied are you in your primary job? 1 2 3 4 5 Don’t Know No Response 1 2 3 4 5 -99 -88 ➔ 6.11 ➔ 6.11 ➔ 6.11 ➔ 6.11 ➔ 6.11 ➔ 6.11 ➔ 6.11 6.10 How long have you been unemployed? (LENGTH) ____________________________ Days Weeks Months Years Don’t Know No Response 1 2 3 4 -88 -99 6.11 Are you looking for work? Yes No Don’t Know No Response 1 0 -99 -88 ➔ 6.13 ➔ 6.13 ➔ 6.13 Evaluation of the Guyana SKYE Program 69 6.12 Why are you not looking for work? (Multiple Responses) (OTHER) _____________________________ Studying Working Health problems Don’t believe will find work Lack qualifications No time Pregnant/care own children Care other children Don’t need to work Other Don’t Know No Response 1 2 3 4 5 6 7 8 9 10 -99 -88 6.13 Have you ever owned a business? Yes No Don’t Know No Response 1 0 -99 -88 ➔ 6.19 ➔ 6.19 ➔ 6.19 6.14 Did you ever own a business before the training? Yes No Don’t Know No Response 1 0 -99 -88 6.15 Do you have an active business now? Yes No Don’t Know No Response 1 0 -99 -88 ➔ 6.19 ➔ 6.19 ➔ 6.19 6.16 How long has the business been active? (LENGTH) ____________________________ Days Weeks Months Years Don’t Know No Response 1 2 3 4 -99 -88 6.17 What type of business do you have? Nature of Business Don’t Know No Response ________ -99 -88 6.18 What is the average weekly income of this business? 0-5,000 5,001-10,000 10,001-20,000 20,001-30,000 30,001+ Don’t Know No Response 1 2 3 4 5 -99 -88 6.19 Did you complete an Individual Livelihood Plan (ILP) / Individual Employability Plan (IEP) with the SKYE project? (EXPLAIN) Yes No Don’t Know No Response 1 0 -99 -88 ➔ 6.21 ➔ 6.21 ➔ 6.21 6.20 On a scale of 1-5 (where 1 is not successful & 5 is very successful), how successful have you been in meeting the goals of your ILP? 1 / 2 / 3 / 4 / 5 / DK / NR 70 Evaluation of the Guyana SKYE Program 6.21 What job sector are you most interested in? (OTHER) ______________________________ Services Mining Agriculture Manufacturing Construction Logging Public service Other Don’t Know No Response 1 2 3 4 5 6 7 8 -99 -88 6.22 Has SKYE helped you learn more about job opportunities in this sector (see response 6.21)? Yes No Don’t Know No Response 1 0 -99 -88 6.23 Has SKYE helped you connect with other trainings in this sector (see response 6.21)? Yes No Don’t Know No Response 1 0 -99 -88 6.24 Has SKYE helped you connect with employers in this sector (see response 6.21)? Yes No Don’t Know No Response 1 0 -99 -88 6.25 Did SKYE get you a job in (see response 6.21)? Yes No Don’t Know No Response 1 0 -99 -88 ➔ 6.27 6.26 Did SKYE get you a job in any sector? Yes No Don’t Know No Response 1 0 -99 -88 ➔ 7.1 ➔ 7.1 ➔ 7.1 6.27 Are you still working in this job? Yes No Don’t Know No Response 1 0 -99 -88 Evaluation of the Guyana SKYE Program 71 I’m now going to ask you some sensitive questions. Remember, you can say “don’t know” or “no response” to any of these and nobody will know how you answered. 7. Legal 7.1 Have you ever been arrested? Yes No Don’t Know No Response 1 0 -99 -88 ➔ 8 7.2 Have you ever been sent to the NOC? Yes No Don’t Know No Response 1 0 -99 -88 7.3 Have you received alternative sentencing? Yes No Don’t Know No Response 1 0 -99 -88 7.4 Have you ever been in prison? Yes No Don’t Know No Response 1 0 -99 -8 8. Scoring I’m going to say five things. After each, I will ask you to mark on the line how strongly you agree or disagree with what I said. The line goes from ‘strongly disagree’ on the left to ‘strongly agree’ on the right. We will do a practice round to make sure you understand the rules of the game. 8.0: “I like to eat pepperpot” 72 Evaluation of the Guyana SKYE Program 8.1: “I am satisfied with the SKYE program” 8.2: “SKYE helped me feel more confident” 8.3: “SKYE helped me improve my standard of living” 8.4: “SKYE helped me stay out of trouble” 8.5: “I would recommend SKYE to other youth like me” Evaluation of the Guyana SKYE Program 73 INTERVIEW PROTOCOL: SKYE PROGRAM PARTICIPANTS Introduction: Thank you so much for taking the time to speak with us. My name is _________ and my colleague’s name is ______. The survey you took will help us greatly in our study, but we wanted to take this opportunity to give you more of a chance to talk about your life and your experiences with SKYE. As with the survey, everything you say will be confidential, and we won’t use your name in our report so please be as honest and open as you can. You also don’t have to answer any question you aren’t comfortable with. Do you have any questions before we get started? Participant Background: 1. To start, could you tell me a little bit about yourself when you first became involved with the SKYE project. What brought you to the SKYE project? (probe as necessary to find out where they are from, family situation, if they were employed, and to the extent possible any ‘at risk’ factors such as incarceration) 2. Now I’d like to know a little bit more about your current situation. Are you working, at school, looking for work, or something else? a. If they currently work: Can you tell me a little more about your job? Does it align with the type of job you would like to have? How did you choose it? 3. Do you think you or your situation has changed since working with SKYE, or is it the same? Why do you think that is? 4. Do you participate in any other similar programs? a. [If yes]: Can you tell me a little bit about them? SKYE Program Feedback: Now that you’ve told me a little bit more about you, I’d like to talk to you more about the SKYE project. 5. When were you involved in the project? What training(s) did you take? 6. How did you first learn about SKYE? 74 Evaluation of the Guyana SKYE Program 7. Why did you decide to participate? 8. Do you know people that decided not to participate? (If yes, probe for why they didn’t) 9. What role does your coach play in the program? In your life? 10. How often do you work with him/her? 11. How easy or difficult is it for you to apply what you learned in SKYE? Why is that? 12. What do you (or did you) think was the best part the SKYE program? (e.g. what worked well) 13. What are some of the challenges or things that aren’t working well with the SKYE program? 14. Is there anything you were hoping to learn or gain through SKYE that was not covered? 15. If USAID were going to do another program working with youth like you, what would be your recommendations? What would change and what would stay the same? Evaluation of the Guyana SKYE Program 75 Key Informant Interview Guide Thank you so much for meeting with us today. I am _______ and this is my colleague _______. We work for an independent research firm hired to undertake an evaluation of USAID’s SKYE project, which works with at-risk youth. As part of this evaluation, we are speaking with a number of key stakeholders as well as surveying and interviewing youth participants. All of this will feed into a report which will be used to inform current and future youth programming for USAID. We ask that you be as open and honest with us as possible. Your answers will be anonymous, and we will not identify you in any way through our final report. Do you have any questions before we proceed? Questions for All Respondents: 1. First I would like to learn a little more about you and your position. Can you tell me about your general roles and responsibilities? How does the work you do intersect with the needs of youth in Guyana? (Probe further if a regional ministry). 2. What do you believe are some of the greatest needs of Guyanese youth? (Probe for the biggest challenges youth face) 3. Could you tell me a little bit about your knowledge and awareness of the SKYE program? (EQ2) 4. How did you become involved in SKYE? Please describe the level and extent of your involvement in the program (EQ 2,3). 5. What do you see as some of the strengths of the SKYE program? What are its weaknesses? 6. To your knowledge, is the SKYE program coordinated with any other programs that are occurring in the area? a. (If they respond affirmatively): Please tell me about those programs. Do they complement, contradict, or duplicate one another? 7. Sustainability: Which elements of SKYE do you think would be feasible to incorporate into your own organization/work? a. To what extent do you plan on doing that? (probe for concrete examples and funding aspects) 8. Based on your knowledge of SKYE and the needs/opportunities for youth in Guyana, do you have any recommendations for SKYE that you think we should include in this evaluation? a. Probe for: Engagement among stakeholders, approach to achieving outcomes (content and format), modifications to encourage sustainability, improve project in final year, if more money/future programming? 76 Evaluation of the Guyana SKYE Program 9. Is there anything we haven’t covered that you think would be useful for us to know? Questions for Coaches: 1. Please tell me more about what it means to be a coach as a part of the SKYE program? a. What is your role with respect to the trainings? b. Outside of trainings? c. Development of a career path? d. Employability Plans: have you seen outcomes? Are they sustainable? 2. Have you seen any important outcomes for individuals who have participated in the program? Do you think those outcomes are sustainable? 3. How do you identify and recruit youth to participate in the program? (Probe for criteria for ‘at risk’) 4. With whom or what organizations are you interfacing in order to do your job? Questions for Implementing Organizations: 1. How long has your organization been working on SKYE? 2. Does your organization implement other youth programming? 3. Does your organization specialize in working with any particular grouping within youth? Questions for Representatives from the Juvenile Justice System: 1. Please describe a little bit about how the juvenile justice system works here in Guyana. Once a youth has been arrested, what is the course of action? 2. What are some of the options for alternative sentencing? Do you think alternative sentencing is effective? a. Once we have list of options: Which of the options do you see as most effective? b. Are there any plans to expand alternative sentencing? 3. What roles does SKYE play in the rehabilitation of youth? What about other programs? Questions for Government Officials 1. To what extent are you satisfied with the level of coordination between SKYE and your Ministry/Department/Bureau? Why? 2. To what extent does your ministry agree or disagree with the SKYE approach? Why? Evaluation of the Guyana SKYE Program 77 3. To what extent has the ministry engaged with SKYE graduates (as interns, staff, or in some other capacity)? a. Is this likely to change or remain the same in the future? Questions for Parents 1. What do you see as the biggest influencers of your child’s success in life? 2. To what extent were/are you satisfied with the level of interaction you had with SKYE? Why? 3. Is there anything you were hoping your child would leave the SKYE program with and didn’t? Questions for Private Sector 1. Which skillsets do you think youth need to succeed in the private sector? Which skills or knowledge do you think is most lacking? 2. Have you hired any SKYE graduates? a. [If yes]: What prompted you to hire them? Have you been satisfied with their performance? (Why or why not?) b. [If no]: Why not? 3. What do you see as the biggest opportunities to enhance youth employability? The biggest obstacles? 78 Evaluation of the Guyana SKYE Program FGD Protocol: Youth Participants Introduction My name is ____________________, and this is ______. Thank you all so much for being here and for speaking with us. As you know, we are here trying to better understand the SKYE program and provide some recommendations for programs which will best meet the needs of youth like you. We’ll talk today as a group so we can share ideas and ask that you be open and honest with us; and that you respect one another and listen openly as well. As with the survey, everything you share with us is completely confidential. We will not put your names in the reports and nobody (coaches, family, or SKYE staff) will know you spoke with us and what you said. In order to keep this discussion open, we also ask that anything shared today between this group is not discussed outside of the group. If there is anything you want to talk about individually rather than in the group setting, feel free to talk to us after we end. Finally, you don’t have to participate and you can leave at any time. Do you have any questions before we begin? Do you all agree to participate? Great! Let’s start by introducing ourselves. We’ll go around in a circle, and please tell everyone your name (or a name you want to be called today), your age, and something you like to do in your free time. I. PART ONE: LIFE MAP EXERCISE: Let’ start by doing a quick exercise. (Show four pieces of paper with the following headings: “Goals”, “Influences”, “Successes”, “Challenges”.) For each of these areas, we want to hear your ideas and experiences. 1. Starting with goals, what goals do you have in your life? Have those changed over time? (Clarify as needed: what do they see as success?) 2. What were some of the key influences in your life in the past in setting your goals? And what about today? 3. What has led to some of your successes in reaching your goals? What about some of your challenges? (LIST THESE OUT ON A LARGE PIECE OF PAPER) a. As needed, probe regarding family, region, etc… II. Great! Now that we have a better idea of the broader context, it would be great to hear more about what you think about SKYE. 4. If you could design your own program from scratch, what would it look like? 5. What are some of the things you think have worked well with the SKYE program? Why are they working well? Evaluation of the Guyana SKYE Program 79 6. What are some of the things that aren’t working well? Why aren’t they working well? 7. How easy or hard is it to apply what you’re learning in SKYE? 8. Finally, if you could give a one-sentence recommendation to USAID on how to make SKYE better, what would you say? 9. Do you have any other comments about the SKYE program before we wrap up the session? If the group is moving quickly, include further probes and questions as relevant to the discussion: - Do you think the project goals were similar or different from your own goals? - Was there anything not included in SKYE that you think it should include? Were you satisfied or not with the program? 80 Evaluation of the Guyana SKYE Program SKYE Qualitative Codebook How MUCH to code-Rule of Thumb: Try to put yourself in the role of the analyst. There is no hard and fast rule about how much to include. Ask yourself, do I need the surrounding information to make sense of the bit I am coding. If the answer is yes, include it in the highlighted segment. You will notice as you code that conversational segments occur over a course of lines. Generally speaking, only coding one line is not sufficient to capture the turn as a whole. On the other hand, coding an entire two pages as a single code increases the burden on the analyst. This is a bit of a balancing act. Don’t feel compelled to code EVERYTHING. The purpose of coding is to capture useful information, but not to lament over whether or not something should be coded. If only a single word is mentioned, or it doesn’t provide much context, then don’t feel you need to code it. Especially if it is captured other places. There is some level of subjectivity to coding. That is okay. We don’t all have to have exactly the same understanding. In fact, because it is qualitative, it is better to have some variation in our interpretation of data. Remember there is an option to add a comment with your codes. So if you want to provide an explanation of why you coded something the way you did, that is the way you can do it. 1.0 Good Quote: Use this code to indicate a quote that may be useful for the report as it is well￾articulated and representative of themes you are noticing in the data. If you use this with something that is not representative of what you’ve read so far or an outlier, make sure to include a comment with this code. There is a high level of subjectivity when determining if something is a good quote. If something is particularly well described or interesting, use that as your guide. You usually have an a-ha with these codes, “Wow, this person finally captured exactly what others have been saying, but with a lot of clarity.” This code will usually accompany other codes, but it doesn’t “have to.” 2.0 Program Participant Background: Any description provided of program participant’s background. This does not include background of key stakeholders. Use one of the sub-codes below, unless the specific background information has not been discussed. 2.1 Education: Use this code when a respondent describes their educational background. This includes Secondary, CXCs, Technical Schools (BIT, GITC, GTI, Kurukuru, TVET, LVET, etc). Any mentions of the fact that they dropped out of formal schooling or encountered bullying in schools should be captured with this code. This does not capture education as an outcome. That is covered under code 6.0. 2.2 Life Skills/Soft Skills: The development of life or soft skills. The skills include how to interact with co-workers and supervisors, problem-solving, communication strategies, personal hygiene, and other skills that are complimentary or necessary for the implementation of hard skills. This is sometimes referred to as emotional intelligence. 2.3 Juvenile Justice System: Any discussion the respondent has about their experience either Evaluation of the Guyana SKYE Program 81 being arrested, going to court, with a probation officer, or in “lock-up,” jail or NOC, use this code. 2.4 Family or Social Services: This may overlap some with code 2.3. It is any mention of experiences in family or social services. This includes fostering, counseling, etc. This does NOT include experiences with SKYE staff. That comes under a separate code, but may be co-coded with this. 2.5 Employment: Use this code when the respondent describes their employment experiences. This does not capture employment as an outcome. That is captured under code 6.0, Outcomes. 2.6 Family Structure: Any description of their family structure or the roles and responsibilities they have within the family, as well as roles and responsibilities other family members have. 2.7 Ethnicity: Any discussion of ethnicity in terms of being Indo-Guyanese or Afro-Guyanese. This can be attributed to themselves or their experiences with individuals from a particular ethnic group. Mentions of religion should fall under 2.8. 2.8 Culture or Religion: Discussions of religion or culture. Culture may be general Guyanese culture or history. 2.9 Gender: Mentions of one’s gender or other’s gender. For example, if a respondent describes the typical educational background of the typical male participant, you would co-code using 2.9 and 2.1. 2.10 Socio-Economic Status: Any description of one’s socio-economic status or the socio￾economic status of one’s family. 3.0 Background of Key Stakeholders: Use this code when the key stakeholder being interviewed describes the nature of his or her position. Use the sub-codes below where appropriate. If they only describe their background, then use the general code 3.0. If they discuss roles and responsibilities in their current position, use 3.1. If they discuss roles and responsibilities and their relationship to SKYE use 3.2. 3.1 Roles and Responsibilities of Job and/or of Office: Use this code when individuals describe the roles and responsibilities related to their current job or jobs as well as the roles and responsibilities of the organizations they work for. There may be some overlap with the discussions of their roles and responsibilities as they relate to SKYE. In those instances you should co-code 3.1 and 3.2 3.2 Roles and Responsibilities in SKYE: Use this code when individuals describe the role they play in the SKYE program. For SKYE coaches and trainers, use only this code, unless they are describing work they do outside of SKYE. Use this code also to capture how individuals 82 Evaluation of the Guyana SKYE Program became involved in the SKYE program. 4.0 Influences: Any mention of something that has served as an influence in a youth’s or youths’ lives. By influences, we mean those things that have an effect on youth outcomes. This should focus on explicit statements that say, “X or Y were a particularly positive influence in this youth’s life.” It is NOT up to the coder to determine if something is a positive influence or a negative influence. Rather, you are coding for explicit mention of influences on youth. You should attempt to use the sub-codes below. However, if the respondent is not explicit about whether it was positive or negative, then use 4.0. 4.1 Positive Influences: Any mention of positive influences in youths’ lives, these can come from both key informants and youth. This should focus on explicit statements that say, “X or Y were a particularly positive influence in this youth’s life.” It is NOT up to the coder to determine if something is a positive influence or a negative influence. Rather, you are coding for explicit mention of influences on youth. 4.2 Negative Influences: Any mention of negative influences in youths’ lives, these can come from both key informants and youth. This should focus on explicit statements that say, “X or Y were a particularly negative influence in this youth’s life.” It is NOT up to the coder to determine if something is a positive influence or a negative influence. Rather, you are coding for explicit mention of influences on youth. 5.0 Needs for Guyanese Youth: Perceived needs of Guyanese Youth as identified by any of the respondents. 6.0 Youth Outcomes of SKYE programming: Use this code to identify any youth outcomes that respondents link to the SKYE programming. If it is possible to distinguish the type of outcome discussed, use the sub-codes below. If not, use the general code 6.0. Changes in policies at the national, regional, or local level, as well as redirection of resources and building of infrastructure falls under the sustainability code 6.1 Education: Identification of additional education (formal or informal) as an outcome of SKYE. 6.2 Life/Soft Skills: Identification of life or soft skills as an outcome of SKYE. 6.3 Employment: Identification of employment as an outcome of SKYE. Do include starting ones own business under this. Make sure to double-code with 7.0. That way the analyst will know that it is starting one’s own business as a result of SKYE. 6.4 Criminal Behavior/Crimes: Identifying changes in crime rates or criminal behavior as an outcome of SKYE 6.5 Civic Engagement: Increased or decreased engagement in civic engagement as an outcome of SKYE 6.6 Positive Outcome: Positive as related to the stated objectives of the program. Increased Evaluation of the Guyana SKYE Program 83 education, improved life skills, increased civic engagement, reduced criminal behavior, etc. Make sure to double code this with 6.1-6.5 above. 6.7 Negative Outcomes: Negative as related to the state objectives of the program. This means moving in the opposite direction of the stated objectives. For example, dropping out of school (rather than enrolling), development of negative social behaviors, increase in criminal behaviors, leaving a job or being fired from a job, etc. If the individual does not make positive progress towards the project objective it is NOT seen as a negative outcome, but as a neutral outcome. Make sure to double code this with 6.1-6.5 above. 7.0 Starting Own Business: Any mention of a youth starting their own business. Also use this code when individuals discuss the entrepreneurship training. 8.0 Program Participant Goals: Discussions by either program participants or key informants of life goals that the youth have. Will likely be mostly used during FGDs and interviews with youth. However, coaches and trainers may also discuss goals. They may also refer to it as an individual employment plan or individual livelihood plan. However, goals may not exclusively focus on employment. 9.0 Recruitment: Discussion of how youth are recruited into the program. 10.0 Role of Coach/Experience with Coach: Perceived role of the coach as described by any of the stakeholders-youth, coaches, trainers, implementing partners, employers, representatives from the juvenile justice system, etc. Also use this code to identify any excerpts where respondents discuss their experience with the coach. 11.0 Role of Trainer/Experience with Trainer: Perceived role of the trainer as described by any of the stakeholders-youth, coaches, trainers, implementing partners, employers, representatives from the juvenile justice system, etc. Also use this code to identify any excerpts where respondents discuss their experience with the trainer 12.0 Experience with Training or Description of Training: Description by youth of their experiences in the training, or description of other key stakeholders of the training 13.0 Experiences with or Descriptions of Other Aspects of the SKYE Program: Descriptions by youth of their experiences with the SKYE program aside from trainings, trainer, coaches, and recruitment. For example, use code 13.0 if a youth describes there interactions with their peers from SKYE. Also descriptions of other aspects of SKYE by other stakeholders. 14. Collaboration, Coordination, or Engagement: Includes those instances where collaboration, coordination or engagement is discussed. There is different terminology used, but this is supposed to captures individuals or groups working together. Engagement captures their active involvement with the program, participants, or EDC. Do not include those instances of lack of collaboration, coordination and engagement. They fall under code 15.0. This information can come either from youth themselves or other stakeholders. There are two primary instances that will be coded under this, the times when individuals explain their experience collaborating or coordinating with the stakeholders below, or when a respondent says that there is collaboration between two parties. If two parties are identified they 84 Evaluation of the Guyana SKYE Program should be co-coded. For example, if someone mentions that the coaches and trainers work together on something, it should be co-coded with coaches 14.1 and with trainers 14.2. 14.1 With Coaches 14.2 With Trainers 14.3 With Youth 14.4 With Private Sector 14.5 With Public Sector National (aside from legal system) 14.6 With Public Sector Regional or Local (aside from legal system) 14.7 With EDC 14.8 With the Legal System 14.9 With families and communities 14.10 With NGOs or INGOs 15.0 Lack of Collaboration, Coordination, or engagement: Mentions of a lack of collaboration and coordination between stakeholders. Just as with the codes under 14, they can be co-coded. These codes can also be used when someone describes a lack of coordination, collaboration or engagement. 15.1 With Coaches 15.2 With Trainers 15.3 With Youth 15.4 With Private Sector 15.5 With Public Sector National (aside from legal system) 15.6 With Public Sector Regional or Local (aside from legal system) 15.7 With EDC 15.8 With the legal system 15.9 With families and communities 15.10 With NGOs or INGOs 16.0 Strengths/successes of the SKYE Project: Perceived strengths or successes of the SKYE program 17.0 Weakness/challenges of the SKYE Project: Perceived weaknesses or challenges of the SKYE program 18.0 Definitions of “At-Risk” or “Targeting of Youth”: Code instances in which someone either defines what at-risk means or mentions targeting at-risk youth and then provides an example. We are trying to capture how it is understood by different stakeholders 19.0 Sustainability: Sustainability will be assessed largely through engagement and collaboration with public and private institutions. However, anytime the word “sustainability” is used in conversation to refer to either sustainability of outcomes or programming, it should be coded with 18.0. Also may use the sub-codes below as indicators of sustainability 19.1 Policy Development and/or implementation of Policy: Discussions of the development of policies to sustain programming or outcomes. Evaluation of the Guyana SKYE Program 85 19.2 Financial Resources: Discussions of needs or provision of financial resources to sustain programming and/or outcomes. 19.3 Institutional Structures: The establishment of new public institutional structures in order to sustain SKYE. This is distinct from code 19.2, as it refers to the creation of institutional structures or programs without reference to financial resources. For example, one may encounter instances in which individuals discusss the implementation of “diversion programs.” This is a structure within the justice system that would allow youth to go before a community committee before being put through the formal system. This would be coded 19.3. Another example is the implementation of mentors in schools. Again, this is 19.3. If these are discussed in conjunction with additional funding, then it should be co-coded 19.2 and 19.3. 19.4 Capacity Building: Discussion of the building of capacity or lack thereof of individual or organizations that may take on some of the roles and responsibilities of SKYE 20.0 Satisfaction: Discussions of the level of satisfaction of SKYE programming. We gauge program satisfaction by youth in the youth survey, but this will help us capture levels of satisfaction by other stakeholders. This can capture either satisfaction OR dissatisfaction. 21.0 Recommendations: Any recommendations provided by respondents to improve SKYE or to sustain the outcomes experienced with SKYE 86 Evaluation of the Guyana SKYE Program Annex III: Desk Review Documents Reviewed ▪ SKYE quarterly reports (primarily from 2012-2014) ▪ SKYE Program Description (November 2012) ▪ SKYE Performance Monitoring Plan ▪ SKYE Assessment Report and Program Design (November 2011) ▪ CBSI Juvenile Justice Sector Assessment Final Report ▪ Dominican Republic Youth Project Design (2011) ▪ USAID/Dominican Republic Cross-Sectoral At-Risk Youth Assessment (August 2010) ▪ A Qualitative Study on HIV Vulnerability among Young Key Affected Populations in Guyana (2013) ▪ EQUIP3 Lessons Learned: Experiences in Livelihoods, Literacy, and Leadership in Youth Programs in 26 Countries (2012) ▪ Gender in Youth Livelihoods and Workforce Development Programs ▪ Guyana Common Country Assessment 2012-2016 by the Government of Guyana and UN Country Team (2011) ▪ Guyana Private Sector Assessment Report (2014) ▪ UNICEF Annual Country Report for Guyana (2012) ▪ Caribbean Youth Development: Issues and Policy Directions (2003) ▪ Eastern and Southern Caribbean Youth Assessment (ESCYA) (2013) ▪ Eye on the Future: Investing in Youth Now for Tomorrow’s Community (January 2010) ▪ Rapid Youth Assessment in the Eastern Caribbean (August 2008) ▪ Skills and Knowledge for Youth Employment (SKYE) in Guyana Assessment Report and Program Design (November 2011) Evaluation of the Guyana SKYE Program 87 Annex IV: Sources of Information Number of KIIs Affiliated Institution Region 4 Key Informants 1 PS Ministry of Culture 6 Trainer, Region 3, 6 1 GNIC 3 Ministry of Labour 1 Director; Volunteer Youth Corp 1 Administrator NOC 1 SIMAP Building 1 Child Care Protection 1 Youth Challenge Guyana 1 New Timerhi Handling Services 1 AEA 1 Youth Business Trust 2 Private Sector-Employer 2 Georgetown Chamber of Commerce 1 Guyana Industrial Training Centre 1 Senior Consultant 1 Board of Industrial Training 1 Magistrate 1 Strategic Management Department, GPF 1 Chief Education Officer 1 Human Resource Officer, Qualfon 2 Human Resource Representatives, BK Inc 88 Evaluation of the Guyana SKYE Program 1 Howard Construction 1 Kuru Kururu Training Centre 1 TVET, Ministry of Education Region 10 Key Informants 2 RDC 1 HR, Bosai 1 Probation & Welfare Officer, MHSSS 1 Linden Technical Institute 1 Commander of Linden Police Force 1 Board of Industrial Training 3 Parents Interview 1 Linden Enterprise Network 3 Trainers Region 6 Key Informants 3 Coaches 2 Olendorff Employer 1 Government Regional Chair’ Government Regional Education Officer 1 Magistrate 1 Technical Institute 1 Coordinator of Ministry of Culture, Youth and Sport 1 Chamber of Commerce 1 Probation Officer 2 Ministry of Education 1 Trainer for SKYE Evaluation of the Guyana SKYE Program 89 Region 9 Key Informants 2 Public School Teachers at Aishalton 2 Chief and Regional Education Officer 2 Private Sector Employer 1 Vice Chairman 1 Probation Officer 1 Trainer/Coach 1 Regional Literacy Officer 2 Tushaos of Aishalton 2 Member of Village Council in Aishalton 1 Tushao of St. Ignatius 1 Head teacher, Secondary school Youth Interviewed Region Number of Youth Interviewed Four 22 Ten 6 Six 7 Youth Focus Group Discussions Region Number of FGD Conducted Four 5 Ten 1 Six 2 90 Evaluation of the Guyana SKYE Program Annex V: YAG Agenda Youth Advisory Group: SKYE Program Evaluation September 10, 2014 AGENDA GOALS: introduce group members to evaluation, assessing our evaluation strategy, providing feedback on instruments 10-10:30 Welcome and Introductions (Activity: Interview your partner) 10:30-11:15 What is evaluation? ● What is the job of an evaluator? ● Evaluation questions ● Design-A strategy for answering questions ● What is your role in this evaluation? 11:15-11:45 Strategy ● Overview of the current approach ● Feedback from YAG on possible changes 11:45-12:45 Lunch 12:45-2:30 Reviewing Tools ● Survey ● Key Informant Interviews-Non-Youth Stakeholders ● Youth Focus Group Discussions ● Key Informant Interviews-Youth 2:30-3:00 YAG Closing ● Summarize activities from the day (Activity: Name something you learned today) ● Next Steps Evaluation of the Guyana SKYE Program 91 Annex VI: Utilization Matrix Recommendati on Accept/Reject If Reject, Why? If Accept, Who is Responsible? If Accept, By When? 92 Evaluation of the Guyana SKYE Program Annex VII: Conflict of Interest Forms Evaluation of the Guyana SKYE Program 93 94 Evaluation of the Guyana SKYE Program Evaluation of the Guyana SKYE Program 95 96 Evaluation of the Guyana SKYE Program Evaluation of the Guyana SKYE Program 97 98 Evaluation of the Guyana SKYE Program Evaluation of the Guyana SKYE Program 99 100 Evaluation of the Guyana SKYE Program U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523