November 2014 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Layla Moubayed, Team Leader, Khaled Qubajah and Wala’a Aqrabawi (through Integrated Services Indigenous Solutions), and Maram Barqawi and Dima al-Naber of MESP. FINAL PERFORMANCE EVALUATION OF THE USAID/JORDAN YOUTH FOR THE FUTURE (Y4F) PROJECT ii FINAL PERFORMANCE EVALUATION OF THE USAID/JORDAN YOUTH FOR THE FUTURE (Y4F) PROJECT November 16, 2014 Contracted Under AID-278-C-13-00009; USAID Monitoring & Evaluation Support Project (MESP) DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. 1 CONTENTS Acronyms............................................................................................................................................................................. 2 Executive Summary ........................................................................................................................................................... 3 Evaluation Purpose and Background .................................................................................................................... 3 Evaluation Questions ............................................................................................................................................... 4 Evaluation Methods and Limitations .................................................................................................................... 4 Findings and Conclusions........................................................................................................................................ 5 Evaluation Purpose and Evaluation Questions .......................................................................................................... 10 Project Background ......................................................................................................................................................... 11 Evaluation Methods and Limitations ............................................................................................................................ 15 Methods .................................................................................................................................................................... 15 Limitations ................................................................................................................................................................ 16 Findings, Conclusions & Recommendations .............................................................................................................. 17 Youth Outcomes .................................................................................................................................................... 17 Findings ............................................................................................................................................................. 17 Education ......................................................................................................................................................... 18 Employment .................................................................................................................................................... 19 Conclusions ..................................................................................................................................................... 23 Effectiveness & Sustainability of Capacity Building .......................................................................................... 24 Findings ............................................................................................................................................................. 24 Conclusions ..................................................................................................................................................... 27 Learning /Core Program Models ........................................................................................................................ 28 Findings and Conclusions ............................................................................................................................. 28 Gender and Cultural Challenges ......................................................................................................................... 29 Findings ............................................................................................................................................................. 29 Conclusions ..................................................................................................................................................... 30 Recommendations ........................................................................................................................................................... 31 Annexes ............................................................................................................................................................................. 33 Annex I: Evaluation Statement of Work ........................................................................................................... 34 Annex II: Evaluation Design Report ................................................................................................................... 43 Annex III: Evaluation Tools .................................................................................................................................. 80 Annex IV: Sources of Information .................................................................................................................... 105 Annex V: Data Collection Methodology......................................................................................................... 108 Annex VI. Youth Survey Data Report ............................................................................................................. 115 2 ACRONYMS CBOs Community-Based Organizations CMU Change Management Unit CV Curriculum Vitae FG Focus Group FGD Focus Group Discussions GoJ Government of Jordan HCY Higher Council of Youth HVAC Heating, Ventilation and Air Conditioning IRs Intermediate Results IT Information Technology IYF International Youth Foundation KIIs Key Informant Interviews M&E Monitoring and Evaluation MESP Monitoring and Evaluation Support Project MoL Ministry of Labor MoSD Ministry of Social Development NET National Employment and Training NGOs Non-Governmental Organizations OSS One Stop Shops PTS Passport to Success SoW Evaluation Statement of Work Y4F Youth for the Future VTC Vocational Training Corporation YWJ Youth: Work Jordan YFS Youth Friendly Services 3 EXECUTIVE SUMMARY EVALUATION PURPOSE AND BACKGROUND The purpose of this performance evaluation carried out by the Monitoring and Evaluation Support Project (MESP) is to assess the process, methodologies, and outcomes of Phase II of the Youth for the Future (Y4F) project and to measure the sustainability of the achievements related to project beneficiaries. Y4F is a $33 million project implemented by the International Youth Foundation (IYF) from March 2009 through December 2014 with a long-term goal of “creating an enabling environment with a greater capacity to more effectively serve youth-at-risk.” This goal is intended to be realized through three intermediate results: improved models and practices for working with youth; building capacity of Community Based Organizations (CBOs) to provide access to long-term employment for youth; and improved provision of youth friendly services. The project targeted vulnerable populations of youth aged 15-24 years old in nine communities: Zarqa, Russaifeh, Irbid, Jordan Valley and Southern Shouneh, Maan, Mafraq, East Amman and Sahab. Y4F worked through local CBOs conducting activities to cultivate positive life skills; to provide work skill training to enhance employability; and to provide services linking youth with employers. In order to sustain these services for youth, the project endeavored to build both the management and programming capacity of its local CBO partners as well as build capacity of select public sector institutions to better meet the needs of targeted youth populations. Rather than evaluating project performance per se, the evaluation instead focuses on project outcomes for youth and sustainability of capacity building to partner organizations. As addressing positive transition for youth into adulthood is a top priority for the USAID Country Development Cooperation Strategy (CDCS), the evaluation is intended to inform USAID as it elaborates its strategy for youth engagement. The Statement of Work (SOW) for the evaluation directs focus on the following areas:  Phase II of the project when Y4F changed to direct implementation with CBOs in 2011.  The outcomes/results achieved with the main target groups: a) beneficiary youths and b) youth￾serving institutions (public and non-governmental).  Assessing the ‘why’ of outcomes, i.e., what factors were relevant to the successful achievement of results and alternatively, what factors impeded the realization of planned results.  Appraising sustainability of outcomes.  Strategic recommendations for future programming based on lessons learned from Y4F evaluation results. Primary stakeholders for Y4F include 13 Community Based Organizations (CBOs), Ministry of Labor (MoL), Ministry of Social Development (MoSD),1 Vocational Training Corporation and Centers (VTCs), and the Higher Council of Youth (HCY). 1 The Ministry of Social Development is responsible for CBOs 4 EVALUATION QUESTIONS The evaluation covers seven questions in four areas designed to evaluate youth outcomes: sustainability and capacity building, effectiveness, gender issues, and lessons learned from the Y4F models and implementation approaches. The specific evaluation questions are as follows: Youth Outcomes 1. As a result of Y4F efforts, are youth in a better position now than they were prior to the project in regards to: a. Re-entering the formal education system then remaining in the formal education system? b. Entering the labor market then remaining in the labor market? c. Demonstrating positive life skills as defined by the Y4F training curriculum on positive life skills? Capacity Building 2. How effective has Y4F been at building organizational capacity (financial viability, staffing, skills and knowledge needed to run the organization and provide intended services to youth, autonomous of any external assistance, etc.) among local CBOs and public sector stakeholders involved in the project? 3. Do CBOs supported by the Y4F project have the capacity, as defined previously, to continue youth programming? 4. Are there other organizations, such as vocational training colleges (VTCs), also strengthened by Y4F that USAID should continue to work with on positive youth development? Learning 5. Were the two core program models (“successful transitions to work” and “sustainable community youth hubs”) appropriate models for achieving the project’s stated results? 6. Are there strategies or practices IYF implemented that should be replicated or avoided in future youth programming? Gender 7. How did Y4F address any gender differential constraints (accessing, participating in, or benefitting from) in project activities? EVALUATION METHODS AND LIMITATIONS The evaluation used a mixed methods approach, combining quantitative and qualitative data to measure outcomes and to provide an explanation to issues underlying those outcomes. Data collection was conducted through phone surveys with youth; focus group discussions with youth and parents; key informant interviews with public sector stakeholders; group interviews with staff and management of all 13 CBO partners; and Y4F reports. The phone survey was conducted with 352 youth; the same survey was conducted with 83 youth during the focus group discussions, totaling 435. A full explanation of the methodology is contained in the Evaluation Design Report in Annex II. Evaluation tools are contained in Annex III. A limitation of the methodology may be in the unknown reliability of responses from the phone survey as information is self-reported and anonymous calls from adults to youth enquiring about their activities and accomplishments is not common practice. Nonetheless, responses from surveys conducted with focus group participants reflected similar responses. 5 FINDINGS AND CONCLUSIONS Evaluation Question 1: As a result of Y4F efforts, are youth in a better position now than they were prior to the project in regards to: a. Re-entering the formal education system then remaining in the formal education system? Given the almost unanimous agreement among stakeholders that life skills training had a positive effect on youth, it could be concluded that youth are indeed “in a better position now” in regards to their decision-making ability and agency relative to returning to school. However, given participants’ low socio-economic status, perceived lack of socio-economic mobility, illiteracy (for some), and lack of activities within Y4F designed specifically to facilitate re-enrollment, youth may well be exercising rational decision-making by not re-enrolling. With the Y4F target age cohort of 15-24, it is likely that re￾enrollment would only appeal to those who dropped out after age 14, as younger drop-outs would have missed so much school that their low literacy and learning levels would create a serious impediment. b. Entering the labor market then remaining in the labor market? Given the synergy among life skills training, vocational skills training, and employment linkages provided by Y4F, it appears that the youth who participated in Y4F are “in a better position now” to enter the workforce, as confirmed by both youth and employers, and particularly for youth who were trained in the vocational areas targeted by the National Employment Strategy. Even the youth who were trained in sectors in which they are unlikely to work due to geographic or cultural inappropriateness are likely to be in “a better position” to enter the workforce if they are able to transfer their skills to occupations that are suitable. However, they may not be in a better position to remain in the workforce; while they are equipped with better technical and soft skills than before Y4F, their expectations for position and salary were heightened by Y4F which may have contributed to their leaving a job. c. Demonstrating positive life skills as defined by the Y4F training curriculum on positive life skills? With youth reporting benefit, usefulness, and positive impact ranging from 86 percent to 94 percent on 16 of 17 personality dimensions of the training program,2 in addition to both parents and employers acknowledging positive behavior change, it may be concluded that youth are demonstrating positive life skills. Nonetheless, the extent to which this intervention may have increased conflict within the family bears further investigation, as does the potentially exponential positive impact of engaging parents in the process. Evaluation Question 2: How effective has Y4F been at building organizational capacity (financial viability, staffing, skills and knowledge needed to run the organization and provide intended services to youth, autonomous of any external assistance) among local CBOs and public sector stakeholders involved in the process? Y4F capacity building among CBOs and partners appears to have been effective in service provision; management capacity building may have been effective on an individual level but less effective on an institution/organizational level. While some CBOs claim to have integrated institutional and operational systems, the extent to which they have is dependent upon their size and level of operations. Those that 2 This tool was introduced by the independent evaluation team Technical and Vocational Training Specialist on the evaluation team based upon a tool used by UNDP to assess the behavior and attitude changes in youth resulting from life skills training. 6 are more established and have funding streams external to Y4F appear to have integrated systems more fully, while the smaller CBOs without funding sources have little activity and therefore minimal opportunity to use the systems. For the small CBOs, most staff trained in management processes and procedures were hired expressly for and paid by the Y4F grant. As Y4F funding ended, CBO staff who had been implementing activities left. However, some program staff continue offering youth services on an ad-hoc volunteer basis. CBOs that have other sources of funding are likely to maintain use of the management practices that Y4F introduced. Y4F was effective in integrating life skills training into the Vocational Training Corporation, making it mandatory in VTC hospitality curricula and certifying VTC staff as trainers of Y4F’s proprietary PTS life skills training package. Regardless of their effectiveness in providing employment services, lack of awareness and negative public perception render Ministry of Labor (MoL) employment directorates under-utilized by youth. The success of One Stop Shops (OSS) in Abdali and Irbid demonstrates that public perception of MoL can be positive if the facility is in good physical condition and is well-equipped. Sustainability of the OSS model and newly-introduced services in Abdali and Irbid employment centers are likely to be sustained as services and systems have been integrated into the work flow, providing expanded youth services and networking with the private sector through job fairs and referral to potential employers. Sustainability of the Higher Council for Youth (HCY) Youth Leadership Training program is likely as it was originally part of the HCY plans. Evaluation Question 3: Do CBOs supported by the Y4F project have the capacity to continue youth programming? While CBOs have demonstrated capacity to continue youth programming, doing so is ultimately dependent upon funding. Nonetheless, motivation exists as demonstrated through some CBOs continuing youth services by charging nominal fees for workshops, tapping into funding streams through other donors, and providing services on an ad-hoc basis through volunteers. Evaluation Question 4: Are there other organizations such as Vocational Training Colleges (VTCs) also strengthened by Y4F that USAID should continue to work with on positive youth development? Given the expansion of life skills training throughout VTC curricula, combined with VTCs’ linkages with employers in high-growth sectors, VTCs offer enormous potential for youth in areas where job opportunities are available in the sectors in which VTCs train. As the HCY Youth Leadership program is in its nascent stage, an assessment of its positive potential is premature. MoL/OSS could be effective in providing services that assist youth in transitioning to work if a holistic package of technical assistance, facilities upgrading, and a campaign addressing public perception were undertaken. Any one form of assistance on its own would likely not result in a substantial increase in youth accessing services. Evaluation Question 5: Were the two core program models “successful transitions to work” and “sustainable community youth hubs” appropriate models for achieving the project’s stated results? While the model of Transition to Work was appropriate, some aspects of implementation may have limited achievement of the project results. The synergy of life skills training and vocational training appears to be critical in youth transitioning to work. While technical training through recognized colleges added a level of prestige, served youth who could not otherwise have afforded it, and linked participants to jobs, the narrow focus on national priority sectors of industry (hospitality, retail and 7 heating/ventilation and air conditioning) was not appropriate for females and for those living in areas without employment opportunities in those priority sectors. While the Community Youth Hubs model is appropriate as places for youth to access services, each institution has unique operational approaches and constraints that determine its effectiveness. For example, CBOs have good reach and relationships of trust with youth due to their location in the community, which encourages attendance. VTCs have a good reputation for technical training and job placement and can supplement this with life skills. One Stop Shops have particular competence in job placement, but are not considered a desirable source of assistance by youth. Evaluation Question 6: Are there strategies or practices IYF implemented that should be replicated or avoided in future youth programming? Practices to replicate include requiring both life skills and technical training for participants, and offering technical training through recognized institutions. Practices to avoid include inflating expectations about the type and level of job; offering technical training only in sectors identified as high-growth but without job opportunities near participants’ homes; training of a duration that is perceived by participants as insufficient to increase their employability; and offering only technical skills that communities recognize as inappropriate for females. Evaluation Question 7: How did Y4F address any gender differential constraints (accessing, participating in, or benefitting from) in project activities? Although the Y4F design does not appear to consider gender constraints, Y4F was responsive in addressing some of the gender constraints. Specific steps taken include segregating training and transportation, and meeting with parents who were resistant to females participating. Nonetheless, technical training remained focused on occupations in which only few females in the target group were able to participate. RECOMMENDATIONS Based on the findings and conclusions describe, the following recommendations are offered to USAID in designing future activities: 1. Activities and indicators should be designed for alignment with objectives; if an objective is to increase re-enrollment, activities should be designed accordingly. 2. Changes to project objectives should be clearly documented in writing in the revised AMEP or contract modification (if applicable). 3. Activities addressing re-enrollment in education should target younger cohorts (ten years old) in order to mitigate the impediment of illiteracy and the increasing reluctance to re-enroll, as well as exclusion of youth who have been out of school for three years. 4. Activities addressing re-enrollment in education should acknowledge and address the issues of financial need to leave school (or not re-enroll), and the perceived lack of value for education to enable socio-economic mobility. 5. Make life skills training an integral component of youth programming. Offer life skills training for females that is not necessarily linked to the objective of employment or education, but rather with the objective of overcoming shyness and increasing their voice and agency. 6. Include life skills training for parents to avoid backlash and to create a more enabling and supportive environment for youth development. 7. Facilitate all partners who are training youth to include life skills training as part of their offering. Consider tailoring life skills to categories of objectives such as employment, education, and parenting. 8 8. Require that implementing partners who are using life skills training funded by USAID to use curricula and approaches that are non-proprietary so as to facilitate broad and sustained use of the curricula and approach. 9. Offer options for technical skills training that include both high-growth sectors (in geographic locations where such jobs exist) and skills training that is more aligned with job opportunities in the local market. 10. Conduct a participatory assessment to determine the types of jobs that would be considered by youth, families and communities as appropriate for females. Engage youth, family, community members, employers and training providers to identify mechanisms that could gently challenge gender norms and make non-traditional occupations available to females. Such mechanisms might include incentives for employers for areas adjusting work hours and providing transportation for females; and inviting family members to places of employment to assure them of safe environment for females. 11. Continue to support CBOs and build their capacity to provide youth services, particularly youth spaces that may be more likely to attract female participants who are not on a career-track per se. Such spaces should have low maintenance cost for sustainability, and offer more recreational than educational services to draw larger numbers of youth through sports and team competitions, social and educational games. 12. Support the Ministry of Labor in replicating the Abdali One Stop Shop Model in other MoL employment directorates with the provision of life skills training, employment counseling to introduce options and manage expectations, and facilitating linkages with employers through hosting employer Q&A sessions, job fairs, and other mechanisms. 13. Concurrent with supporting public institutions to improve services and upgrade facilities, support MoL in changing its public perception to enhance youth demand for their services. 14. Continue to support the Vocational Training Centers and other institutions and programs that offer a combination of life skills training and vocational training. 15. Support the VTCs in identifying vocational training for females that is both marketable and considered culturally appropriate. 16. Determine whether three months of theoretical and three months of on-the-job training is sufficient to increase employability from the perspective of both employers and participants. Evaluate the duration of the training program through the lens of youth segmentation/sub￾groups. 17. Determine effectiveness of career counselors in assisting participants in securing employment. 18. Implementing Partners should work with providers of technical training and life skills training in locations that are easily reached by the target population. 19. USAID should consider including in project designs rigorous M&E systems that enable long-term evaluation of components of similar youth programs; the recommended M&E system would enable tracing the trajectory of organizations and individuals over a period of time and a deeper evaluation of the efficacy of individual components and sub-grantees. Consider a follow-up study at an interval of one or two years to provide information on long-term sustainability. 20. USAID should support the Government of Jordan to explore opportunities for VTCs to bring training (in vocations that are locally relevant) to population centers where VTCs do not exist. This off-campus training could be conducted in youth centers, CBOs or OSS facilities. 21. Expand technical training to include employment opportunities at the local level and employment opportunities that are culturally feasible for females. 22. Implementing partners should take measures to pro-actively manage youth expectations for employment salary and working conditions through career counseling, work site visits, and meeting youth who are already working in the sector. 23. Engage partners according to their areas of strength, e.g., CBOs for outreach to and mobilization of youth and youth friendly spaces (particularly in attracting females); VTCs for 9 technical training, employer linkages; and One Stop Shops to assist with job placement. IPs should facilitate collaboration among partners to acknowledge their core competencies and leverage each partner’s strengths as a condition of assistance to minimize competition for resources. 24. Require implementing partners that are using life skills training funded by USAID to use curricula and approaches that are non-proprietary so as to facilitate broad and sustained use of the curricula and approach. 25. Offer life skills training to youth (and particularly females) who do not necessarily have educational or employment objectives. 26. USAID should work with the Government of Jordan to implement public perception/behavior change communication targeting male family members as they are often the decision-makers for females, and community and religious leaders who often “police” behavior of families who are considered permissive by allowing females to work. 27. Require a gender analysis of female transition to work in order to identify employment opportunities that are culturally acceptable, and therefore accessible to females. The gender analysis should also identify approaches to increase female participation in other activities offered through youth centers in order to ensure equitable participation in and benefit from USAID resources. 28. Design future activities addressing youth to be in alignment with USAID Gender Equality and Female Empowerment policies for gender-sensitive design, management, implementation, indicators and budget. This may require challenging social norms and should be done in close consultation with females, families, community members and employers to ensure alignment with the U.S. “do no harm” policy for foreign assistance. 29. Require the IP to align technical training offerings with job skills that are marketable and that are accessible to females, given the cultural impediments in their region. 10 EVALUATION PURPOSE AND EVALUATION QUESTIONS This Performance Evaluation is intended to provide USAID/Jordan with overall feedback and strategic recommendations for positive youth development strategies and best practices for the capacity building and sustainability of youth serving institutions, particularly CBOs. The management purpose of the evaluation is to help USAID/Jordan obtain the necessary information to more effectively allocate resources for future programming. USAID/Jordan is aware of many of the successes and shortcomings of the Y4F project and thus, would like to focus on what it doesn’t already know. For this reason, this evaluation focuses primarily on PHASE 2 of the project, although it is assumed that capacity building initiatives and youth outcomes that were a result of Phase 1 activities will be included. Rather than evaluating project performance per se, the evaluation instead focuses on project outcomes for youth and sustainability of capacity building to partner organizations. As addressing positive transition for youth into adulthood is a top priority for the USAID Country Development Cooperation Strategy (CDCS), the evaluation is intended to inform USAID as it elaborates its strategy for youth engagement. The Statement of Work (SOW) for the evaluation directs focus on the following areas:  Phase II of the project when Y4F changed to direct implementation with CBOs in 2011.  The outcomes/results achieved with the main target groups a) beneficiary youths and b) youth￾serving institutions (public and non-governmental).  Assessing the ‘why’ of outcomes, i.e., what factors were relevant to the successful achievement of results and alternatively, what factors impeded the realization of planned results.  Appraising sustainability of outcomes.  Strategic recommendations for future programming based on lessons learned from Y4F evaluation results. The evaluation questions as stated in the SOW are as follows: Youth Outcomes 1. As a result of Y4F efforts, are youth in a better position now than they were prior to the project in regards to: a. Re-entering the formal education system then remaining in the formal education system? b. Entering the labor market then remaining in the labor market? c. Demonstrating positive life skills as defined by the Y4F training curriculum on positive life skills? Capacity Building 2. How effective has Y4F been at building organizational capacity (financial viability, staffing, skills and knowledge needed to run the organization and provide intended services to youth, autonomous of any external assistance, etc.) among local CBOs and public sector stakeholders involved in the project? 3. Do CBOs supported by the Y4F project have the capacity, as defined previously, to continue youth programming? 11 4. Are there other organizations, such as vocational training colleges (VTCs), also strengthened by Y4F that USAID should continue to work with on positive youth development? Learning 5. Were the two core program models (“successful transitions to work” and “sustainable community youth hubs”) appropriate models for achieving the project’s stated results? 6. Are there strategies or practices IYF implemented that should be replicated or avoided in future youth programming? Gender 8. How did Y4F address any gender differential constraints (accessing, participating in, or benefitting from) in project activities? PROJECT BACKGROUND Youth for the Future (Y4F) started in 2009 with a broad scope, working with a wide array of local community stakeholders to create an enabling environment that will help disadvantaged young people (i.e., out of school, out of work, age 15-24) get and keep a job, open and maintain a business, or go back to school, while becoming more connected to their communities through greater civic engagement. In so doing, the program is also designed to raise the awareness among key stakeholders in Jordan of the issues facing the youth population and to empower them to increase the orientation and capacity of youth-serving institutions to more effectively serve youth and prepare them for the future. Funded by USAID and implemented by the International Youth Foundation (IYF), Y4F is executed in partnership with the Government of Jordan (GoJ) and a network of Community-Based Organizations (CBOs) working in target communities as well as national public, private, and civil society partners. Following a midterm project evaluation, a project audit, and internal strategic reviews with USAID, Y4F modified its activities as follows:  A change in the operational model from implementation through national coordination NGOs (CNGOs) to direct implementation through CBOs.  Expansion of geographic areas from the original target communities of Zarqa, Rusiefeh, Irbid, Jordan Valley, Ma’an, and East Amman to include additional communities in Mafraq, Sahab and the Jordan Valley.  An increase in the program budget from $30 million to $33 million3 in order to expand public sector capacity building activities to other youth serving Ministries including the Ministry of Labor (MoL), the Vocational Training Corporation (VTC) and the Higher Council of Youth (HCY).  Discontinuing the civic engagement component and entrepreneurship activities, and consolidating all activities within two core program models: 1) Preparing Youth for Successful Transitions for Work; and 2) Building Sustainable Community Youth Hubs. 3 The life-of-project budget as of December 2014 was $30,331,477. 12  Re-branding the project name from Youth: Work Jordan (YWJ) to Youth for the Future (Y4F).  Extension to December 21, 2014 to continue its scaling and sustainability activities, particularly with public sector partners. This evaluation focuses on the second phase of Y4F, which began in July 2011. In this phase, Y4F focused its work on the two core program models as follows: Model 1- Preparing Youth for Successful Transition to Work: This model encapsulates Y4F’s employability interventions including quality life skills and career guidance support.  Life Skills Training: Y4F focused on establishing sustainable delivery mechanisms for Passport to Success (PTS), IYF’s proprietary life skills curriculum, to teach youth personal competency, problem solving skills, productive work habits, CV writing and job search skills. Y4F worked through selected CBOs to establish partnerships with key service providers including VTC and Al-Quds College to sustain the delivery of PTS beyond the life of the program. In addition, Y4F worked with VTC to integrate PTS into its youth training and make PTS mandatory for all VTC hospitality training centers.  Sector-Specific Employment: In cooperation with CBOs and private (Ammon, Al-Quds college and LG/Al Asalah) and public sector (VTCs) partners, Y4F offered demand-driven sector-specific training programs to prepare youth for employment. The sectors, which were selected with MoL, included hospitality, retail, and facility maintenance including heating, ventilation and air conditioning (HVAC).  Career Development: Y4F developed My Career, My Future toolkit, a comprehensive how-to guide for implementing career guidance in a community. The toolkit was piloted with Y4F’s CBO partners, VTC, and MoL One Stop Shops (OSS). Model 2 - Building Sustainable Community Youth Hubs: This model empowered community partners to build and sustain a network of quality youth services and to expand the cadre of qualified youth workers and community trainers. Key strategies included: (1) Enhance technical competencies of community partners to organize and deliver effective youth programs; (2) Build institutional and operational capacity of CBOs to manage and sustain youth programs and services; (3) Enhance competencies of public sector field directorates to organize and deliver effective youth programs; (4) Build community alliances and engage public sector field directorates to coordinate efforts, provide infrastructure, leverage support, and build community sustainability models; (5) Integrate structured approaches to engage parents and involve youth in the design and management of activities; and (6) Build a pool of professional youth workers trained and certified in core competencies for positive youth development. Y4F criteria for youth beneficiary selection included:  Youth 15 to 24 from both genders, with a special emphasis on the recruitment of females;  Non-completers of primary or secondary school system and including Tawjihi drop outs;  Unemployed/inactive, or employed in unsuitable jobs: a) job that does not have a contract of employment; (b) requires more than 40 hours per week, with no additional compensation; (c) operates in dangerous and or unhealthy working conditions; (d) does not meet minimum wage requirements; (e) has no specific job description; or (f) does not comply with the Jordanian Labor Laws); and  Youth residing within or in the near vicinity of Y4F targeted neighborhoods. Results Framework and Theory of Change 13 According to Y4F’s revised M&E plan updated in February 2012, the project’s long-term goal is to “Create an enabling environment with a greater capacity to more effectively serve youth-at-risk.” This goal is intended to be realized through three intermediate results (IRs): IR1 Improved Youth Models, Practices and Policies; IR2 Capacity of CBO’s to provide access to sustainable livelihoods for youth strengthened; and, IR3 Improved provision of youth friendly services and expanded civic engagement of youth to improve their local communities. The Results Framework is shown below graphically providing the overall vision of how program activities should lead to expected results. The framework illustrates the connection between program activities and the achievement of short-term outcomes, long-term outcomes and ultimately to overall program impact. 14 15 EVALUATION METHODS AND LIMITATIONS METHODS The evaluation used a mixed methods approach, combining quantitative and qualitative data to measure outcomes and to explain issues underlying those outcomes. Quantitative Methods: A phone survey was conducted with a randomly selected sample of 352 youth participants who completed Y4F training programs. The sampling strategy was based on random sampling for a 95 percent confidence level and +/-5 confidence interval. It should be noted that the sample was not selected to provide any power for statistical analysis below the top level, e.g., comparing male to female outcomes. The sample was randomly selected from the Y4F database of completers for Phase II4 , which contains a population of 3,873, with a resultant sample of 350. The initial strategy planned for a stratified sample for the different program training interventions: PTS only, PTS and technical skills, PTS, technical skills and other types of trainings such as IT and Business English. This stratification proved difficult to implement due to the information structure of Y4F database. Hence, the selection strategy was revised to a simple random sample (rather than stratification) with the possibility to add a booster sample in case representation of the various strata was poor. An explanation of the sampling strategy is contained in Annex V. The evaluation reached 352 youth through phone surveys (32 percent female and 68 percent male), and focus group discussions with an additional 91 youth (83 of whom responded to the same survey) from age 17 to 27;5 geographic coverage from all nine communities, and participants from all training programs (PTS, technical, business English, IT and others). As a result, it was determined that a booster sample was not needed. The evaluation also conducted a phone survey with employers of Y4F participants however this group proved challenging to reach; employer contact information was requested from youth, yielding 22 successful surveys. Responses from this group were too scattered and the sample too small6 to derive statistically valid conclusions although some of the information obtained is included in the evaluation. Phone surveys of both groups were conducted from October 16 through 22, 2014. Qualitative Methods: Qualitative data was drawn from: a) Key Informant Interviews (KIIs) with public sector partners (central and local): Ministry of Labor (MoL), Ministry of Social Development (MoSD), Higher Council of Youth (HCY) and Vocational Training Corporation and Centers (VTCs); b) Group interviews with CBOs and Y4F grant project staff; and c) Focus group discussions with youth and parents resulting in the following number of interviews:  Key Informants Interviews: 34 4 Per guidance from USAID, the total universe from which the random sample was identified included only the youth beneficiaries of Phase II as reflected in the Y4F database at 3,873 youth. 5 Phase II of Y4F began in 2011 and targeted youth 15-24. The randomly selected sample resulted in respondents 17-27, which may be attributable to them completing the program over the past two years. 6 Of the 167 employed youth, contact information was obtained for 52. Of those, 22 were matched with database and contacted. A full description of the approach is contained in Annex VII Sampling Plan. 16  Focus Group Discussions: nine FGDs with 91 youth (83 completed the same survey as that conducted by phone); twelve FGDs with 99 parents.  Group Interviews: 13 CBO management/staff and 42 Y4F staff.  Observation of youth spaces: In conducting interviews, the evaluation team had the opportunity to observe CBO youth friendly spaces, MoL Abdali OSS, and Higher Council of Youth Creativity Centers in 17 locations.  Desk review: Y4F project documents and reports. Detailed evaluation design, data analysis approach and evaluation tools are included in the Evaluation Design Report in Annex II. Upon request from USAID or closure of MESP, both electronic and hard￾copy data files will be transferred to USAID. In the meantime, electronic files are on the MESP file server and hard copies are warehoused with MESP. Prior to conducting the evaluation, all evaluation team members signed forms indicating that they had no conflicts of interest related to the evaluation; these forms are on file with MSI’s home office and are available upon request. LIMITATIONS Whether for qualitative data or quantitative survey results, the evaluation’s main source of information is based on self-reported feedback of project participants and beneficiaries. It is important to mention that the evaluation was not designed to validate reported figures from Y4F, especially in terms of ‘actual’ return to school or enrolling in the labor force. Hence, the reported statistical information should be considered as potentially indicative of project outcomes rather than 100% accurate data on what was achieved. Further, the data, while considered valid for the surveyed group, can not necessarily be extended to the youth who could not be located. 17 FINDINGS, CONCLUSIONS & RECOMMENDATIONS YOUTH OUTCOMES FINDINGS The Y4F Expansion Roadmap presumed that “by creating an enabling environment of CBOs and community partners that support and deliver best practice and sustainable youth programming, out of school and unemployed youth would be more likely to get a job, keep a job or go back to school.” Evaluation Questions: As a result of Y4F efforts, are youth in a better position now than they were prior to the project in regards to: a. Re-entering the formal7 education system then remaining8 in the formal education system? b. Entering the labor market then remaining in the labor market? c. Demonstrating positive life skills as defined by the Y4F training curriculum on positive life skills? The evaluation solicited feedback of youth participating in Y4F interventions through a phone survey and validated responses through focus group discussions with parents and CBOs’ management and project staff with results displayed in the above chart. The random sample of phone survey respondents closely reflected male/female participation of participants in Phase II of Y4F: Y4F has 64 percent male and 36 percent female; the survey respondents were 68 percent male and 32 percent female. 7 Formal (until Tawjihi level) and informal education such as enrolling in a vocational skills training. 8 With the intention of attaining Tawjihi degree or a vocational training certificate. 9 139 8 83 9 19 1 84 0 20 40 60 80 100 120 140 160 In education program Working Studying and working Not studying or working Youth Status by Gender (n=352) Male Female 18 4 3 1 8 1 0 17 11 0 6 11 10 5 4 1 14 2 1 27 0 5 10 15 20 25 30 Currently enrolled in education (n = 27) Male Female Total 4 5 1 7 17 2 1 2 5 10 66 3 12 27 0 5 10 15 20 25 30 Less than a month - this school year 1- 3 months 3- 6 months More than year Total Return to Education as a Result of Y4F (n = 27) Male Female Total 15 10 25 2 0 2 17 10 27 0 5 10 15 20 25 30 Male Female Total Intent to Continue Education (n=27) Yes I don't know / Not sure Total EDUCATION A survey with 352 youth randomly selected from Y4F Phase II participants yielded the following results. Twenty-seven youth (eight percent) stated that they are currently enrolled in an education program: 14 in universities; five in schools; four in vocational trainings; two in college; one internship and one “other.” Out of the 27 youth who are currently pursuing their education (less than ten percent of those sampled), 78 percent have been enrolled for more than three months, and the remaining six for less than a month or this school year. Fifteen survey respondents (55 percent of the youth who stated that they are currently enrolled) stated their resumption of education was inspired by their participation in Y4F, with the remaining twelve youth attributing their return to personal factors such as to augment their vocational/professional skills and to increase their employability. Of respondents that credited Y4F for their return to education, 14 respondents cited life skills training provided by Y4F as the project support that had the greatest influence on their decision to return to school. Twenty-five youth confirmed their intention to continuing their education and the remaining two were unsure. Although the information comes from a relatively small number of respondents, some reasons offered by CBOs, parents and youth themselves for not returning to school include financial need that encourages youth to pursue income generation (cited by four CBOs and seven youth), and early dropout age/Illiteracy. As some youth dropout at 9-10 years old, they are so behind their age cohort at 15 that they are very reluctant to return to school (cited by four CBOs, seven youth and two 19 Yes 29% No 71% Employment Secured as a Result of Y4F Support (n=167) parents). Furthermore, youth who have been out of school for three years or who have failed the Tawjihi three times are not allowed to return. According to Y4F management, the project never had activities designed with the specific purpose of encouraging or otherwise facilitating re-enrollment in the education system, much less remaining through completion. The project originally included an indicator tracking the number of Y4F “completers returning to the educational system;” this indicator was subsequently changed to “pursuing further education,” which could include vocational skills. This indicator was tied to the results framework Outcome 2.2, “Support for the preparation of youth to be more productively engaged in school and work increased.” Activities feeding into this indicator included life skills training, case management for “placement services and post-program contact” and building capacity of CBOs, yet none of the activities directly addressed re-enrollment in formal education. This indicator was dropped at the request of USAID in 2013 because data reported against this indicator did not reflect long-term attendance or completion of education; in Jordan a student may be considered “enrolled” with minimal attendance. Students are only considered as having dropped-out when they do not enroll in school the following year. The disconnect between the indicator and activities was noted by consultants who were hired by IYF to gain insight into the impact of Y4F among target groups. In their report, one consultant comments, “There is something that I feel is missing: we teach youth how to go to the local market but we do not prepare them if they want to go back to school. I feel there are things that we could provide.”9 EMPLOYMENT Among the 352 Y4F participants surveyed by phone, 47 percent are currently employed. In addition to life skills and vocational training, Y4F supported youth with individual case management, mentoring, career counseling and market research that were intended to help youth find and maintain employment. Yet in response to the question, “did you find your job as a result of the support and training provided by Y4F,” only 29 percent responded affirmatively, while 71 percent of respondents report that work was secured through other means. Thirty-six percent of respondents said they were primarily responsible for securing their own employment (with males and females reporting in similar percentages), which could in part be attributable to their improved life skills (although the specific question was not asked). Y4F provided youth with a range of services to help them transition to work. The Y4F interventions cited most in contributing to youth securing employment were life skills training (33 percent) and vocational training (26 percent), followed at a distance by Business English and computer skills, both at less than five percent.10 It is interesting to note that 45 percent of female participants (although a small number) attribute success in their job search to life skills training. 9 Rania Sabbah & Maisa Thweib, focus group discussions with Y4F participants and stakeholders, November 2012 – April 2013, p. 30. 10 49 respondents to the phone survey acknowledged Y4F as being influential in their job search. 20 4% 10% 6% 10% 47% 51% 0% 10% 20% 30% 40% 50% 60% Type of Y4F Support Most Influential on Job Search (n=49) The situation of youth in Jordan abandoning employment was raised as an issue by Y4F management and is acknowledged by USAID, as evidenced by the evaluation question that includes the condition of maintaining a job. In fact, a Jordan Times article reports that 36 percent of the 32,000 Jordanians who secured private sector jobs through the National Employment Campaign have quit their jobs,11 citing low wages, interference with studies, and transportation challenges. Responses among youth surveyed mirror this figure, with 35 percent reporting that they intend or want to leave their current job,12 and 55 percent confirming their intention to remain. Focus group results further confirm the data; the most commonly cited reasons for youth wanting to leave are poor working conditions, low salary, and distance from work as. During group discussions, participants expressed higher expectations for the type of job and salary than what resulted from the Y4F program. One participant explained that the ceremony and robes that were worn during “graduation” ceremony led to higher expectations than the job she was offered of cleaning toilets in a hotel. According to Y4F, CBOs, and public sector institutions, sectors were selected for vocational training based upon the Jordan National Employment Strategy 2011-2020 that identifies hospitality, retail and Heating, Ventilation and Air Conditioning (HVAC) as growth industries at the national level. While some participants live in urban areas that reflect employment demand in the industries identified as growth sectors at the national level, some participants reside in rural areas where work opportunities are limited to agriculture, or where job opportunities in the national sectors are too far from home to be practical. For example both Jordan Valley and Mafraq have virtually no job opportunities in the national sectors; securing employment in these locations would require relocation of residence. Participants, parents and CBOs 11 http://jordantimes.com/36-of-jordanians-secured-work-under-employment-campaign-quit-their-jobs 12 According to VTCs interviewed, it is common practice for hospitality sector employers to recruit VTC graduates for a six-month period as “trainees,” terminate employment at the end of the six-month period, and hire another cohort for six months in order to avoid paying employee benefits. 21 cite the implications of this as high cost of transportation or the requirement for youth to relocate to an urban area. Among the 167 employed Y4F participants who were surveyed, 38 percent are employed in hospitality and 25 percent in retail. The next highest sectors of employment include services at ten percent, industry at seven percent, and construction services at six percent. The issue of training in locally relevant skills was addressed inconsistently by Y4F. During interviews, a CBO in Rusiefeh said that they attempted to secure training for 20 youth in graphic design in response to three local employers agreeing to hire those trained, and a group of 30 youth requested training in hair salon services. Both requests were rejected by Y4F with the justification that such skills were irrelevant in the labor market. In another case, non-sector training was accommodated for an individual to be trained in photography and videography in Khreibet al Souq. The high-growth employment sectors identified in the National Employment Strategy have gender implications. During group discussions, parents, CBOs and youth point to hospitality and retail as inappropriate for females in conservative communities, given the mixed gender environments, late working hours that keep them from home until after dark, and often extensive travel time. The HVAC sector was sometimes described as “impossible” for females. Among employed youth surveyed, twelve percent (20) are female, with five employed in retail and four employed in tourism. Youth indicating increased positive life skills as a result of Positive Life Skills training. Among 321 participants who completed Y4F life skills training, 94 percent reported that they benefitted and 86 percent claim that it has been useful/very useful. ‘Passport to Success,’ the life skills training course branded by IYF and provided by Y4F is the foundation of the Y4F transition to work model, required for all participants and as a 24% 40% 8% 1% 6% 3% 1% 0% 8% 3% 5% 1% 25% 20% 20% 15% 5% 5% 5% 5% 0% 0% 0% 0% Sales / Retail Tourism Service Education Construction Government/ Public Sector Financial Media Industry Maintenance Military Oil and gas Sector of Employment (n=167) Male Female 22 prerequisite to vocational skills training. Among the 17 life skills dimensions youth were asked to identify as having a positive impact on their lives, over 90 percent cite eleven dimensions including self￾confidence, an increased sense of responsibility, assertiveness, commitment, motivation, conflict management skills, communication skills and better relationships with parents. The lowest scores, “relationship with employers” and “relationship with colleagues,” is attributed to the high percentage that was not employed. The difference in responses for these between males and females is likely due to the significantly lower number of females employed. Focus group discussions with parents, CBOs and youth confirmed and further elaborated on the survey results by highlighting the positive impact of life skills training and volunteer programs on improving youth confidence and the perception of self-worth. “Now I know that I can change my life,” reported one youth in Khreibet El Souk. In 2012, Y4F became aware that youth who had attended life skills training were experiencing increased conflict within their families. In response, Y4F created a Parent Engagement training component that entailed a two-day workshop for parents for the purpose of “enhancing their understanding of the physical, psychological, social and emotional development of young people.” According to Y4F, 42 facilitators within CBOs and “other Y4F partners” were trained to conduct these workshops that were attended by 2,000 parents.13 In an attempt to evaluate the effectiveness of this approach, parents were asked during focus groups if they had attended these workshops. A few parents reported that they had attended a session that explained the Y4F program and life skills training, but none indicated that they attended the two-day Parent Engagement workshops. Table 1. Personality Dimensions Positively Impacted by Life Skills Training Male Female Relationships with Employers 61% 20% Relationships with Colleagues 91% 79% Productivity 85% 93% Constructive Thinking Skills 84% 95% Creativity 86% 94% Relationships with Community 90% 89% Motivation 90% 94% Conflict Management Skills 90% 96% Commitment Level 90% 94% Relationships with Parents 92% 94% Entrepreneurial Spirit 92% 95% Assertiveness 91% 97% Responsibility 93% 97% Adaptability to Change 93% 97% Quest for Self-improvement 93% 97% Communication Skills 94% 96% Self Confidence 94% 96% Total 210 109 13 Y4F internal report, “Improving Youth Employability Outcomes,” undated. 23 The survey results were confirmed during group discussions in which both youth and parents echoed the positive impact of life skills training, citing better relationships with family, girls demonstrating more assertiveness for rights, and boys presenting themselves to integrate better into society. A parent from Khreibet Al Souq observed that now their son communicates better with them and has expanded his relationships with friends, family and community. One female participant said that prior to the life skills training she was too shy to stop a bus or even to ask for change when making a purchase. A number of females in focus groups echoed this transformation in overcoming extreme shyness and expressing themselves in public. The training supervisor of the Marka Vocational Training Center (VTC) claimed that even employers noticed positive behavior changes in youth, observing that employers more often hired students who received life skills training than those who did not. CONCLUSIONS Evaluation Question 1: As a result of Y4F efforts, are youth in a better position now than they were prior to the project in regards to: a. Re-entering the formal education system then remaining in the formal education system? Given the almost unanimous agreement that life skills training had a positive effect on youth, it could be concluded that youth are indeed “in a better position now” in regards to their decision-making ability or agency relative to returning to school. However, given participants’ low socio-economic status, perceived lack of socio-economic mobility, illiteracy (for some), and lack of activities within Y4F designed specifically to facilitate re-enrollment, youth may well be exercising the most rational decision￾making by not re-enrolling, as income generation fulfills a more immediate need. In addition, with the Y4F target age cohort of 15-24, it is likely that re-enrollment would only appeal to those who dropped out after age 14, as younger drop-outs would have missed so much school that their low literacy and learning levels would create an impediment. Although there is a lower percentage of females than males re-enrolling, the lack of activities specifically directed toward re-enrolling students does not allow for gender-specific conclusions. b. Entering the labor market then remaining in the labor market? Given the synergy among life skills training, vocational skills training, and employment linkages provided by Y4F, it appears that the youth who participated in Y4F are “in a better position now” to enter the workforce, as confirmed by both youth and employers, particularly for work in the focus areas of the National Employment Strategy. Even the youth who were trained in sectors in which they are unlikely to work due to geographic or cultural inappropriateness are most likely in “a better position” to enter the workforce if they are able to transfer their skills to occupations that are suitable. However, they may not be in a better position to remain in the workforce; while they are equipped with better technical and soft skills than before Y4F, their high expectations for position and salary may have contributed to their leaving a job. The significantly lower percentage of females working, combined with gender norms relative to working hours and distance from home indicates that females benefitted less relative to entering the workforce. c. Demonstrating positive life skills as defined by the Y4F training curriculum on positive life skills? With youth reporting benefit, usefulness, and positive impact from life skills training, and both parents and employers acknowledging positive behavior change, it may be concluded that youth are demonstrating positive life skills. Nonetheless, the extent to which this intervention may have increased conflict within the family bears further investigation, as does the potentially exponential impact of engaging parents in the process. It appears that life skills training affected males by diminishing their 24 aggression in communicating with family members, and affected females by diminishing their extreme shyness in public. EFFECTIVENESS & SUSTAINABILITY OF CAPACITY BUILDING FINDINGS Y4F extended capacity building support to Community Based Organizations (CBOs) and public sector institutions including the Ministry of Labor (MoL), Higher Council for Youth (HCY) and the Vocational Training Corporation (VTC). Evaluation Question for CBOs: Do CBOs supported by the Y4F project have the capacity, as defined previously, to continue youth programming? Thirteen CBOs were provided grants to extend services to youth and to establish youth friendly spaces/centers such as computer labs, libraries, playgrounds and creativity areas. Additionally, CBOs were supported with training in financial management, strategic planning and proposal writing. CBOs received further capacity building on subject matters such as grant management, Monitoring and Evaluation (M&E) and Youth Friendly Services (YFS). Interviews were conducted with all 13 CBOs to assess capacity building and the likelihood that new management practices continue beyond Y4F. The majority of CBOs claim that they are using the financial and management procedures developed and adopted as a result of Y4F assistance. For example, the Family and Child Protection CBO reported that their other donors are impressed with the quality of their current financial reporting. Several CBOs specified that grant management procedures such as procurement (three quotations, open selection of bids, documenting transactions) have been institutionalized as it will allow them to access additional funding beyond Y4F. While four CBOs claim to have adopted monitoring practices and beneficiary record keeping, most felt that the requirements for tracking participant status on a monthly basis were too onerous, and simply stopped collecting data and reporting on it. Another M&E practice requested by Y4F was a youth attitude survey that was intended to be conducted with each participant at inception and at exit. According to Y4F staff, responses to the attitude questions showed a certain level of acquiescence bias; youth tended to rate themselves higher on their attitudes at the beginning of the program than at exit. Y4F staff surmise that the respondents probably did not have the capacity for self-awareness or self-reflection to be able to accurately reflect their attitudes, and could also have responded more positively because they believed this would help their chances of being accepted into the program. When the benefits of adopting and integrating new practices and procedures were obvious to the CBOs, they were adopted, adapted and institutionalized; otherwise they were discontinued. Based on interviews with CBOs, the extent of knowledge acquired through Y4F and the extent to which practices are integrated are dependent upon CBOs’ starting point with management systems; more established CBOs such as That Al Nitakayn and Iskan Talal Association reported lesser learning, yet better integration of grant management procedures. Prior to working with Y4F, many CBOs operated at the community-level on a shoestring budget without paid staff or formal management procedures. Several had never conducted youth-focused services before. With funding from Y4F, CBOs recruited staff specifically to work on Y4F activities. In fact, some CBOs reported that Y4F managed the recruitment process for staff. As a result, the majority of training and capacity building was directed to CBO program staff including project managers, accountants, M&E staff, and youth mentors who were hired specifically to work on Y4F and were paid with Y4F funds. At the closure of the Y4F grant, most CBOs had no funding to pay staff that had been hired with Y4F funding; staff and the knowledge they acquired left when Y4F funding ended. 25 Y4F’s training and mentoring in proposal-writing has been integrated into more than half of the CBOs, demonstrated by eight CBOs submitting proposals for funding from other grant making organizations. One CBO reported being awarded a grant from FHI 360. Beyond building management capacity, several CBOs point to an unanticipated positive outcome from their relationship with Y4F. Through the process of interacting with Ministries, VTCs, and employers, several CBOs observe that they enjoy more recognition and a higher status in the community than before Y4F. Nine of the 13 CBOs report that as a result of Y4F they have expanded their mandate to include youth￾focused services and continue efforts to access funding for youth services. Prior to involvement with Y4F, most of these CBOs’ activities focused on families: females and children. As such, females had more familiarity with the CBOs than males. Among the 352 youth surveyed 22 percent report having used/benefited from the youth spaces provided by CBOs, with computer labs having the highest demand. All CBOs were offered computer equipment from Y4F, however the computers were donated by a private firm rather than purchased with project funds. Three CBOs reported that the computers were so old as to be almost useless; one CBO refused to accept the computers. Despite a lack of funding, some CBOs continue to offer youth-friendly services such as career counseling, mentoring, networking and linkages with employers using volunteers on an ad hoc basis. Although Y4F trained staff of CBOs to deliver life skills training, Y4F used a proprietary training package (Passport to Success) that may only be used by certified trainers. Y4F did not certify trainers within CBOs to continue using the package. Some CBOs such as the Jordanian Women Charity Association charge nominal fees for training workshops, while Khreibet Al Souk has other sources of donor funding to conduct computer and English language courses in facilities that were funded by Y4F. Evaluation Questions for Public Sector Institutions a. The impact of Y4F assistance on the type and quality of services provided by the public sector institution to youth. b. Which impact will be sustained on the long term and why? c. Future recommendations for continuing involvement with public sector institutions on youth development. Y4F extended capacity building support to the Ministry of Labor, Higher Council for Youth, and the Vocational Training Corporation with the objective of strengthening these organizations’ capacity to effectively work as youth service providers. The type and depth of project assistance varied across 26 institutions ranging from technical assistance and staff training to facility rehabilitation, furnishing and equipment. The Ministry of Labor reports that Y4F assisted some of their Employment Directorates in Abdali and Irbid to become One Stop Shops in the following ways:  Integrating career guidance services to youth seeking employment.  Providing Training of Trainers to certify staff in delivering the IYF proprietary life skills training to youth seeking the centers’ services.  Expanding services available through the One Stop Shop to include the NET (National Employment and Training), DEF (Development and Employment Fund), and VTC (Vocational Training Center). Nonetheless, no representatives of these offices were present during the evaluation team visit.  Increasing the number of youth/job seekers through One Stop Shop employment and ancillary services. Abdali OSS reports that its image has been enhanced as more user friendly as a result of Y4F assistance for facility rehabilitation, equipment, and furniture.  Increasing coordination, networking and referrals instituted formally through memoranda of understanding and informally through a referral system among MoL employment offices, CBOs, municipalities and the private sector to refer youth to the appropriate service and organization. Among the 352 youth surveyed, one-third (32 percent) reported that they had used the services of the MoL employment office for help in finding a job. When the 67 percent of youth that reported that they have not used MoL services were asked why, more than one-third (35 percent) reported that they did not know about it, nearly one-quarter (24 percent) stated that they had already found a job and nine percent explained that they were studying or wanted to complete their education first. As the MoL employment offices are located in the administrative centers of each governorate, youth living outside these city centers would not be likely to know about them or avail themselves of their services. Of the 114 youth who had used the MoL employment office, 33 percent stated that they found the services useful or very useful while 51 percent felt that they were not useful. During visits to the two OSS offices it appeared that males and females were accessing services in approximately equal numbers; MoL employment offices were not visited for comparison of male/female usage. Not surprisingly, youth perceptions of the utility of the MoL employment office appears to be directly related to success in finding good work through the office. Among the 58 youth who reported their experience with this office was not useful, three-quarters (74 percent) explained that the office did not help them find a job and nine percent viewed the job found by MoL as unsuitable. For youth reporting that employment office assistance was useful or very useful, 22 percent explained that the office had found them a job. According to the Vocational Training Corporation (VTC) and Training Centers, Y4F assistance has helped them to integrate their life skills training with IYF’s proprietary life skills training package, Passport to Success (PTS) and making it a compulsory part of the VTCs’ hospitality training curricula. The VTC hospitality curricula had previously included some life skills training but not of the extent and depth of PTS. Although life skills training is still limited to VTC’s twelve hospitality centers and three centers of excellence, it has committed to incorporating life skills training in all of its vocational skills training programs. VTC has informally established a Change Management Unit (CMU) to ensure integration of life skills training into its programs and to prepare certified PTS trainers in cooperation with IYF. Expansion of PTS to all VTC centers is well underway with plans to certify additional trainers in November 2014 and to secure formal approval of the CMU. Most courses offered by VTCs are for vocations in which females typically do not work, and are generally considered culturally “inappropriate” for females, such as vehicle maintenance and repair, 27 commercial driver training, and electronics. VTCs offer some courses specifically targeting females, such as salon services, sewing and handicrafts. The Higher Council for Youth (HCY) credits Y4F with assistance in developing a Youth Leadership Training Program, including a Youth Leadership training manual and training for staff on conducting youth leadership workshops. Y4F covered the first phase of the training and HCY will continue to implement the remaining two phases either from HCY’s own budget resources or with other donor assistance. Y4F also supported three youth centers to become creativity centers equipped with workshops for craft production, robotics and computers labs; however, the centers do not have trainers due to lack of funds. The centers are open during business hours and close at a time when youth would be most likely to use the center. Part of the Y4F creativity center model was to fund a local NGO to provide training in handicraft production that could be sold through HCY centers. Unfortunately, youth were not trained in entrepreneurship and were disappointed to discover that the youth centers are prohibited by law from conducting sales activities, given that they are public institutions. Focus group discussions conducted through CBOs explored the areas of youth demand, access and perception of HCY centers. The majority of youth consulted were not aware of/did not know about HCY youth centers. One youth interviewed at the That Al Nitakayn CBO said, “I know that there is a youth center but it is always closed and no activities are taking place. When you enter the premises they ask you, ‘what are you coming here for?’” CONCLUSIONS Evaluation Question 2: How effective has Y4F been at building organizational capacity (financial viability, staffing, skills and knowledge needed to run the organization and provide intended services to youth, autonomous of any external assistance) among local CBOs and public sector stakeholders involved in the process? Effectiveness of Y4F capacity building among CBOs and partners appears to have been effective in service provision; management capacity building may have been effective on an individual level but less effective on an institution/organizational level. While most CBOs claim to have integrated institutional and operational systems, the extent to which they have is dependent upon their size and level of operations. Those that are more established and have funding streams external to Y4F appear to have integrated systems more fully, while the smaller CBOs without funding sources have little activity and therefore, minimal opportunity to use the systems. For the small CBOs, most staff trained in management processes and procedures were hired expressly for and paid by Y4F. As Y4F funding ended, staff left. CBOs that have other sources of funding are likely to maintain use of the management practices that Y4F introduced. Y4F was effective in integrating life skills training into the Vocational Training Corporation making it mandatory in VTC hospitality curricula and certifying VTC staff as trainers of Y4F’s proprietary PTS life skills training package. Regardless of their effectiveness in providing employment services, lack of awareness and negative public perception render MoL employment offices under-utilized by youth. OSS centers demonstrate that public perception of MoL can be positive if the facility is in good physical condition and well-equipped. Sustainability of the OSS model and newly-introduced services in Abdali and Irbid employment centers are likely to be sustained as services and systems have been integrated into the work flow, providing expanded youth services and networking with the private sector through job fairs and referral to potential employers. 28 Sustainability of the HCY Youth Leadership Training program is likely as it was originally part of the HCY’s plans. Evaluation Question 3: Do CBOs supported by the Y4F project have the capacity to continue youth programming? While CBOs have demonstrated capacity to continue youth programming, doing so is ultimately dependent upon funding. Nonetheless, motivation exits as demonstrated through some CBOs charging nominal fees for workshops, some tapping into funding streams through other donors, and some providing services on an ad-hoc basis through volunteers. Evaluation Question 4: Are there other organizations such as vocational training colleges (VTCs) also strengthened by Y4F that USAID should continue to work with on positive youth development? Given the expansion of life skills training throughout VTC curricula, combined with VTCs’ linkages with employers in high-growth sectors, VTCs could offer enormous potential for youth in areas where job opportunities are available in the sectors in which VTCs train. As the HCY Youth Leadership program is in its nascent stage, an assessment of its potential is unknown at this time. MoL/OSS could be effective in supporting youth transition to work if a holistic package of technical assistance, facilities upgrading and a campaign addressing public perception were offered. Any one form of assistance on its own would likely not result in substantial increase in service demand by youth. LEARNING /CORE PROGRAM MODELS FINDINGS AND CONCLUSIONS Findings from Youth Outcomes and Capacity Building are aggregated to address the following two questions, which by their design necessitate conclusions. Evaluation Question 5: Were the two core program models “successful transitions to work” and “sustainable community youth hubs” appropriate models for achieving the project’s stated results? While the model of Transition to Work was appropriate, some aspects of implementation may have limited achievement of the project results. The synergy of life skills training and vocational training appears to be critical in youth transitioning to work. Although some youth singled out the usefulness of employer linkages, half of those surveyed that are employed report that they secured employment with their individual effort. This finding is interesting in itself, as they may not have been as able to secure employment prior to participating in life skills training. Another notable aspect of the Y4F Transition to Work model is the inclusion of CV writing, self-presentation and job interview skills, which stakeholders said was particularly important. Nonetheless, the proprietary nature of IYF’s PTS life skills training requires certification of trainers, which was not done with CBOs (but which IYF is doing with VTCs under a separate agreement). Unfortunately, the pre-requisite for youth to complete life skills training before entering technical training resulted in some attrition of participants who had to wait for availability in either course. This problem was resolved in some of the training colleges such as Al Quds and Hospitality VTC that offered life skills training concurrent with vocational training. While technical training through recognized colleges added a level of prestige, included youth who could not have afforded attendance, and linked participants to jobs, the narrow focus on national priority sectors of industry was not effective for females and for those living in areas without employment in those sectors. Despite the attractiveness of attending these colleges, some youth interviewed felt that 29 the three months of theoretical training followed by three months of on-the-job training was insufficient to increase their attractiveness to employers. Relative to Community Youth Hubs, while the model is appropriate as a place for youth to access holistic services, each institution has unique operational approaches and constraints that determine their effectiveness. For example, CBOs are able to conduct outreach to and mobilization of youth and its location in the community encourages attendance; VTCs have a good reputation for technical training and employer linkages; and One Stop Shops have particular competence in job placement. Evaluation Question 6: Are there strategies or practices IYF implemented that should be replicated or avoided in future youth programming? Practices to Replicate: 1. Requiring life skills training in conjunction with technical training. 2. Offering technical training through recognized institutions. Practices to Avoid: 1. Extended waiting periods for youth who are eager to join courses. 2. Inflating expectations about the level of employment that participants will be offered. While some youth may have unrealistically high expectations, attending well-known training institutions and graduation ceremonies with gowns may serve to inflate expectations. 3. Offering technical training only in sectors identified as high-growth but without job opportunities near participants’ homes. 4. Offering only technical skills that communities recognize as inappropriate for females, or technical skills considered appropriate but not marketable. GENDER AND CULTURAL CHALLENGES FINDINGS Y4F encountered challenges to engage equal numbers males and females, and experienced significant attrition of females as program engagement graduated from life skills training to technical training to additional youth services for English language and computer skills. This was particularly pronounced in conservative areas. Evaluation Question 7: How did Y4F address any gender differential constraints (accessing, participating in, or benefitting from) in project activities? The sample phone survey of 352 youth participants (32 percent females and 68 percent males) reveal 27 percent (ten) females enrolled in an educational program. Of the 167 youth who report being employed twelve percent (20) are female. Among the 20 females who are employed, seven live in Amman. 31 32 35 112 27 41 16 25 0 20 40 60 80 100 120 PTS PTS + Other Only PTS + technical training only PTS + Technical Training + Other Training and Other Assistance Received by Gender (n=319) 30 As a result of discussions on gender during focus groups, the following themes arose that pointed to gender inequities in decision-making within families:  Cultural norms in conservative areas restrict females from working in gender mixed contexts such as restaurants, hotels and malls, working in the evening and working outside their immediate area. The technical training offered by Y4F was almost exclusively in hospitality and retail.  Although families were comfortable with girls attending training in life skills, they were often prohibited from working.  In conservative communities, girls’ motivation to work is limited by cultural norms that encourage marriage as an alternative to work. In addition to cultural norms restricting the environments in which females can work, Y4F encountered challenges in mixed gender trainings and while transporting youth to training locations. Y4F addressed these challenges by segregating transport and training sessions. Sanabel Al Khair CBO noted, "We faced a problem at the beginning when trainings were mixed; afterwards everyone was comfortable as trainings were gender segregated.” Youth and parents reported in group discussions that on the few occasions when Y4F offered vocational training in skill areas that are “appropriate” for females such as crafts,14 hair dressing and make-up, some female trainees were able to work from home and thus generate income. Y4F claims that it attempted to address parents’ limitations on female participation through the Parent Engagement workshops previously described, however, no information is available to assess the effectiveness of these workshops relative to loosening cultural impediments to female employment. At the local level, CBOs addressed gender issues by allocating separate and different time slots in Youth Friendly Spaces for males and females. When confronted by parents’ resistance to allow females to participate, CBO leadership visited the families, met with parents, and explained the project aim and context in order to secure parents’ consent. The USAID Policy on Gender Equality and Female Empowerment (2012) requires that gender equity be reflected in project design, procurement processes, performance indicators and project evaluations. The USAID Automated Directive Assistance (ADS) 201, 202, 203 and 205 provide further guidance on integrating gender equality and female empowerment into the program life cycle, including gender analyses to inform program design, and gender-sensitive activity design, implementation, indicators, and budget. As Y4F was conceived prior to this policy, its design and implementation were not in alignment with these guidelines. CONCLUSIONS Evaluation Question 7: How did Y4F address any gender differential constraints (accessing, participating in, or benefitting from) in project activities? Although the initial Y4F design does not appear to consider gender constraints, Y4F was responsive in addressing some gender issues as they surfaced. It may be concluded that the steps taken to segregate males and females in training and transportation, and meetings with parents who were resistant to females participating increased female participation, the efficacy of these actions are not specifically documented. Activities focusing on vocational training are challenged with the seemingly mutually 14 This training was offered through HCY. 31 exclusive objectives of providing training in vocations that are both marketable and considered culturally appropriate for females, primarily in terms of proximity to home and hours. RECOMMENDATIONS 1. Activities and indicators should be designed for alignment with objectives; if an objective is to increase re-enrollment, activities should be designed accordingly. 2. Changes to project objectives should be clearly documented in writing in the revised AMEP or contract modification (if applicable). 3. Activities addressing re-enrollment in education should target younger cohorts (ten years old) in order to mitigate the impediment of illiteracy and the increasing reluctance to re-enroll, as well as exclusion of youth who have been out of school for three years. 4. Activities addressing re-enrollment in education should acknowledge and address the issues of financial need to leave school (or not re-enroll), and the perceived lack of value for education to enable socio-economic mobility. 5. Make life skills training an integral component of youth programming. Offer life skills training for females that is not necessarily linked to the objective of employment or education, but rather with the objective of overcoming shyness and increasing their voice and agency. 6. Include life skills training for parents to avoid backlash and to create a more enabling and supportive environment for youth development. 7. Facilitate all partners who are training youth to include life skills training as part of their offering. Consider tailoring life skills to categories of objectives such as employment, education, and parenting. 8. Require that implementing partners who are using life skills training funded by USAID to use curricula and approaches that are non-proprietary so as to facilitate broad and sustained use of the curricula and approach. 9. Offer options for technical skills training that include both high-growth sectors (in geographic locations where such jobs exist) and skills training that is more aligned with job opportunities in the local market. 10. Conduct a participatory assessment to determine the types of jobs that would be considered by youth, families and communities as appropriate for females. Engage youth, family, community members, employers and training providers to identify mechanisms that could gently challenge gender norms and make non-traditional occupations available to females. Such mechanisms might include incentives for employers for areas adjusting work hours and providing transportation for females; and inviting family members to places of employment to assure them of safe environment for females. 11. Continue to support CBOs and build their capacity to provide youth services, particularly youth spaces that may be more likely to attract female participants who are not on a career-track per se. Such spaces should have low maintenance cost for sustainability, and offer more recreational than educational services to draw larger numbers of youth through sports and team competitions, social and educational games. 12. Support the Ministry of Labor in replicating the Abdali One Stop Shop Model in other MoL employment directorates with the provision of life skills training, employment counseling to introduce options and manage expectations, and facilitating linkages with employers through hosting employer Q&A sessions, job fairs, and other mechanisms. 13. Concurrent with supporting public institutions to improve services and upgrade facilities, support MoL in changing its public perception to enhance youth demand for their services. 32 14. Continue to support the Vocational Training Centers and other institutions and programs that offer a combination of life skills training and vocational training. 15. Support the VTCs in identifying vocational training for females that is both marketable and considered culturally appropriate. 16. Determine whether three months of theoretical and three months of on-the-job training is sufficient to increase employability from the perspective of both employers and participants. Evaluate the duration of the training program through the lens of youth segmentation/sub-groups. 17. Determine effectiveness of career counselors in assisting participants in securing employment. 18. Implementing Partners should work with providers of technical training and life skills training in locations that are easily reached by the target population. 19. USAID should consider including in project designs rigorous M&E systems that enable long-term evaluation of components of similar youth programs; the recommended M&E system would enable tracing the trajectory of organizations and individuals over a period of time and a deeper evaluation of the efficacy of individual components and sub-grantees. Consider a follow-up study at an interval of one or two years to provide information on long-term sustainability. 20. USAID should support the Government of Jordan to explore opportunities for VTCs to bring training (in vocations that are locally relevant) to population centers where VTCs do not exist. This off-campus training could be conducted in youth centers, CBOs or OSS facilities. 21. Expand technical training to include employment opportunities at the local level and employment opportunities that are culturally feasible for females. 22. Implementing partners should take measures to pro-actively manage youth expectations for employment salary and working conditions through career counseling, work site visits, and meeting youth who are already working in the sector. 23. Engage partners according to their areas of strength, e.g., CBOs for outreach to and mobilization of youth and youth friendly spaces (particularly in attracting females); VTCs for technical training, employer linkages; and One Stop Shops to assist with job placement. IPs should facilitate collaboration among partners to acknowledge their core competencies and leverage each partner’s strengths as a condition of assistance to minimize competition for resources. 24. Require implementing partners that are using life skills training funded by USAID to use curricula and approaches that are non-proprietary so as to facilitate broad and sustained use of the curricula and approach. 25. Offer life skills training to youth (and particularly females) who do not necessarily have educational or employment objectives. 26. USAID should work with the Government of Jordan to implement public perception/behavior change communication targeting male family members as they are often the decision-makers for females, and community and religious leaders who often “police” behavior of families who are considered permissive by allowing females to work. 27. Require a gender analysis of female transition to work in order to identify employment opportunities that are culturally acceptable, and therefore accessible to females. The gender analysis should also identify approaches to increase female participation in other activities offered through youth centers in order to ensure equitable participation in and benefit from USAID resources. 28. Design future activities addressing youth to be in alignment with USAID Gender Equality and Female Empowerment policies for gender-sensitive design, management, implementation, indicators and budget. This may require challenging social norms and should be done in close consultation with females, families, community members and employers to ensure alignment with the U.S. “do no harm” policy for foreign assistance. 29. Require the IP to align technical training offerings with job skills that are marketable and that are accessible to females, given the cultural impediments in their region. 33 ANNEXES Annex I: Evaluation Statement of Work Annex II: Evaluation Design Report Annex III: Evaluation Tools Annex IV: Sources of Information Annex V: Data Sampling Methodology Annex VI: Youth Survey Data Report 34 ANNEX I: EVALUATION STATEMENT OF WORK Youth for the Future (Y4F) Performance Evaluation DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK I. INTRODUCTION USAID/Jordan requires an external final performance evaluation of the USAID/Jordan Youth for the Future (Y4F) Project. The purpose of the evaluation is to provide USAID/Jordan with overall feedback and strategic recommendations for future programming related to youth development, including capacity building and sustainability approaches for youth serving institutions, particularly CBOs. The evaluation will also help USAID/Jordan obtain the necessary information to more effectively allocate resources for future programming. Specifically, the external evaluation will use qualitative and quantitative data to the extent possible, as well as identify lessons learned, to evaluate Y4F’s approach to youth programming, capacity building and sustainability of local partner organizations. Based on the findings, the evaluation will make recommendations on approaches and challenges to be addressed in potential future projects targeting disadvantaged youth. Details of the project to be evaluated: Project Title: USAID – Cooperative Agreement Youth: Work Jordan (amended to Youth for the Future (Y4F)) Implementing Partner: International Youth Foundation (IYF) Total Cost: $30,000,000 (amended to $33,352,223 on May 11, 2011) Duration: March 22, 2009 to March 21, 2014 (amended to December 21, 2014 on January 21, 2014) II. BACKGROUND A. PROJECT DESCRIPTION Youth: Work Jordan (YWJ), later changed to, and henceforth in this document referred to as, Youth for the Future (Y4F), began in 2009 with the primary goal of delivering comprehensive, community-based activities for at risk youth ages 15-24. Phase One (March 2009 - June 2011) The Y4F project was designed to mobilize and empower key actors at the local and national levels to work together in harnessing the productivity and potential of Jordan’s youth. The primary components of the project were as follows: • Provision of life, employability, and entrepreneurship skills for the 21st century to youth; • Provision of youth-friendly services; 35 • Improvement of community infrastructure/environment; and, • Improvement of youth-related policy development processes and coordination Y4F developed six main strategies that cut across implementation of all four components of the program. They were: • Focus on Employment: Job preparedness for young people to put them on the pathway for successful long-term employment. • Fostering Coordination and Collaboration among Actors in Youth Employment: Creating a program management structure that facilitates coordination, collaboration, and synergy as well as action-oriented, alliance-based frameworks for responding to needs and challenges. • Building Local Ownership through Community Alliances: Nurturing alliances of actors at the community level to develop strategies to assist young people to get the services they need, prepare them for employment, and to ensure they find jobs. • Strengthening Capacity: Capacity building focused on three levels. o Working with the Ministry of Social Development (MOSD), Y4F worked to strengthen the capacity of central level ministerial staff as well as staff in the field directorates to develop, implement, monitor, and evaluate programs. o Increase the capacity of Coordination Non-Governmental Organizations (CNGOs) who serve as the focal point for local, community-level implementation. This included working to strengthen their financial and management operations as well as their ability to work with community level groups to increase their programmatic, financial and monitoring and evaluation abilities. o Improve the quality and relevance of projects at the community level implemented by Community-Based Organizations (CBOs) and government entities. Additionally, strengthen the capacity of youth as they learn to identify, plan, implement, communicate about, and monitor community projects. • Continuous Learning through Monitoring and Evaluation: Ensuring that all parties are clear about program expectations, benchmarks for success, and the rationale for decision making concerning the allocation of resources and identification of best practices. • Sustainability: Fostering ownership for program interventions in the relevant ministries and an Advisory Committee that will provide a mechanism for continued dialogue and collaboration beyond the life of the program. Y4F activities were initially focused on specific neighborhoods within six communities: Zarqa, Russeifeh, Irbid, Jordan Valley, Ma’an, and East Amman. These were selected in collaboration with USAID and the MOSD based on poverty, social factors, population density, and representation of urban and rural areas. Activities in each area were guided by participatory community appraisals to identify existing good practices, as well as needs and gaps in services. Initially, three national level Coordination NGOs (CNGOs) were competitively selected to oversee community activities through 22 CBO sub-grants. The CNGOs executed community action plans and built the capacity of local CBOs, who in turn implemented activities and proposed approaches that best address youth needs in the target areas. In addition, Y4F’s original operational model included the imbedding of 2-3 MOSD staff within Y4F’s program management unit, as well as the active engagement of MOSD field directorates, to increase collaboration with the MOSD and leverage local capacity. This structure was managed by the Y4F Program Director and Deputy Program Director and their supporting Jordanian and Baltimore based staff. 36 Mid-term Project Modifications A mid-term evaluation was conducted from May to July 2011 followed by an internal strategic review carried out in coordination with USAID in 2012 and a USAID audit completed in the first quarter of 2013.15 These three studies led to the following amendments to the project: • July 2011: IYF began piloting the provision of direct grants to CBOs in three additional communities (Mafraq, Sahab, and Jordan valley) while CNGO grants continued implementation in the original target communities. Once CNGO grants came to an end in December 2011, a decision was made not to extend the CNGO grants. Subsequently, it was decided that IYF should directly manage the work of CBOs in all Y4F neighborhoods in order to closely lead the capacity building activities. • May 2012: IYF’s Cooperative Agreement was modified and the program budget increased from $30 million to $33 million in order to expand public sector capacity building activities to other key youth serving ministries beyond the MOSD including the Ministry of Labor, the Vocational Training Corporation and the Higher Council of Youth. This consisted of training key public sector personnel working at the central and directorate levels on youth development models and best practices as well as supporting these ministries in undertaking direct activities and services enhancements in Y4F target neighborhoods. • September 2012: Following a strategic review with USAID of the program’s brand and priorities, IYF consolidated all of the program’s work streams and implementation activities under two core program models: Preparing youth for successful transitions for work and Building sustainable community youth hubs. • December 2012: Based on the strategic review, the branding strategy and marketing plan were changed in their entirety, including the project name which was changed to Youth for the Future (Y4F). • January 2013: Y4F underwent a project audit. As a result, the project components were streamlined through the removal of the civic engagement component and the entrepreneurship activities. Additionally, the project went through a verification of all of their M&E data. • January 2014: Y4F received a no cost extension in January 2014 in order to continue its scaling and sustainability activities, particularly with public sector partners. Phase 2 (July 2011 - December 2014) Per the amendments already noted, the civic engagement and entrepreneurship activity components of the project were discontinued. Additionally, the sub-granting mechanism through CNGOS was also discontinued. Y4F refocused its efforts to directly building local capacity to provide youth services as ensuring better youth outcomes from these services. The following two core program models were utilized to achieve its goal of “creating an enabling environment with a greater capacity to more effectively serve youth-at-risk and the subsequent intermediate results.” • Successful Transitions to Work: Prepare at-risk youth for transformative life skills and career development ; and 15 Youth for the Futures program audit was led by a team from the USAID mission in Cairo and was initially scheduled for 2011 but was subsequently delayed for several years due to the turmoil and unrest in Egypt. 37 • Sustainable Community Youth Hubs: Empower community partners to build and sustain a network of quality youth services and expand the cadre of qualified youth workers and community trainers. Additionally, during the second phase of implementation, targeted communities were expanded to include Mafraq, Sahab, and southern Shouneh. The geographical boundaries of the program’s original target neighborhoods were also expanded. B. Y4F RESULTS FRAMEWORK AND THEORY OF CHANGE C. PROJECT MODIFICATIONS III. PURPOSE This Performance Evaluation is intended to provide USAID/Jordan with overall feedback and strategic recommendations for positive youth development strategies and best practices for the capacity building and sustainability of youth serving institutions, particularly CBOs. The management purpose of the evaluation is to help USAID/Jordan obtain the necessary information to more effectively allocate resources for future programming. USAID/Jordan is aware of many of the successes and shortcomings of the Y4F project and thus, would like to focus on what it doesn’t already know. For this reason, this evaluation should focus primarily on PHASE 2 of the project, although it is assumed that capacity building initiatives and youth outcomes that were a result of Phase 1 activities will be included. IV. EVALUATION QUESTIONS Goal: Creating an enabling environment with a greater capacity to more effectively serve youth-at-risk IR 1: Improved Youth Models, Practices and Policies IR 2: Capacity of CBO’s to provide access to sustainable livelihoods for youth strengthened IR 3: Improved provision of youth friendly services and expanded civic engagement of youth 38 The consultants are expected to design and implement the appropriate approaches and tools needed to answer the following evaluation questions (presented in no particular order of importance)16: Youth Outcomes 1. As a result of Y4F efforts, are youth in a better position now than they were prior to the project in regards to: a. Re-entering the formal education system then remaining in the formal education system?* b. Entering the labor market then remaining in the labor market?* c. Demonstrating positive life skills as defined by the Y4F training curriculum on positive life skills?* Capacity Building 2. How effective has Y4F been at building organizational capacity (financial viability, staffing, skills and knowledge needed to run the organization and provide intended services to youth, autonomous of any external assistance, etc.) among local CBOs and public sector stakeholders involved in the project? 3. Do CBOs supported by the Y4F project have the capacity, as defined previously, to continue youth programming? 4. Are there other organizations, such as vocational training colleges (VTCs), also strengthened by Y4F that USAID should continue to work with on positive youth development? Learning 5. Were the two core program models (“successful transitions to work” and “sustainable community youth hubs”) appropriate models for achieving the projects stated results? 6. Are there strategies or practices IYF implemented that should be replicated or avoided in future youth programming? Gender 7. How did Y4F address any gender differential constraints (accessing, participating in, or benefitting from) in project activities? V. IMPLEMENTATION A. EVALUATION APPROACH The evaluation will focus primarily on two aspects of the Y4F project. The first is that of local capacity and sustainability of stakeholders involved in Y4F. The second is that of youth outcomes resulting from project activities, which will include following up on where youth beneficiaries are now compared to where they were at the beginning of the project, as well as the environment that facilitated or prevented these outcomes from being reached. This will require selecting a sample of youth beneficiaries, 16 Evaluation questions or sub-questions with an asterisk (*) following them require sex-disaggregated data to be included in the responses. 39 reviewing their file and activity interventions and where they are now. The consultants will decide on the suitable sampling method and size; however the sampling should take into consideration the evolution of the project including its several amendments, which affected the geographic scope and structure of stakeholder engagement. In addition, sampling will be conducted to reflect various youth outcomes, including re-entering the formal school system, entering the labor force, and youth exhibiting improved positive behaviors, including life skills. The Evaluation Team should use, as appropriate, both qualitative and quantitative data collection and analysis methodologies that address all evaluation questions. This may include, but is not limited to, document review, in-depth interviews, or surveys. The methodology will be developed and finalized by the Evaluation Team. In consultation with USAID, the Evaluation Team shall perform the following tasks: Preparation and Planning: ‐ Review all relevant information and additional materials for sufficient background information, and to better understand what data already exists that the Evaluation Team can build upon. ‐ The AOR will be the primary point of contact for obtaining project documents. ‐ Meet with USAID to discuss the evaluation work plan and methodology. ‐ Present an evaluation work plan to USAID for approval by the close of the 3rd working day after start of the evaluation. This shall include a list of tasks to be completed, the level of effort for each task, and the deliverables upon the completion of that task. ‐ Based on the desk review of the available information and in-brief with USAID, develop evaluation tools which may include interview guides, a survey, and/or field verification checklists for site visits. ‐ Present the detailed evaluation design report (methodology, work plan and tools) to USAID for approval within seven (7) working days of the start of the evaluation. Data Collection: ‐ Conduct a desk research and document review of all relevant project documents and secondary sources. Project documents available to the team will be provided by the AOR in collaboration with Y4F staff, and include:  Quarterly and Annual reports  Performance monitoring data including all versions of the M&E plan  Contract/ agreement with any modifications  Project Workplan  2011 Mid-term evaluation report  2013 Project Audit Report ‐ Conduct interviews with the appropriate staff of USAID/Jordan, the GOJ, IYF, stakeholders and beneficiaries of the project. USAID will provide the initial list of in￾country contacts. ‐ Perform field trips as needed to interview project beneficiaries and visit project sites. ‐ If deemed appropriate by the Evaluation Team, perform a survey of project beneficiaries and/or stakeholders. ‐ Any additional data collection efforts the Evaluation Team feels are appropriate and would like to propose given the scope and time constraints of the evaluation. While no specific data analysis methods are proposed, and this will determined by the Evaluation Team, USAID expects analytic methods to be chosen that are in line with evaluation best practices. 40 Reporting ‐ Provide periodic updates to USAID on progress of the evaluation ‐ Conduct a debriefing presentation for USAID on initial evaluation findings, conclusions, and recommendations. ‐ Conduct a workshop with project stakeholders to share initial evaluation findings, conclusions, and recommendations. ‐ Prepare a final report which must include an executive summary, table of contents, body and appendices, and must not exceed 25 pages for the body of the report (excluding the executive summary and appendices). ‐ The final report should be in line with USAID expectations as stated in Appendix 1 to the 2011 USAID Evaluation Policy (see table below). ‐ The final report should be written in English with the executive summary translated into Arabic. ‐ Upload and store all quantitative data collected for the purposes of the evaluation in a central database accessible to USAID/Jordan. ‐ Submit the final approved evaluation report to the agency’s Development Experience Clearinghouse (DEC). The completed evaluation must include a 3- to 5-page summary of the purpose, background of the project, main evaluation questions, methods, findings, conclusions, recommendations and lessons learned (if applicable) of the evaluation. DELIVERABLES AND TIMELINE Deliverable Timeline (working days following the USAID in￾briefing) Weekly progress reports or meetings with USAID to provide updates, verify and/or clarify information, and address any logistical Weekly, on going 41 issues Work Plan 3 Detailed evaluation design report (design, methodology, work plan, instruments). 7 Debriefing presentation for USAID management on evaluation findings, initial conclusions and recommendations 32 Workshop with project stakeholders to share evaluation findings, initial conclusions and recommendations 34 Draft report detailing the findings from the evaluation, lessons learned and recommendations for future interventions, incorporating comments from the USAID debriefing and validation workshop 37 USAID provides written comments on the draft evaluation report 44 Submit final report together with supporting materials detailing the findings from the evaluation and lessons learned with a concrete set of recommendations for future interventions 52 B. PERFORMANCE PERIOD The evaluation is expected to take place during October – December 2014 with the final report being submitted no later than December 7, 2014 so as to be used in the design of future programming. USAID anticipates that approximately 55 working days are needed to conduct the evaluation. The following schedule is envisioned: ‐ Preparation work and document review: Up to 7 working days. ‐ Interviews, field work, debriefings, and report writing: Up to 40 working days. ‐ Finalizing evaluation report: Up to 8 working days. The Evaluation Team shall make arrangements for a six-day workweek although the formal workweek in Jordan is Sunday through Thursday. The Evaluation Team shall budget for all travel and administrative support costs within Jordan as needed. The Evaluation Team is expected to arrange all logistics needed for the evaluation. The team should be aware that Eid-al-Adha is a national holiday in Jordan and falls within the evaluation timeframe. In 2014 Eid-al-Adha will begin on or around October 4 and will last for four days. C. TEAM COMPOSITION The Evaluation Team will consist of two technical experts, one of which will serve as Team Leader. The Evaluation Team will be composed of local experts with significant knowledge on youth programming and capacity building in Jordan countries. One of the team members should be an evaluation specialist, or an evaluation specialist should be a part of the team in some capacity. In the event that a local evaluation specialist with the requisite experience cannot be identified, this role may be filled by MESP staff. In addition to this expertise, the team should have: 42 ‐ Experience in monitoring and evaluation, preferably of USAID or donor programs; ‐ Excellent writing and communication skills in both English and Arabic; ‐ Demonstrated strong data analysis skills that show causality; ‐ Academic background in social science and evaluation methodologies is a plus. The USAID M&E Specialist will accompany the evaluation team periodically in order to build their knowledge of evaluations and evaluation methodologies. They will not participate as a full evaluation team member. All members of the team are expected to sign statements confirming that there are no conflicts of interests with their working on this evaluation. D. MANAGEMENT The Evaluation Team will report to the Program Office at USAID/Jordan, and will work closely with the Agreement Officer’s Representative (AOR) of Y4F. The Team can expect to receive support from the Y4F/AOR to determine plans, methods of action and timelines. The Team will be managed by the Monitoring and Evaluation Support Project (MESP) Senior M&E Specialist with regular oversight by the Chief of Party (COP) and in close contact with technical evaluation experts in MSI/Washington. The Team will provide briefings to USAID prior to commencing the evaluation, on a regular weekly basis throughout the evaluation, and prior to the submittal of the draft report. The evaluation will be carried out in its entirety in Jordan. 43 ANNEX II: EVALUATION DESIGN REPORT EVALUATION DESIGN REPORT: YOUTH FOR THE FUTURE (Y4F) PROJECT: FINAL PERFORMANCE EVALUATION OCTOBER 8, 2014 This publication was produced for review by the United States Agency for International Development. It was prepared by MSI Evaluation Team, Management Systems International. 44 MONITORING AND EVALUATION SUPPORT PROJECT Design Report Youth for the Future (Y4F) Final Performance Evaluation Contracted under AID-278-C-13-00009 Jordan/USAID Monitoring and Evaluation Support Project DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. 45 CONTENTS A. INTRODUCTION ................................................................................................................................................. 46 B. EVALUATION DESIGN ....................................................................................................................................... 51 C. DATA COLLECTION METHODS .................................................................................................................... 52 D. DATA ANALYSIS METHODS ............................................................................................................................ 54 ANNEXES ......................................................................................................................................................................... 54 ANNEX I: Getting to Answers ........................................................................................................................... 55 ANNEX II: Work plan .......................................................................................................................................... 59 ANNEX III: Tools ................................................................................................................................................... 61 ANNEX IV: References ........................................................................................................................................ 78 ANNEX V: Team Composition, Roles and Responsibilities ....................................................................... 79 46 A. INTRODUCTION Project Summary USAID Jordan Youth for the Future (Y4F) was originally designed as a five year $30 million project with the goal “to improve social services and protection for vulnerable populations, with an overarching focus on youth employability and civic engagement.” Implemented by International Youth Foundation (IYF) through national NGOs, the project planned to achieve this goal “through the delivery of comprehensive, community-based activities … that fall under four Components: 1) Life, Employability, and Entrepreneurship Skills for the 21st Century; 2) Provision of Youth Friendly Services; 3) Infrastructure and Environment Improvements; and 4) Policy Coordination.” Project activities aimed “to meet the needs of disadvantaged youth aged15-24 years…in six communities in Jordan selected in collaboration with USAID and the Ministry of Social Development (MOSD) based on poverty, social factors, population density, and representation of urban and rural areas”17. Since its inception in March 2009, and following a midterm project evaluation, a project audit and internal strategic reviews with USAID, IYF implemented various program changes in 2011 continuing in 2012 and early 2013. The most notable modifications to the original project design are:  A change in the operational model from implementation through national coordination NGOs (CNGOs) to an IYF direct implementation approach with Community Based Organizations (CBOs). Pilot phase of IYF direct granting to CBOs started July 2011.  The pilot phase of direct granting to CBOs also entailed an expansion of the six original target communities (Zarqa, Russeifeh, Irbid, Jordan Valley, Ma’an, and East Amman) to include three additional neighborhoods in Mafraq, Sahab and the Jordan valley.  IYF’s Cooperative Agreement was modified in May 2012 to increase the program budget from $30 million to $33 million in order to expand public sector capacity building activities to other key youth serving ministries beyond the MOSD including the Ministry of Labor(MoL), the Vocational Training Corporation (VTC) and the Higher Council of Youth (HCY).  By late 2012 / early 2013 IYF had discontinued the civic engagement component and entrepreneurship activities and effectively consolidated all of the program’s work streams and implementation activities under two core program models: 1)Preparing Youth for Successful Transitions for Work and 2) Building Sustainable Community Youth Hubs.  IYF changed the project branding and marketing strategy thus effectively changing the project name from Youth: Work Jordan (YWJ) to Youth for the Future (Y4F).  In January 2014 the project timeframe was extended to December 21, 2014 so that Y4F can continue its scaling and sustainability activities, particularly with public sector partners. Y4F Project Phase II (July 2011 - December 2014) The Youth for the Future project mission is to “mobilize and empower key actors at the local and national levels to work together in harnessing the productivity and potential of Jordan youth.” This 17 Attachment 2: Program Description – USAID Award contract 47 mission will be realized through two core objectives: 1) building the capacities and ownership of local communities to organize, deliver and sustain quality programs and services for youth and 2) empowering Jordanian government, business and civil society to nurture, sustain and scale locally proven best practice youth development models. In Phase II, Y4F streamlined its project approach to focus on two core models and the subsequent scaling of the most successful employability initiatives: Model 1- Preparing Youth for Successful Transitions to Work: Prepare at-risk youth for transformative life skills and career development. Model 2- Building Sustainable Community Youth Hubs: Empower community partners to build and sustain a network of quality youth services and expand the cadre of qualified youth workers and community trainers. Model 1 – Preparing Youth for Successful Transitions to Work through:  Life Skills Training:Y4F focused on establishing sustainable delivery mechanisms for Passport to Success (PTS), IYF’s globally certified life skills curriculum, to teach youth personal competency, constructive problem solving skills, productive work habits and the importance of playing an active role in community life.Y4F worked through selected CBOs to establish partnerships with key service providers including VTC and Al-Quds College to sustain the delivery of PTS beyond the life of the program. In addition, Y4F worked with VTC to integrate PTS into its youth trainings and make PTS mandatory for all VTC hospitality training centers.  Sector-Specific Training: In cooperation with CBOs and private (Ammon and Al-Quds college) and public sector (VTCs) partners, Y4F offered demand-driven sector-specific training programs to prepare youth for employment. The sectors which were selected with the Ministry of Labor (MoL) included hospitality, retail, and heating, ventilation and air conditioning (HVAC).  MoL One-Stop Shop (OSS):Y4F supported MoL in the development of one-stop shop models to enable its employment offices to offer life skills training and career guidance to interested youth.  Career Guidance: Y4F developed My Career, My Future toolkit, a comprehensive how-to guide for implementing career guidance in a community. The toolkit was piloted with Y4F’s CBO partners, VTC, and the MoL OSS. Model 2- Building Sustainable Community Youth Hubs through:  CBO Youth Hubs: Y4F worked to establish youth hubs within the targeted communities with its partner CBOs as focal points through which youth could gain access to available educational, skills training, recreational, and volunteerism opportunities. To build strong sustainable hubs capable of providing comprehensive demand-driven services to empower at-risk youth, Y4F centered its work on mobilization of key community actors, including youth centers, parents, chambers of commerce, municipal governments, local ministry offices, and vocational training centers that would form a robust youth support network. The model includes case workers within each CBO who track individual youth cases. Y4F criteria for youth beneficiary selection include: - Between the ages of 15-24from both genders, with a special emphasis on the recruitment of females; 48 - Non-completers of primary or secondary school system and including Tawjihi drop outs; - Unemployed / inactive, or employed in unsuitable jobs (e.g., a job that does not have a contract of employment (b) requires more than 40 hours per week, with no additional compensation; (c) operates in dangerous and or unhealthy working conditions; (d) does not meet minimum wage requirements; (e) has no specific job description; or (f) does not comply with the Jordanian Labor Laws); - Youth reside within or in the near vicinity of Y4F targeted neighborhoods. Results Framework and Theory of Change According to Y4F’s revised M&E plan updated on February 2012, the project long-term goal is to “Create an enabling environment with a greater capacity to more effectively serve youth-at-risk.” This goal is intended to be realized through three intermediate results (IRs): IR1Improved Youth Models, Practices and Policies; IR2 Capacity of CBO’s to provide access to sustainable livelihoods for youth strengthened; and IR3 Improved provision of youth friendly services and expanded civic engagement of youth to improve their local communities. In designing the M&E plan and setting targets, IYF noted some critical assumptions, the most important of which are: - Downward fluctuations in the labor market will not dramatically increase and - the influx of new populations—both internally and externally— will not dramatically increase the numbers of at‐ risk out‐of school/out of work youth. Evaluation Purpose and Scope The evaluation Statement of Work (SoW) identifies the purpose of the Y4F performance evaluation “to provide USAID with overall feedback and strategic recommendations for future programming related to youth development, including capacity building and sustainability approaches for youth serving institutions, particularly CBOs. The evaluation will also help USAID/Jordan obtain the necessary information to more effectively allocate resources for future programming.” The Y4F final performance evaluation will be primarily concerned with the project Phase II programming that took into consideration and integrated recommendations of numerous assessments and internal reviews. This evaluation will seek not to replicate assessment of known issues– the results of which have been acknowledged by USAID- but rather focus on evaluating final project outcomes (as framed by the evaluation questions below) for lessons learned and recommendations for future youth programming. Gender Aspects of Questions The evaluation intends to sample an equitable balance of female and male respondents. The analysis of data will also compare results by gender and address the gender aspect of the program by assessing girls’ challenges to access the program as compared to boys. The Statement of Work frames the scope of Y4F performance evaluation with the following questions: 1. Youth Outcomes USAID Question: As a result of Y4F efforts, are youth in a better position now than they were prior to the project in regards to: a. Re-entering the formal education system then remaining in the formal education system? 49 b. Entering the labor market then remaining in the labor market? c. Demonstrating positive life skills as defined by the Y4F training curriculum on positive life skills? Question clarifications and definitions18: IYF implemented a variety of capacity building activities with the objective of empowering at-risk youth to ‘transition to work’. This question and its subsets will seek to assess the effectiveness of Y4F activities in bringing about the desired ‘change’ of: a. Youth drop outs re-enrolling and remaining in the education system19as a result of Y4F interventions.20 b. Youth finding and maintaining a job for the past six months as a result of Y4F interventions. c. Youth indicating increased positive life skills21as a result of PTS trainings and other Y4F interventions. Most importantly, USAID is interested in ‘understanding the why’ of the resultant change or the lack of it. 2. & 3. & 4. Effectiveness & Sustainability of Capacity Building USAID Question: 2. How effective has Y4F been at building organizational capacity (financial viability, staffing, skills and knowledge needed to run the organization and provide intended services to youth, autonomous of any external assistance, etc.) among local CBOs and public sector stakeholders involved in the project? 3. Do CBOs supported by the Y4F project have the capacity, as defined previously, to continue youth programming? 4. Are there other organizations, such as vocational training colleges (VTCs), also strengthened by Y4F that USAID should continue to work with on positive youth development? Question clarifications and definitions: Y4F provided direct (financial training, strategic planning and proposal writing) and indirect capacity building (grant management, M&E, technical approaches, and employability) to local CBOs. Y4F also supported CBOs to establish Youth Friendly Spaces (YFS) to create local access to services whether recreational, educational, employment or volunteerism opportunities to youth. This question will seek to appraise the outcome/results of Y4F capacity building of CBOs in terms of: a. What has actually been implemented and integrated into the CBO internal institutional and operational systems following IYF capacity building (e.g. 18Clarifications and definitions were derived through the evaluation team in-briefing with USAID. 19Formal (till tawjihi level) and informal education such as enrolling in a vocational skills training. 20With the intention of attaining Tawjihi degree or a vocational training certificate. 21Y4F PTS training curriculum does not define criteria or a measure for positive life skills. The Youth survey will attempt to elicit youth feedback on positive improvements across a spectrum of life skills. 50 management, finances, M&E)? b. What are the youth services which have been learned and integrated into the portfolio of CBO community services? c. What is the likelihood that CBOs will maintain the ‘management’ changes that resulted from Y4F direct and indirect capacity building activities? d. What is the likelihood that CBOs will be able to maintain provision of youth friendly services and spaces to local communities? Specifically, USAID seeks ‘to understand’ the factors that achieved results (or lack of it) in terms of capacity building and sustainability of CBO youth initiatives. Y4F extended capacity building support to public sector partners. Y4F assistance included integration of PTS into curriculum (VTCs), ToT with staff, technical support, physical re-habilitation, re-furbishing and equipment. The evaluation will appraise: a- The impact of Y4F assistance on the type and quality of services provided by the public sector institution to youth. b- Which impact will be sustained on the long term and why? c- Future recommendations for continuing involvement with public sector institutions on youth development. 5. & 6. Learning USAID Question: 5. Were the two core program models “successful transitions to work” and “sustainable community youth hubs” appropriate models for achieving the projects stated results? 6. Are there strategies or practices IYF implemented that should be replicated or avoided in future youth programming? Question clarifications and definitions:Primarily, Y4F 2 models aspire to support youth integration and resumption of a productive life through empowering them with positive life skills, providing them with a technical and other related skills and supporting re-integration with mentorship, career counselling and youth friendly spaces. To extend these services to youth in a sustainable manner, Y4F built local capacity of CBOs and public sector institutions. The evaluation understands USAID questions as: a- Which of the two models components, activities or strategies were instrumental in delivering on expected results and why? Alternatively, which components, activities or strategies failed to deliver expected results and why? b- What needs to be replicated, added or dropped in future youth programming to enhance youth programming results. 7. Gender USAID Question: How did Y4F address any gender differential constraints (accessing, participating in, or benefitting from) in project activities? Question clarifications and definitions: Y4F project faced predominately cultural challenges in girls’ continuation in project activities, particularly relative to 51 internships and jobs. This evaluation question will assess the cultural challenges in outreach, mobilization and participation of girls in Y4F project activities and appraise the approaches implemented by IYF that successfully addressed these challenges. B. EVALUATION DESIGN This Performance Evaluation is intended to provide USAID/Jordan with overall feedback and strategic recommendations for positive youth development strategies and best practices for the capacity building and sustainability of institutions and the services they provide for youth. Y4F final performance evaluation approach will focus on:  Phase II of the project when IYF changed to direct implementation with CBOs (starting July 2011).  The outcomes/results achieved with the main target groups a) beneficiary youths and b) youth￾serving institutions (public and non-governmental).  Assessing the ‘why’ of outcomes, i.e., what factors were relevant to the successful achievement of results alternatively what factors impeded the realization of planned results.  Appraising sustainability of outcomes.  Strategic recommendations for future programming based on lessons learned from Y4F and Y4F evaluation results. The Y4F evaluation will employ a qualitative and quantitative approach to answer the evaluation questions. A quantitative approach will facilitate outreach to hundreds of youth beneficiaries for their feedback on the core outcomes of the program. Quantitative random sampling will provide statistically valid data for analysis of project outcomes for the different groups of Y4F youth beneficiaries. To augment quantitative data, qualitative approaches such as interviews and focus group discussions aim to ‘explain’ the quantitative results with in-depth assessments of underlying issues, challenges, stakeholders’ perspectives and context. The following project stakeholders and groups will be consulted for their feedback in order to answer the evaluation questions:  Youth beneficiaries  Community Based Organizations (CBOs)  Parents and community members (for each CBO)  Public sector partners (central and local): Ministry of Labor(MoL), Ministry of Social Development (MoSD), Higher Council of Youth (HCY), Vocational Training Centers (VTCs)  Private sector: Employers and training institutes  IYF program management and staff 52 C. DATA COLLECTION METHODS Data collection methods will consist of: a) Surveys; b) Key Informant Interviews; c) Group interviews; d) Focus Group discussions; e) Observation; and f) Document Review. a) Phone surveys will be conducted with a stratified random sample of 350 youth who completed Y4F training programs clustered in three strata: PTS only; PTS and technical skills; PTS, technical skills and other types of trainings (IT and Business English). The Y4F online database provides the total youth population figure for Phase II programming who completed22 Y4F trainings program as 4,125 youth. The percentage of each stratum out of the total population will be calculated and this percentage will be used as guideline to determine the sample size for each of the stratum identified above. A phone survey will then be conducted with the randomly selected participants in each stratum. A phone survey will also be carried out with private sector employers. Questions for the survey with employers will be developed following the pilot phase of phone survey with youth. The specific employers who will be contacted and total sample number will be determined by the surveyed youth who confirm that they are presently employed and notes the same name of employer as the one listed in Y4F database or provides the contact reference for his current employer. Phone surveys with youth employers will be helpful –among other things- in triangulating youth survey data. b) Key informant interviews will be conducted with public sector partners (central and local): Ministry of Labor, Ministry of Social Development, Higher Council of Youth and Vocational Training Centers. The evaluation will conduct 15 interviews with the staff of the ‘Change Unit’ established in the VTCs, Youth Leadership Centers (YLCs), Youth Affairs Directorate and Creativity Centers of the Higher Council of Youth and One Stop Shops (OSS) of the Ministry of Labor and with the CBOs directorate of the Ministry of Social Development. c) Group interviews will be conducted with 13 CBOs partners of IYF in Phase II. Group discussions will involve CBO management and, to the extent possible, the CBO staff who were involved in Y4F project implementation most particularly youth workers, mentors and career counselors. A total of 13 group interviews will be carried out. d) Focus Group (FG) discussions will be conducted in each of the 13 CBOs with parents and community members. In addition to this group, the evaluation will undertake 9 FGs (one in each of the project target neighborhoods) with an estimated 108 CBO youth beneficiaries23 (9FGs with 12 participants each). FG participant selection – parents and community members as well as youth beneficiaries- will be based on a convenience sample and facilitated by the CBO. Focus groups with CBO youth specific beneficiaries aim to provide the space for in-depth discussions with youth to elaborate on factors that support or impede actualization of project results. These discussions will seek to elaborate – explain the statistical results of the phone surveys. A total of 22 FGs will be conducted through the CBOs. Data collection through focus groups and surveys will be simultaneous. Phone surveys are mostly close ended questions and thus able to generate statistical results on the sample level. FGs will 22According to our meeting with IYF staff, PTS training was a preliminary requirement for any and all other types of trainings such as technical, IT, Business English…etc. 23To note that youth FG participants will also be requested at the end of the meeting to complete a similar survey as the youth reached through the phone survey. Thus, the total surveyed youth will become 450. 53 provide the “why” to “explain” the statistical results. This will be instrumental in the analysis phase of the evaluation. e) Observation: The evaluation team will visit the sites of CBO youth friendly spaces, MoL Abdali OSS and Higher Council of Youth Creativity Centers (3) which have been established by the program. Site visits will be conducted after centers’ managers/directors are interviewed. A total of 17 site visits will take place. f) Document Review: The evaluation team conducted a review of the project documentation focusing on Phase II of the Program. The review aimed to gain a thorough understanding of the Program background, including its goals and objectives, intended beneficiaries and partners, implemented activities and results, and key structural and programmatic changes. The review yielded valuable information on the evaluation issues, identified information gaps, and generated additional questions that helped inform the evaluation design. The annex section contains an informative list of the key desk review documents. Table 1: Data Collection Methods for the Seven Evaluation Questions Data Collection Methods Evaluation Questions Phone Survey 1, 5, 6 Key Informant Interviews 1, 4, 5, 6 Group Interview 1, 2, 3, 5, 6, 7 Focus Group Discussions 1, 2, 3, 5, 6, 7 Field Observation and Site Visits 2, 4 Document Review 1, 2, 3, 4, 5, 6, 7 Data collection (phone surveys, KIIs and FGs) will be conducted simultaneously during the period from October 15 - 23. Sampling Plan The evaluation will employ a stratified random sampling approach for the selection of the youth to be targeted by phone surveys. Youth beneficiaries who completed Y4F’ main training programs will be stratified by:  Total number of youth who completed PTS only;  Total number of youth who completed PTS and technical skills;  Total number of youth who completed PTS, technical skills and other types of trainings (IT and Business English). The Y4F online database will be used to identify the total number of youth in each stratum and conduct phone surveys with randomly selected youth in each stratum. The selected sample will provide a 95 percent confidence level with a +/- 5 percent interval of the total youth population. Per guidance from USAID, the total youth population from which the different strata will be identified include only the youth beneficiaries of Phase II as reflected in the Y4F database at 4,125 youth24. 24At the date of this design report 54 Phone surveys will also be conducted with a sample of employers (private sector) that associated with the program by hiring Y4F youth graduates. The sample number and specific employers who will be contacted for a phone survey will be determined by the surveyed youth who are employed at the time of the survey. The employers of these youths will be contacted to validate youth data and to elicit their feedback on some of the relevant evaluation questions. Sampling for qualitative assessments will either include the whole population (all 13 CBOs of phase II) or a selected number of public sector managers/ directors and staff trainees provided the sample comprise all type of public institutions that associated with the program and is biased towards public sector institutions that benefitted from the larger package of project assistance i.e. training with technical assistance and facility rehabilitation and furnishing. D. DATA ANALYSIS METHODS The main evaluation data analysis methods will be triangulation from different sources, and comparison of before and after self reported change of youth beneficiaries and organizations (public sector institutions and CBOs). The evaluation analysis will be structured by questions and sub-questions. Data collected from the qualitative and quantitative methods cited earlier will be triangulated for each question, e.g., information collected from surveys with the youth beneficiaries will be compared to Y4F online database, validated with employers when relevant and confirmed by qualitative interviews with CBO management and staff and parents and community groups. Assessment of ‘effectiveness and sustainability of capacity building’ will analyze data from focus groups with CBO management and project staff with community members, CBO records and observation. Quantitative data collected from surveys will be cross referenced for the different population attributes (such as gender, age, rural versus urban) but the main focus of the analysis will be for the different type of programming of participants and assessment how it affected project results. Table 2: Data Analysis Methods for the Seven Evaluation Questions Data Analysis Methods Evaluation Questions Comparison of before and after 1, 2, Triangulation from multiple stakeholders 1, 3, 4, 5,6,7 Validation from multiple sources 2,7 ANNEXES I. Getting to Answers (G2A) II. Workplan III. Tools (draft survey) IV. References V. Roles & Responsibilities Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 55 ANNEX I: GETTING TO ANSWERS Evaluation Questions Type of Answer/ Evidence Needed Methods for Data Collection Sampling/ Selection Approach Data Analysis Methods Method Data Source(s) 1. Youth Outcomes As a result of Y4F efforts, are youth in a better position now than they were prior to the project in regards to: a. Re-entering the formal education system then remaining in the formal education system?* b. Entering the labor market then remaining in the labor market?* c. Demonstrating positive life skills as defined by the Y4F training curriculum on positive life skills?* Comparison of youth status pre and post intervention. Description of achievements and/or deficiencies. Triangulation from multiple sources Document Review Annual and quarterly reports, M&E data, key achievements report NA Comparison of updated workplan and activity M&E plan with reported results Comparison of desk review to primary data collection Key Informant Interviews Y4F staff M&E, program staff, management Qualitative analysis of transcripts and notes triangulated with findings from document review and phone surveys Group Interviews and Focus Groups CBOs, parents, community members, and youth participants Group interviews with all13 CBO partners from Phase II: youth workers, mentors and career counselors and management. One focus group in each of the 13 CBOs with parents and community members as selected by CBOs. One focus group in each of the nine project target neighborhoods with an estimated 108 youth Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 56 Evaluation Questions Type of Answer/ Evidence Needed Methods for Data Collection Sampling/ Selection Approach Data Analysis Methods Method Data Source(s) beneficiaries (9 FGs with 12 participants each) selected based upon a convenience sample. Phone Survey Youth who have participated in Y4F training Stratified random sample of 350 youth who completed Y4F varying levels of training: PTS only; PTS and technical skills; PTS, technical skills and other types of training (IT and business English). Quantitative analysis of the survey results and correlation among different factors. Employers who have hired Y4F graduates Sample of employers Capacity Building 2. How effective has Y4F been at building organizational capacity (financial viability, staffing, skills and knowledge needed to run the organization and provide intended services to youth, autonomous of any external assistance, etc.) among local CBOs and public sector stakeholders involved in the project? 3. Do CBOs supported by Description of achievements and/or deficiencies including capacity assessments, existence of systems, documented plans for continued activities and evidence of funding sources Document Review Annual and quarterly reports, M&E data, CBO capacity assessments, institutions’ strategic plans, grant proposals/funding documentation NA Triangulation with interviews and observation Key Informant Interviews Y4F staff, CBO staff and management, partner public sector staff and management All CBOs, all public sector institutions in which directors/managers participated and centers rehabilitated. Triangulation and validation with documents, interviews and observation Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 57 Evaluation Questions Type of Answer/ Evidence Needed Methods for Data Collection Sampling/ Selection Approach Data Analysis Methods Method Data Source(s) the Y4F project have the capacity, as defined previously, to continue youth programming? 4. Are there other organizations, such as vocational training colleges (VTCs), also strengthened by Y4F that USAID should continue to work with on positive youth development? Group Interviews and Focus Groups CBOs, parents and community members All CBOs; one group of parents/community members from each CBO selected by CBO Triangulation with, documents, interviews and observation Learning 5. Were the two core program models (“successful transitions to work” and “sustainable community youth hubs”) appropriate models for achieving the projects stated results? 6. Are there strategies or practices IYF implemented that should be replicated or avoided in future youth programming? Youth outcomes Document Review Annual and quarterly reports, mid-term evaluation, FGD report NA Identification of those activities showing the most results. Content analyses of transcripts and notes to identify most effective activities, counterparts and stakeholders Key Informant Interviews Y4F management, CBO management, public sector managers, employers, VTCs Relevant partner institutions Gender 7. How did Y4F address any gender differential Description of the gender differentials Document Review Project documents: annual and quarterly reports, M&E N/A Content analyses Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 58 Evaluation Questions Type of Answer/ Evidence Needed Methods for Data Collection Sampling/ Selection Approach Data Analysis Methods Method Data Source(s) constraints (accessing, participating in, or benefitting from) in project activities? and actions taken to overcome challenges. Gender comparison for the project results by technical skill and employment data, FGD summary, and gender analysis report. Key Informant Interviews CBO staff and management, public sector partners Per sampling selection for previous questions Identification of those activities showing the most results. Content analyses of transcripts and notes to identify most effective activities, counterparts and stakeholders Focus Group and In-Depth Interviews Youth beneficiaries, parents and community members, CBO management and project staff Per sampling for previous questions Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 59 ANNEX II: WORK PLAN DESCRIPTION DELIVERABLE RESPONSIBLE TIMELINE Inception Phase Desk review of project documents  Desk review and gap analysis  List of evaluation reference documents Team Leader Technical Expert Evaluation Team Sept 25 – Oct 9 Work plan submitted to USAID for approval  Work plan/schedule Team Leader Thursday Oct 2 USAID approves Work Plan Wednesday Oct 8 Design of the evaluation methodology and tools  Design report  Draft instruments in English Team leader Technical Expert Evaluation Team 4 – 8 October Design report submitted to USAID  Final Design report October 9 by OOB USAID approves evaluation design report October 9 by COB Finalizing and translating data collection tools Training data collectors (Mindset) Piloting instruments(Oct 12 &13)  Final tools in Arabic  Training delivered  Piloting completed Team Leader Technical Expert Evaluation team +MESP Mindset data collectors October 8 - 13 Implementation Phase Data Collection through Key Informants Interviews (KII), Focus Groups (FG), Surveys, and Observation  KII notes and summary report  Survey data (statistical analysis tables and graphs)  Focus group reports  Analysis report by CBO Evaluation Team Mindset Oct 14 - 23 Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 60 DESCRIPTION DELIVERABLE RESPONSIBLE TIMELINE  (soft & hard copies for all deliverables) Analysis, De-briefing and Reporting Phase Analysis of data collected and drafting of reports  FCR Table  Draft report Team Leader& Evaluation team Oct 24 – Oct 30 Refine FCR, draft report  Draft PPT presentation  Draft report Team Leader & Evaluation team Oct 31 – Nov 3 De-briefing of findings to USAID  PPT presentation Team Leader, Evaluation team and MESP Nov 4 Validation Workshop with Stakeholders  PPT presentation Team Leader, Evaluation team and MESP Nov 5 Draft report incorporating USAID comments and stakeholders feedback from de-briefing and validation workshops.  Draft evaluation report to USAID MESP, Team Leader and Evaluation team Nov 11 USAID comments on draft evaluation report by Nov 20 Final evaluation report incorporating USAID comments  Final evaluation report MESP, Team Leader and Evaluation team Nov 27 Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 61 ANNEX III: TOOLS YOUTH OUTCOMES - PHONE SURVEY (draft for Pilot) Name: _______________________ Gender: ________________________________ Age: _________________________ Location: ______________________________ List All Type of Youth Involvement with Y4F according to the Project Database: a. Life Skills Training (PTS) ________________ b. Technical Training ________________ (+ type of technical training) c. IT training ________________ d. Business English training ________________ e. Career Counseling ________________ f. Others ________________ Hello, my name is…, I work for Mindset – a survey company. We are conducting a survey for the USAID funded project Youth for Future. This survey is conducted as part of a program evaluation and its results will help USAID design future programs for youth. The information you provide is confidential, your name will not be mentioned; only the aggregate results of this survey will be reported to USAID. Can you spare some time to answer few questions about your experience with the program? Thank you Q1-In which of the following did you participate (researcher to list the trainings offered by Y4F and check what is applicable … also seeking to validate database information) a. Life Skills Training (PTS) ________________ b. Technical Training ________________ (+ type of technical training) c. IT training ________________ d. Business English Language ________________ e. Mentoring and Career Counseling ________________ f. Others (specify) ________________ Q2- Which of these situations apply to your present condition? (Researcher to check appropriate answer) a. I am enrolled in an education program ________ (move section III then V) I - Data According to IYF Database (to be completed by Mindset researcher prior to conducting the phone call) II- Introduction Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 62 b. I am working ________ (move to section IV) c. I am studying and working (move tosection IIII, and IV) d. I am unemployed and not studying __________ (move to section V) Q3: Which of these education programs are you enrolled in? (Researcher to check applicable school program) a. ____ Formal education (Schools) b. ____ Vocational Training c. ____ Internship d. ____ University e. ____ others, specify ____________________________________________________ Q4: Did the Y4F project incite you / inspire you to return to school? a. NO ______________________; (researcher to move to Q7) b. YES _____________________; (researcher to continue with Q5) Q5- How long has it been since you re-enrolled in this education program? a. _______ Less than a month (this academic school year) b. _______ Between 1 and 3 months c. _______ Between 3 and 6 months d. _______ More than a year Q6: Which of the following project interventions was most effective in inciting you to resume education (if any)? a. Life Skills Training (PTS) ________________ b. Technical Training ________________ (+ type of technical training) c. IT training ________________ d. Business English Language ________________ e. Career Counseling ________________ f. Others ________________ g. None ________________ III- Return to School Y4F Outcome (section applicable ONLY to youth who responded (a) “I am enrolled in an education program” in Q2) Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 63 Q7: Do you plan on continuing your schooling/studies? a. NO ______________________; Why? ________________________________________ b. YES _____________________; Why? ________________________________________ (After Q7, researcher to move to section V) Q8- In what job-industry are you currently working? (Researcher to check applicable industry/job) a._________________ Tourism; b. _________________ Industry; c. _________________ Retail; d. _________________ Maintenance e. _________________ Others; specify _______________________________________ Q9: Did the Y4F project assist you in finding a job? a. NO ______________________; (researcher skip Q10 and go to Q 11) b. YES _____________________; (researcher, continue Q10) Q10: Which of the following project interventions was most effective in securing work/ a job for you? (more than one selection is allowed) a. Life Skills Training (PTS) ________________ b. Technical Training ________________ (+ type of technical training) c. IT training ________________ d. English Language ________________ e. Mentoring and Career Counseling ________________ f. Market Research ________________ g. Others (specify) ________________ h. None _________________ IV- Y4F Youth Work Outcome (section applicable ONLY to youth who responded “I am working” in Q2) Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 64 Q11: How long have you been holding this job? (Researcher to check applicable timeframe) a. _______ Less than a month b. _______ Between 1 and 3 months c. _______ Between 3 and 6 months d. _______ More than a year Q12: Do you plan on maintaining this work / job? a. NO ______________________; Why? ________________________________________ b. YES _____________________; Why? ________________________________________ Q13- In your opinion, did you benefit from attending the Y4F ‘Passport to Success’ training? a. NO ______________; (move to Q14) b. YES _____________;(move to Q15) Q14- Can you please tell me why?(Researcher attempts to fit respondent answer under either one or more of the following categories)(Multiple selections allowed if applicable) a. __________ Attended similar trainings previously b. __________ Training subject matters are not applicable to my personal condition b. __________ Training subject matters are not applicable to my cultural context c. __________ Quality of the training is poor d.__________ Duration of the training is insufficient e. __________ the trainer is not qualified f. __________ others, please specify ____________________________________________ Q15- How would you rate the benefits that you derived from the PTS trainings? a. Very Useful b. Useful c. Somewhat Useful d. Not Useful Q16- Out of the following list, which of your behaviors and or attitudes have been most positively affected by the PTS trainings (more than one selection is applicable) V- Passport to Success (PTS) or Positive Life Skills Trainings (questions in this section to be addressed to Youth who attended PTS trainings) Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 65 Character Element Motivation Creativity Communication Skills Constructive thinking skills Conflict management skills Responsibility Self Confidence Relationships with Parents Relationships with Community Relationships with Colleagues Relationships with Employers Commitment level Productivity Adaptability to Change Entrepreneurial spirit Quest for Self-improvement Community engagement Assertiveness Positive Gender Perspective Q17: Did you ever benefit from youth services/ youth spaces offered by your local CBOs? (such as IT center, youth spaces, to list those offered by local CBOs through the Y4F program) a. NO ______________________;(move to Q19) b. YES _____________________; which ones? (-) IT Lab; (-) Activity Hall; (-) playground; (-) Library; (-) handicraft center (continue with Q18) Q18: How would you rate the usefulness of these centers in providing you with a youth space and or youth friendly services? a. Very Useful b. Useful c. Somewhat Useful d. Not Useful Why (for Very Useful and Useful selections) ______________________________________________ VI- CBO Youth Friendly Services (YFS) & Public Employment services (section applicable to All youth respondents Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 66 Why (for Somewhat Useful and Not useful selections) ______________________________________ Q19: Have you ever sought (or would you ever seek) the services of the Ministry of Labor such as the employment office/one stop shop (local or central) to help you find a job (job fairs, advertisements for job openings…etc.) a. NO _____; Why _______________________________________________ (Researcher to thank the respondent and end the survey) b. YES _____________________; (researcher to continue with Q 20) Q20: How would you rate the usefulness of these centers in helping you find a job? a. Very Useful b. Useful c. Somewhat Useful d. Not Useful Why (for Very Useful and Useful selections) ______________________________________________ Why (for Somewhat Useful and Not useful selections) ______________________________________ Thank the Respondent and End the Survey Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 67 Group Interview CBO Management and Staff Interviewers: List of Interviewees: Names Position Introduction: Hello, we are conducting an evaluation for the USAID funded project Youth 4 the Future. Your input will help USAID design future program for youth. The information you provide is confidential, your names are for our records only. Meeting duration is expected to be 90 minutes. Shall we begin? Organization information – completed by the interviewer prior to the interview Name of the organization: Year of establishment: Location: Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 68 Institutional and Management Capacity building  What specific trainings (Strategic planning, Financial Management, Monitoring and Evaluation, Proposal Writing, Marketing and public relations, Grants Management) did you receive from Y4F?  Who are the staff who attended these trainings? Are they still working in your organization? If not, why not?  What specific aspects of these trainings were implemented in your internal management, operational and financial systems?  Please list the specific changes that occurred in your organization as a result of Y4F program? Validate through documents, materials when possible Before Y4F After Y4F Mandate Internal systems Grants management processes Funding received No. of beneficiaries Staff Others (specify)  Have those changes been permanently adopted? Why or Why not? Technical Capacity Building in youth Programming  Did you receive training through Y4F to develop/improve CBO youth program? What are these trainings?  Who are the staff who received those trainings? Are they still working in your organization? Why or Why not?  Were these trainings effective in building the capacity of the organization/ staff to provide your community with youth services?(Referral System, mentoring, career counseling…etc)  What type of services was offered to youth through the Y4F grant? Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 69  What changes occurred in youth programming within your organization as a result of Y4F program? Validate through documents, materials when possible Before IYF After IYF Provision of Life Skills trainings Provision of Career counseling and mentoring Provision of Youth friendly services (better infrastructure) Market research Referral System Parents engagement Experienced Staff No. of youth/ beneficiaries  Are you still providing your community/ youth with such services? How, Why or Why not?  Do you charge fees for the services? Are you receiving support from other sources (other than USAID) for these same services (such as MOSD, other donors, local community, private sector – businesses…).  Are those services much in demand by your community/ youth?  Do you have a plan to continue to provide your community/ youth with youth services? Models:  How did the project impact the youth who participated in the project? (Employment, return back to school, others)?  Is this project effective in helping youth resume education or transition to work (become more employable)? Why& How? Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 70 o What is the most effective element/ factor of this project and why? o What would you recommend to be added/ dropped/ replicated in future projects?  What was the most challenging part of the program? How did you address the challenge? And what recommendations do you have for similar future programs?  Did you encounter any challenges in accessing/ enrolling girls in the program? In accessing/enrolling boys in the program? and how did you address these challenges?  In your opinion, do you think that the CBOs are the best local partner to implement youth programming in your community? Why or why not? Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 71 Focus Group Meeting (Through CBOs) Youth who Participated in Y4F Program Note to the facilitator: Focus group discussions should include youth – Y4F program participants only (excluding parents and or CBO staffs) I-INTRODUCTION Thank you for joining us. We are conducting an evaluation for the USAID funded Y4F project implemented in partnership with (name of the CBO hosting the meeting). This meeting and our discussions will help us assess the results of the project for Youth and propose recommendations for future youth programs. The information you provide is confidential and serves the purpose of our analysis only. Your names are for our own internal records. Our meeting is expected to last 90 minutes. Shall we begin? II- Type of involvement with the Y4F project: What type of support, assistance or services did you receive through your CBO/Y4F program? III- Program Outcomes: What has changed in your life as a result of your participation in Y4F? Before Y4F After Y4F Personal aspect Education aspects Employability prospects a. Did the program inspire you to resume/continue your education? How and why? (What was the most critical factor that encouraged you to resume education?) b. Did the program prepare you to become a productive member in your community (transition to work)? How? And why? (volunteering, open your business, seeking employment) c. What was the most effective element (part) of the project assistance that you received? And please explain why? Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 72 IV- CBO Sustained Services: Are you still seeking /receiving services and support assistance from your CBO? If yes, what are they? If no, why not? V- Youth Services Offered by the Public Sector a- Did you ever seek youth services offered by the Higher Council of Youth such as through the Creativity Centers? If yes, what type of services? If no, why not? b- Did you ever seek youth services /employment services offered by the Ministry of Labor such as through the employment offices/one stop shops? If yes, what type of services? If no, why not? VI- Challenges and Recommendations a- In your opinion, what are the main challenges in finding a job or in completing your education? What are your suggestions on how to address these challenges? b- In your opinion, what are the main challenges accessing/enrolling girls in such programs? Accessing and enrolling boys in such programs? What are your suggestions on how to address these challenges in the future? c- Based on your experience with this program, what are your recommendations for future programs to youth? (program to help youth drop outs and at-risk youth either resume education, or enroll in a productive life- labor force) Thank you for your time. Please complete the following survey before leaving. I will go through each question with you Thank You for your time and valuable input Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 73 Focus Group Meeting Parents and community Members (thru the CBO) Interviewers: List of Interviewees: Name Relationship to Youth Y4F Participant Introduction: Hello, we are conducting a focus group session for the evaluation of the USAID- funded project Youth 4 the Future. Your input will help USAID design future programs for youth. The information you provide is confidential, your names are for our records only. Meeting duration is expected to be 90 minutes. Shall we begin? Questions:  How much do you know about the Y4F program?  What was your involvement or role in the program? Did you have youth participating in the program? Were you on some committee formed by the program?  Did you attend/participate in awareness trainings offered by the program?  Have those sessions/trainings changed your awareness and understanding of youth issues? Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 74  Have those sessions/trainings changed your awareness and understanding of the challenges faced by youth?  Have you noticed any positive changes in attitudes and/or behavior of your children that you attribute to their involvement in the Y4F program? (family, peers, community at large, workplace) if yes, what are those changes?  Do you believe the program improved your children’s education prospects? Future employment prospects? If yes, How? If no, Why?  What aspects of the program did you find particularly effective in helping your children find jobs or resume education?  Are there any aspects of the program that you found inappropriate in your cultural context? Any aspects of the program that you found inappropriate from a gender perspective? If yes, Why?  Would you recommend this type of program to other youth? Why or Why Not?  What program elements do you believe should be replicated/added or avoided in future programs addressing youth development?  Can you recommend other types of projects, programs and or interventions that you consider effective for tackling at-risk youth issues? Thank You for your time and valuable input Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 75 Key Informant Interview (KII) Public Sector Stakeholders Interviewers: Date: Organization Information:(to be completed by the interviewer prior to the interview) Organization name: Type of organization: Department: Contact person name: Title: Phone #: Address: Email: Introduction: Hello, we are conducting an evaluation for the USAID funded project Youth 4 Future. Your input will help USAID design future program for youth. The information you provide is confidential, your names are for our records only. Interview duration is expected to be 75 minutes. Shall we begin? 1. What was your involvement with the Y4F program? What type of support, assistance and trainings did you receive through the program? (Technical assistance, infrastructure development, furnishing, PTS training, ToT, Leadership, creativity centers…etc.) 2. What was the impact/ effect of the trainings received through Y4F on the following: o Personnel development of staff / trainees ______________________________________ o Organizational level _______________________________________________________ o Youth Services (type and quality of services offered to youth)_____________________________________________________________ _______ Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 76  What changes occurred in your organization as a result of Y4F program? Before Y4F After Y4F Management capacity Staff capacity Strategic planning Others (specify) Validate through observation, document and materials when possible.  What changes occurred in youth programming within your organization as a result of Y4F? Before Y4F After Y4F Provision of Life Skills trainings Provision of Career counseling and mentoring Provision of Youth friendly services (better infrastructure) Others (specify) Validate through observation, documents and materials when possible 3. What challenges have been encountered during Y4F program implementation? And, how did you address them? 4. Did Y4F support positively impact -client satisfaction and or increase the number of youth served by- your organization? 5. Are those services (those that have been supported or developed by Y4F) much in demand by your community/ youth? Why and why not? 6. If the program is to be implemented again, what changes/ additions would you recommend to enhance program results and sustainability? Thank You for your time and valuable input Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 77 Observation Checklist CBO Name: Researcher Name: Center Services (IT, Library…): Date and Time of Visit: Number of Youth / Users, Gender and Type of Services: When visiting centers (of CBOs and Public Sector), researcher to request that the interviewee permit him to visit the ‘center’/premises which have been furnished and or rehabilitated by Y4F. During the visit, check for the following: 1- Check for signs that the premises are being used/visited by youth. 2- Look for how many youth are presently using the center facilities and record number of users. 3- If number of users is too small, ask if the center is more crowded at other times (than when the researcher visit is taking place) 4- Look or ask for supporting documentation such as records, training materials, posters… as appropriate. 5- Try to ‘interview’ youth who are present and using the premises why do they come here, what type of services do they use most? Are they satisfied with the services offered by the center? What are the benefits derived from the center …and other questions as relevant to each specific center. 6- Interview the Center Manager (or any other person supervising the center) separately from the youth; pose the same questions as for the youth, also check for gender differences and triangulate all respondents feedback, to be substantiated when possible with observation. Note: Researcher to record his personal observation and notes Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 78 ANNEX IV: REFERENCES Youth: Work Jordan (YWJ) Annual Report, October 1, 2011 to September 30, 2012 Youth for the Future (Y4F) Annual Report, October 1, 2012 to September 30, 2013 Y4F Quarterly Report, October 1, 2012 to December 31, 2012 Y4F Quarterly Report, April 1, 2013 to December 31, 2013 Y4F Quarterly Report, April 1, 2013 to June 30, 2013 Y4F Quarterly Report, July 1, 2013 to September 30, 2013 Y4F Quarterly Report, October 1, 2013 to December 31, 2013 Y4F Quarterly Report, January 1, 2014 to March 31, 2014 Y4F Quarterly Report, April 1, 2014 to June 30, 2014 YWJ Year Four Work Plan, October 1, 2011 to September 30, 2012 Y4FYear Six Work Plan, October 1, 2013 to September 30, 2014 Y4F Monitoring and Evaluation Plan, April 10, 2011 (REVISED) Y4F Monitoring and Evaluation Plan, February 9, 2012 (UPDATED) Y4F Monitoring and Evaluation Plan, June 5, 2013 (REVISED) Mid-term Evaluation of Youth: Work Jordan (YWJ) Project, July 30, 2011 Audit of USAID/Jordan's Youth for the Future Program, April 15, 2013 Youth: Work Jordan (YWJ) Phase II Roadmap, 2012-2014 Focus Group Discussions with Youth for the Future Participants and Stakeholders, November 2012 to April 2013 (FINAL REPORT): Gender Findings and Strategies for Addressing Gender in Youth: Work Jordan (YWJ) Interventions, February 17, 2012 Y4F Improving Youth Employability Outcomes Report: The Role of Parents, August 25, 2014 The International Youth Foundation (IYF) Passport to Success (PTS) Life Skills Training Description Y4F Online M&E System (www.ywjme.org) Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 79 ANNEX V: TEAM COMPOSITION, ROLES AND RESPONSIBILITIES Evaluation Team Members and Designated Tasks Layla Moubayed, Team Leader/Evaluation Specialist, will be responsible for team organization, scheduling, and primary liaison with MSI M&E Advisor regarding technical aspects of the evaluation. She will have overall responsibility for the preparation and submission of the final report with substantial input from the other team members. The other team members will report to her on evaluation issues. The team leader will take the lead in preparing the project schedule and work plan, and will work closely with the other team members to determine information requirements, develop key questions, conduct interviews, and gather other relevant information. She will also lead the team’s effort to prepare and deliver a presentation on the team’s response to the evaluation questions, as well as the findings, conclusions and recommendations for future action of the final evaluation report. Ms. Moubayed will also supervise the preparation of the final report, and will ensure the quality of its contents. Upon receiving USAID’s comments on the final draft report, she will be responsible for making any final corrections and improvements. Khaled Qubajah, Evaluation Technical Expert will be responsible for designing and implementing appropriate data collection and analysis methodologies required to assess project results against evaluation questions and generate a high-quality report which documents findings, conclusions and recommendations. The particular role of the Evaluation Technical Expert is to assure that the technical expertise in evaluating youth programming and capacity building interventions are applied to effect the most rigorous evaluation possible and appropriate to the purpose and to assure that the relevant USAID and MSI evaluation standards and best practices are applied. Wala’a Aqrabawi, Local Evaluation Specialist (LES) to work with the evaluation team in preparing and implementing the overall evaluation. The Local Evaluation Specialist will be participating in the design, data collection and analysis. As one of the three Evaluators, the LES will be responsible for conducting in-depth interviews with a variety of project staff and stakeholders and providing the respective findings, conclusions and recommendations that will feed into the final evaluation report. Maram Barqawi, MESP Monitoring and Evaluation Specialist she will provide support to the evaluation team, help in the organization and delivery of evaluation tasks. She will also follow up with the data collection firm (Mindset) and perform quality assurance checks during the evaluation. Mindset, Data collection firm who will be responsible for conducting part of the field data collection, recruiting the enumerators and field supervisors, in addition to developing the sample plan to conduct the youth and employers surveys, they will also facilitate the youth and parents focus groups and produce the needed reports. Mindset will arrange to pilot the tools and report the suggested updates before starting the official data collection. Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 80 ANNEX III: EVALUATION TOOLS Public Sector Stakeholders Key Informant Interview Guide CBO Management and Staff Focus Group Discussion Guide Youth Focus Group Guide Parent and Community Members Focus Group Guide Observation Checklist Youth Phone Survey Questionnaire Employers Phone Survey Questionnaire Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 81 Key Informant Interview (KII) Public Sector Stakeholders Interviewers: Date: Organization Information: (to be completed by the interviewer prior to the interview) Organization name: Type of organization: Department: Contact person name: Title: Phone #: Address: Email: Introduction: Hello, we are conducting an evaluation for the USAID funded project Youth 4 Future. Your input will help USAID design future program for youth. The information you provide is confidential, your names are for our records only. Interview duration is expected to be 75 minutes. Shall we begin? What was your involvement with the Y4F program? What type of support, assistance and trainings did you receive through the program? (Technical assistance, infrastructure development, furnishing, PTS training, ToT, Leadership, creativity centers…etc.) What was the impact/ effect of the trainings received through Y4F on the following: Personnel development of staff / trainees ______________________________________ Organizational level ______________________________________________________ Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 82 Youth Services (type and quality of services offered to youth)_________________________________________________________________ What changes occurred in your organization as a result of Y4F program? Before Y4F After Y4F Management capacity Staff capacity Strategic planning Others (specify) Validate through observation, document and materials when possible. What changes occurred in youth programming within your organization as a result of Y4F? Before Y4F After Y4F Provision of Life Skills trainings Provision of Career counseling and mentoring Provision of Youth friendly services (better infrastructure) Others (specify) Validate through observation, documents and materials when possible 3. What challenges have been encountered during Y4F program implementation? And, how did you address them? 4. Did Y4F support positively impact -client satisfaction and or increase the number of youth served by￾your organization? Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 83 5. Are those services (those that have been supported or developed by Y4F) much in demand by your community/ youth? Why and why not? 6. If the program is to be implemented again, what changes/ additions would you recommend to enhance program results and sustainability? Thank You for your time and valuable input Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 84 Group Interview CBO Management and Staff Interviewers: List of Interviewees: Names Position Introduction: Hello, we are conducting an evaluation for the USAID funded project Youth 4 the Future. Your input will help USAID design future program for youth. The information you provide is confidential, your names are for our records only. Meeting duration is expected to be 90 minutes. Shall we begin? Organization information – completed by the interviewer prior to the interview Name of the organization: Year of establishment: Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 85 Location: Institutional and Management Capacity building What specific trainings (Strategic planning, Financial Management, Monitoring and Evaluation, Proposal Writing, Marketing and public relations, Grants Management) did you receive from Y4F? Who are the staff who attended these trainings? Are they still working in your organization? If not, why not? What specific aspects of these trainings were implemented in your internal management, operational and financial systems? Please list the specific changes that occurred in your organization as a result of Y4F program? Validate through documents, materials when possible Before Y4F After Y4F Mandate Internal systems Grants management processes Funding received No. of beneficiaries Staff Others (specify) Have those changes been permanently adopted? Why or Why not? Technical Capacity Building in youth Programming Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 86 Did you receive training through Y4F to develop/improve CBO youth program? What are these trainings? Who are the staff who received those trainings? Are they still working in your organization? Why or Why not? Were these trainings effective in building the capacity of the organization/ staff to provide your community with youth services?(Referral System, mentoring, career counseling…etc) What type of services was offered to youth through the Y4F grant? What changes occurred in youth programming within your organization as a result of Y4F program? Validate through documents, materials when possible Before IYF After IYF Provision of Life Skills trainings Provision of Career counseling and mentoring Provision of Youth friendly services (better infrastructure) Market research Referral System Parents engagement Experienced Staff No. of youth/ beneficiaries Are you still providing your community/ youth with such services? How, Why or Why not? Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 87 Do you charge fees for the services? Are you receiving support from other sources (other than USAID) for these same services (such as MOSD, other donors, local community, private sector – businesses…). Are those services much in demand by your community/ youth? Do you have a plan to continue to provide your community/ youth with youth services? Models: How did the project impact the youth who participated in the project? (Employment, return back to school, others)? Is this project effective in helping youth resume education or transition to work (become more employable)? Why& How? What is the most effective element/ factor of this project and why? What would you recommend to be added/ dropped/ replicated in future projects? What was the most challenging part of the program? How did you address the challenge? And what recommendations do you have for similar future programs? Did you encounter any challenges in accessing/ enrolling girls in the program? In accessing/enrolling boys in the program? and how did you address these challenges? Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 88 In your opinion, do you think that the CBOs are the best local partner to implement youth programming in your community? Why or why not? Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 89 Focus Group Meeting (Through CBOs) Youth who participated in Y4F Program Note to the facilitator: Focus group discussions should include youth – Y4F program participants only (excluding parents and or CBO staffs) I-INTRODUCTION Thank you for joining us. We are conducting an evaluation for the USAID funded Y4F project implemented in partnership with (name of the CBO hosting the meeting). This meeting and our discussions will help us assess the results of the project for Youth and propose recommendations for future youth programs. The information you provide is confidential and serves the purpose of our analysis only. Your names are for our own internal records. Our meeting is expected to last 90 minutes. Shall we begin? II- Type of involvement with the Y4F project: What type of support, assistance or services did you receive through your CBO/Y4F program? III- Program Outcomes: What has changed in your life as a result of your participation in Y4F? Before Y4F After Y4F Personal aspect Education aspects Employability prospects a. Did the program inspire you to resume/continue your education? How and why? (What was the most critical factor that encouraged you to resume education?) b. Did the program prepare you to become a productive member in your community (transition to work)? How? And why? (volunteering, open your business, seeking employment) Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 90 c. What was the most effective element (part) of the project assistance that you received? And please explain why? IV- CBO Sustained Services: Are you still seeking /receiving services and support assistance from your CBO? If yes, what are they? If no, why not? V- Youth Services Offered by the Public Sector a- Did you ever seek youth services offered by the Higher Council of Youth such as through the Creativity Centers? If yes, what type of services? If no, why not? b- Did you ever seek youth services /employment services offered by the Ministry of Labor such as through the employment offices/one stop shops? If yes, what type of services? If no, why not? VI- Challenges and Recommendations a- In your opinion, what are the main challenges in finding a job or in completing your education? What are your suggestions on how to address these challenges? b- In your opinion, what are the main challenges accessing/enrolling girls in such programs? Accessing and enrolling boys in such programs? What are your suggestions on how to address these challenges in the future? c- Based on your experience with this program, what are your recommendations for future programs to youth? (program to help youth drop outs and at-risk youth either resume education, or enroll in a productive life- labor force) Thank you for your time. Please complete the following survey before leaving. I will go through each question with you Thank You for your time and valuable input Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 91 Focus Group Meeting Parents and community Members (through the CBO) Interviewers: List of Interviewees: Name Relationship to Youth Y4F Participant Introduction: Hello, we are conducting a focus group session for the evaluation of the USAID- funded project Youth 4 the Future. Your input will help USAID design future programs for youth. The information you provide is confidential, your names are for our records only. Meeting duration is expected to be 90 minutes. Shall we begin? Questions: How much do you know about the Y4F program? What was your involvement or role in the program? Did you have youth participating in the program? Were you on some committee formed by the program? Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 92 Did you attend/participate in awareness trainings offered by the program? Have those sessions/trainings changed your awareness and understanding of youth issues? Have those sessions/trainings changed your awareness and understanding of the challenges faced by youth? Have you noticed any positive changes in attitudes and/or behavior of your children that you attribute to their involvement in the Y4F program? (family, peers, community at large, workplace) if yes, what are those changes? Do you believe the program improved your children’s education prospects? Future employment prospects? If yes, How? If no, Why? What aspects of the program did you find particularly effective in helping your children find jobs or resume education? Are there any aspects of the program that you found inappropriate in your cultural context? Any aspects of the program that you found inappropriate from a gender perspective? If yes, Why? Would you recommend this type of program to other youth? Why or Why Not? What program elements do you believe should be replicated/added or avoided in future programs addressing youth development? Can you recommend other types of projects, programs and or interventions that you consider effective for tackling at-risk youth issues? Thank You for your time and valuable input Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 93 Observation Checklist CBO Name: Researcher Name: Center Services (IT, Library…): Date and Time of Visit: Number of Youth / Users, Gender and Type of Services: When visiting centers (of CBOs and Public Sector), researcher to request that the interviewee permit him to visit the ‘center’/premises which have been furnished and or rehabilitated by Y4F. During the visit, check for the following: 1- Check for signs that the premises are being used/visited by youth. 2- Look for how many youth are presently using the center facilities and record number of users. 3- If number of users is too small, ask if the center is more crowded at other times (than when the researcher visit is taking place) 4- Look or ask for supporting documentation such as records, training materials, posters… as appropriate. 5- Try to ‘interview’ youth who are present and using the premises why do they come here, what type of services do they use most? Are they satisfied with the services offered by the center? What are the benefits derived from the center …and other questions as relevant to each specific center. 6- Interview the Center Manager (or any other person supervising the center) separately from the youth; pose the same questions as for the youth, also check for gender differences and triangulate all respondents feedback, to be substantiated when possible with observation. Note: Researcher to record his personal observation and notes Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 94 Youth For the Future Phone Survey Can I ask you some questions? Yes No IDI_1. Name of respondent IDI_2. Respondent age IDI_3. Governorate Irbid Ajloun Jerash Mafraq Balqa Amman Zarqa Madaba Karak Tafilah Ma'an Aqaba IDI_4. CBO Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 95 IDI_5. Gender Male Female IDI_6. Mobile Number Section Two Q1. Which of the following activities did you participate in? PTS Technical training ______________________ IT English language Career counselling Other ______________________ Q2. Which of the following applies to your current status? I am enrolled in an education program I am working I am studying and working I am not studying and unemployed Section Three Q3. Which of the following educational programs are you currently enrolled in? Formal education (Schools) Vocational Training Internship University College Other ______________________ Q4. Was your return to the educational program as a result of the support and training provided by Youth for the Future? Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 96 Yes No Q4_1. What encouraged you to enroll in the educational program? Q5. When did you return to the educational program? Less than a month - this school year 1- 3 months 3- 6 months More than year Q6. What was the type of support or intervention that was implemented by Youth for the Future that had the greatest influence on your decision to return to school? PTS Technical training ______________________ IT training English language Career counselling Other ______________________ None Q7. Do you plan to continue your education? Yes ______________________ No ______________________ I don't know / Not sure ______________________ Section Four Q8. In which sector do you work? Tourism Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 97 Industry Sales / Retail Construction Maintenance Other ______________________ Q8_1. What is the name of the company you work in? Q9. Did you find this job opportunity as a result of the support from Y4F? Yes No Q9_1. What helped you find this job opportunity? Q10. What was the support provided by Y4F that had the biggest influence on your job search? PTS Technical training IT training English language Career counselling Other ______________________ Market research None Q11. Since when have you been employed in this position? Less than a month 1- 3 months Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 98 3- 6 months More than a year Q12. Do you plan to continue in this position? Yes ______________________ No ______________________ Don't know / Not sure ______________________ Filter Question: Have you participated in PTS? Yes No Section Five Q13. Did you benefit from participating in PTS? No Yes Q14. Can you tell me why? I attended similar trainings The trainings topics don't interest me The training topics don't suit our social context Weak training topics Inadequate training period Weak trainers / poorly-trained trainers Other ______________________ Q15. How do you evaluate the benefits from participating in PTS? Very useful Useful Useful to some extent Not useful Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 99 Q16. What were the factors from PTS that had a positive impact on you personally? Yes No Does not apply (No answer) Motivation Creativity Communication Skills Constructive thinking skills Conflict management skills Responsibility Self Confidence Relationships with Parents Relationships with Community Relationships with Colleagues Relationships with Employers Commitment level Productivity Adaptability to Change Entrepreneurial spirit Quest for Self-improvement Assertiveness Section Six Q17. Have you benefited from the youth support provided by Youth for the Future? No Yes Q17_1. What was this support? Q18. What is your evaluation of those friendly services provided by your CBO? Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 100 Very Useful Useful Useful to some extent Not useful Q18_1. Why? Q19. Have you resorted to the local employment office? No ______________________ Yes Q20. What is your evaluation of the services provided by those offices? Very Useful Useful Useful to some extent Not useful Q20_1. Why? End of interview. Thank respondent. Employers Survey ID 1 - Name IWJ Code Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 101 ID 2 - Gender Male Female ID 3 - Training participant said he/she received Life skills (PTS) Technical training ______________________ IT Business / English Vocational training Other ______________________ ID 4 - Sector participant works in ID 5 - Participant position at the workplace Position ______________________ (Not available) Company Details C1 - Company name C2 - Sector C3 - Name of supervisor C4 - Position of supervisor C5 - Phone number Mobile ______________________ Landline ______________________ C6 - Email Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 102 C7 - Address Survey Q1: Can you please confirm that (name of youth) is currently employed in your company? No, he/she does not work here Yes, he/she works here Q2: Did he (name of youth) ever work in this company? No, he/she never worked here before Yes, he/she worked here before ______________________ I don't know / Don't remember Q3: How did he/she leave his/her job? He/she resigned He/she was fired ______________________ Other ______________________ Q4: Do you have any suggestion or recommendation that you would like to share with us following the company experience with _______ name of youth? No Yes ______________________ Q5: Since when? _____________ (number of months) Q6: How would you appraise (name of youth) on the following elements: High degree Medium degree Low degree (Does not apply) Technically appropriate for this work Has good relations with superiors Has good relations with colleagues Displays a positive attitude to work problems Demonstrates commitment to work Demonstrates motivation to improve on the long term Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 103 Company's satisfaction with his/her performance in general Company's satisfaction with attitude Dealing with customers Q7: Did you employ / recruit more than one youth through the CBO name/Y4F program? No Yes: Specify number ______________________ Don't know Q8: In your opinion, was the training degree / diploma an important element in the company’s decision to hire these youths? No Yes Don't know Q9: What is the company usual recruitment channel for future job openings? Advertisements in newspapers Advertisement in specific internet websites (such as Bayt.com) Posting with Ministry of Labor - One Stop Shop Personal relationships Directly with training schools or colleges Through the community Others, specify ______________________ Don't know Q10: Would you consider the same CBO for future staffing needs? Yes ______________________ No ______________________ Don't know Q11: What would be your recommendation for additional skills training or other type of trainings that would enhance youth employability? Q12: Any final comment that you would like to share with us following the company’s experience with _______ name of youth? No Youth for the Future (Y4F) Project Final Performance Evaluation – Design Report 104 Yes ______________________ End. Thank respondent. 105 ANNEX IV: SOURCES OF INFORMATION Key Informant Interviews Focus group discussions Bibliography Key Informant Interviews from 10 employees of the International Youth Foundation (IYF), 16 representatives of the Jordanian public sector, and 4 USAID representatives, conducted from September 17th to October 29th. Personally Identifying Information withheld. Focus Group Discussions Name of CBO KII FGD CBO Manageme nt and Staff FGD Pare nts FGD Yout h Total # of CBO Management and Staff Total # of Parent s Total # of youth # Of FGD conducted Iskan Prince Talal Association 0 1 1 1 5 9 11 3 Khreibit Al Souq Association 0 1 1 1 2 8 10 3 Shuaa Association 0 1 1 1 4 4 6 3 Training and Rehabilitation of Jordanian Women 1 0 0 0 0 0 0 0 Al Ta'awon Association 0 1 1 1 7 10 12 3 Zein Al Sharaf Association 0 1 1 1 4 5 9 3 That Al Nitaqain Association 0 1 1 1 4 8 10 3 Waqqas Association 0 1 1 1 5 11 12 3 106 Family and Child Protection Association 0 1 1 1 7 9 11 3 Al Jawasreh Association 0 1 1 0 5 9 0 2 Al Rawda Association 0 1 1 0 5 6 0 2 Khawla Bint Al Azwar 0 1 1 0 5 10 0 2 Sanabel Al Khair Association 0 1 1 1 2 10 10 3 1 12 12 9 55 99 91 33 Bibliography International Youth Foundation. Technical Specifications and Legal Requirements for Establishing and Licensing Childcare Centers for Children of Working Women. A study on how to best activate Article 72 of the Labor Law on Establishing Childcare Centers for Children of Working Women at Jobsites. (2013) International Youth Foundation. Transport Services and their Impact on Youth Employment and Development in Jordan. 2014 International Youth Foundation. Youth Friendly Services Manual. 2013 International Youth Foundation. Strategic Plan for Youth Empowerment in Irbid. (2014-2016) International Youth Foundation. Strategic Plan for Youth Empowerment in Zarqa. (2014-2016) International Youth Foundation. Parent Engagement Manual. (2013) International Youth Foundation. Annual and Quarterly Reports. (2009-2014) Mid-term Evaluation of Youth: Work Jordan Project Report. (2011) Youth for the Future. Monitoring and Evaluation Plan. (2013) International Youth Foundation. Final Report Focus Group Discussions with Youth For the Future Participants And Stakeholders. (2012-2013) Youth: Work Jordan. Roadmap Phase Two. (2012-2014) Youth for the Future. Streamlined Activities. (2013) Youth for the Future. Gender Findings and Strategies for Addressing Gender in YWJ Interventions 107 Youth: Work Jordan. Labor Market Demand Assessment For YWJ Expansion in New Communities (Shouneh, Mafraq and Sahab). (2011) Youth for the Future. Wrap up report Community Youth Hubs & Sustainability. (2014) Community Based Organization Factsheets Office of Inspector General. Audit of USAID Jordan’s Youth for the Future Program Audit Report No. 6-278-13-010-P. (2013) The USAID Funded Youth: Work Jordan Program. Capacity Building Workshop For the Ministry of Social Development and Women Affairs. (2010) Youth: Work Jordan. Work plans. (2009-2013) Youth for the Future. Passport to Success Brochure Jordan Program 108 ANNEX V: DATA COLLECTION METHODOLOGY Data Collection Methodology Annex Youth for the Future Program Assessment November 6th, 2014 109 Inception meeting and workplan Review of IYF: Y4F database to extract sample Design of quantitative sample Project staffing Design of qualitative and quantitative tools Review of tools Training of staff Formatting of tools and uploading onto survey system Collaboration and coordination with CBOs to recruit focus group participants Pilot of both qualitative and quantitative methods Modification of assessment tools based on pilot results Fieldwork‐ quantitative survey, qualitative focus groups and observation visits Data Cleaning, coding and entry Transcription and analysis grids for qualitative focus groups Charting of phone survey data Reporting on assessment findings PROCESS OF IYF ASSESSMENT-DATA COLLECTION PHASE Below are the steps followed for the data collection: Done by MESP and Mindset Done by MESP Done by Mindset 110 QUANTITATIVE SURVEY METHODOLOGY ANNEX: RANDOM SAMPLE, CONVENIENCE SAMPLE, AND EMPLOYERS SAMPLE Sample Selection for the Random Sample The USAID-funded Monitoring and Evaluation Support Project (MESP) conducted an assessment of the Youth for the Future project. For the assessment, MESP required a robust probabilistic sample to collect data on the project. This document describes the sample selection process that was done by Mindset, MESP’s research partner, to select a probabilistic sample from the Youth for the Future database of participants. It also provides a response rate calculation for the random sample. This document also provides the sampling and fieldwork details for the convenience and employers samples. First Step: Database Review Mindset was given a user name and password to access the IYF database. The initial objective of accessing the database was to determine the total population of the sample frame, in this case, lists of participants who have participated in the activities that MESP needs to assess. The following is a step-by￾step list of how Mindset accessed the database on the 13th of October, 2014. 1- Visited link: www.ywjme.org 2- Selected “Completer Youth Training Reports” 3- Selected “CNGO IYF” 4- Clicked “Run” The resulting report contained 3,873 participant names, YWJ ID numbers, gender, employment status and type of training the youth has taken. (However, the list did not contain contact information.) The report of 3,873 participants was exported to Excel. This 3, 873 figure represents the universe of the sample. Category Universe Number Successfully Completing (Success Based on Attendance) Life Skills Training 2020 52% of total Technical Training 1302 34% of total Business English + IT 551 14% of total Second Step: Sampling Approach A simple random sampling approach was used to develop a probabilistic sample. 111 Third Step: Sample Size Calculation To calculate the sample size, a formula for calculating sample sizes to estimate proportions within a small population was used: n = (Z^2 × P(1 – P))/e^2 (Where: n = sample size; Z = Z-score; P = proportion; and e = precision level) For the current sample, the following parameters were used: Z-score = 1.96 (this is the critical value for a 95% confidence level) e = .05 (this is a margin of error of plus or minus 5%) P = .5 (this is the most conservative proportion level) Using the above values, we arrived at a sample of 384 respondents. However, since we have a small population (3,873), we did an adjustment of the sample size. This adjustment, ((N*n)/(N+n)), yielded a final sample size of 350 respondents. Fourth Step: Random Selection of 350 Participants SPSS was used to generate a random sample of approximately 350 participants. To select a random sample, the following steps were taken:  From the open SPSS file, “Data” chosen  From “Data”, “Select Cases” was chosen  From “Select Cases”, “Random Sample of Cases” was chosen  From “Random Sample of Cases”, 10% was entered as Sample Size, in order to randomly generate 10% of cases. This was repeated three times (to generate approximately 900 phone numbers).  From “Select Cases”, “Copy selected cases to a new dataset” was chosen  The new dataset was named, saved and then exported to Excel The Excel sheet contains the YWJ ID, but not the contact information for the participants. The database website was accessed again, and the YWJ ID was inputted to recall the contact information for each participant. After this step was completed, the excel sheet was now ready to be handed over to the interviewers to initiate the phone calls. Random Sample Fieldwork Dates The following table provides the main dates for the survey: Date Activity 12-10-2014 – 13-10-2014 Interviewer training 14-10-2014 – 15-10-2014 Pilot (50 questionnaires) 16-10-2014 – 22-10-2014 Data collection 22-10- 2014 – 25-10-2014 Cleaning and back-check 25-10-2014 Data file submitted to MESP Response Rate Calculation 112 In order to reach the requisite sample of 350 trainees, we generated approximately 900 numbers, over three random runs. Of these 736 numbers were dialed. The table below lists the cases that were encountered during the survey, followed by the response rate calculation. Outcome Description Number of such cases in our random sample Valid Response Successful interview 357 Refusal Respondent refused to participate in the interview 25 Out of scope Potential respondent did not have a working number we could reach them on. This includes incomplete numbers, wrong numbers, numbers belonging to relatives, disconnected lines, and expired numbers. 239 Sample loss These are persons we should have reached but could not because their valid lines were either switched off or lines were busy when we tried to call. (We try a line at least twice.) 115 To calculate the response rate, we used the following formula: Response rate = Valid response (357) ------------------------------------------------ Total numbers tried (736) – Out of scope (239) Therefore, the response rate for the random sample was 72%. (This is a high and healthy response rate.) During cleaning, 5 questionnaires were discarded, bringing down the final total to 352 respondents. Sample Selection for the Convenience Sample The convenience sample is comprised of youth focus group participants. After each focus group was concluded, participants were provided with the same questionnaire that was implemented for the random sample. This survey was self-administered. Once the questionnaires were returned to the office, they were edited and entered into the data system. (We called back respondents when data were missing.) In total, 83 questionnaires were gathered from the convenience sample. Convenience Sample Fieldwork Dates 113 Date Activity 14-10-2014 – 19-10-2014 Nine focus groups (with youth) 19-10-2014 – 21-10-2014 Questionnaires cleaned and entered into the data system Sample Selection for the Employers Sample The employers sample was designed after the completion of the random survey. From the random survey, 167 respondents said they were working (either working alone or working and studying at the same time). These respondents were asked to provide the name of the establishment in which they work. We looked up the respondent’s profile on the IYF Database and compared between where the respondent said they work in the survey and what was reported in the database. If the establishments matched, we called the establishment directly to conduct the interview. On the other hand, if the establishments didn’t match, the assumption was that the respondent transitioned to a different employer. In total, 22 employers were reached through the process described above. Employer’s Sample Fieldwork Dates Date Activity 28-10-2014 Phone interviews 29-10-2014 Cleaning and coding Quality Measures implemented for the Quantitative Sample 1- Each team of phone interviewers (each three callers) was supervised by one senior supervisor who monitors all their calls. 2- Data is cleaned by a data analyst as it is being entered on the system (missing responses, illogical responses, incomplete or unclear responses are flagged to be called back and verified). 3- 30% of all interviews were called back randomly to perform back check on random questions to verify if they are compatible with the answers entered by the interviewer. QUALITATIVE FOCUS GROUPS AND OBSERVATION VISITS Sample Selection for the Qualitative Sample For the assessment, MESP needed to conduct focus groups and observation visits within all 13 CBOs in which the project was implemented. Three segments were targeted by the qualitative methodology: 1- CBO workers who worked with Youth for the Future Program 2- Youth who were trained by the program within the evaluated CBOs 3- Parents of youth who were trained by the Youth for the Future Program. 114 CBO Name Date of conducting the focus groups and observation visit Youth CBO workers Parents of youth trained by Y4F Observation visit Iskan Talal Association 13-10-2014 Khreibet Al Souq Association 14-10-2014 Shoua'a Association 15-10-2014 Training and Rehabilitation of Jordanian Women 16-10-2014 Al Ta'awon Association 16-10-2014 Zein Al Sharaf Association 16-10-2014 That Al Nitaqain Association 18-10-2014 Waqqas Association 18-10-2014 Family and Child Protection Association 18-10-2014 Sanabel Al Khair Association 19-10-2014 Khawla Bint Al Azwar 19-10-2014 Al Jawasreh Association 19-10-2014 Al Rawda Association 19-10-2014 Total Focus groups and visits 9 13 12 13 Deliverables of the Qualitative Sample Focus groups and observation visits were documented by the following methods: 1- Audio recordings of focus groups 2- Photos of observation visits 3- Arabic transcripts of focus groups 4- Qualitative analysis grids for the parents and youth focus groups Quality measures implemented for the Qualitative Sample In total, 34 focus groups were conducted. Eighteen of those focus groups (52% of the total sample) were attended by a supervisor from either MESP or the data collection agency. They reported on the following aspects: 1- Readiness of the venue and its suitability for conducting the focus groups 2- Professionalism of focus group facilitator 3- Adherence to the questions of the research 4- Preparedness of facilitator in terms of equipment and knowledge of the topic Five recordings (15% of the total sample) were listened to after the focus groups to check for quality of the questions and adherence to the research objectives. After transcribing the focus groups, 8 transcripts (24% of the total sample) were compared to the audio to ensure faithfulness to the proceedings of the sessions. 115 ANNEX VI. YOUTH SURVEY DATA REPORT Youth Survey Data Report Section 1: Age, Gender, Governorate and CBO Age, Gender and Governorate 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 0 10 20 30 40 50 60 70 80 90 17 18 19 20 21 22 23 24 25 26 27 32 Number/percent of respondents by age (n=352) Number Percent Male, 68% Female, 32% Gender (n=352) Irbid, 8.2% Ajloun, 0.3% Mafraq, 10.8% Balqa, 13.6% Amman, 39.8% Zarqa, 17.6% Madaba, 2.3% Tafilah, 3.4% Ma'an, 4.0% Respondents by governorate (n=352) 116 CBO (as reported by respondent) Section 2: Type of Activity and Current Status Types of Activity (as reported by respondent) 39 36 27 25 25 25 22 20 19 18 14 12 10 9 7 5 4 4 4 3 3 3 3 2 2 2 2 2 1 1 1 1 1 Khreibet Al Souq Ladies Association Queen Zain Al-Sharaf Charity Association Prince Talal Bin Mohammed Housing Association for Social… Sahab Association for Social Development Sanabel Al Khair Association for Social Development Family and Childhood Protection Society Training and Rehabilitation of the Jordanian Woman Charity… Khawla Bint Al Azwar Doesn't know / Doesn't remember VTC - Marka That Al Nitakayn Al-Rawda Charity Association Shua'a Al-Jawasreh Ladies Association VTC (Not specified) "Amoun" (Not on the list) "Al-Asala Center" (Not on the list) Al-Taawon Association Waqqas Association "Ma'an Center for Youth" (Not on the list) "Labour Office" (Not on the list) "Tatweer Al-Mar'a / Women's Development Association" (Not on… "Nady Al-Mu'alimeen / Teachers Staff Club" (Not on the list) Al-Shihab Social Association VTC - Sahab "All for Jordan Youth Commission" (Not on the list) VTC - Salt "IYF" "Al-Thahabia" (Not on the list) "Al-Islah Association" (Not on the list) "Al-Tanmiyeh Al-Ijtima'eyeh / Social Development" (Not on the list) "Islamic Association" (Not on the list) VTC - Mafraq Number of respondents by CBO (as reported by respondent) (Total=352) 117 Type of technical training N % Hospitality 123 59.7% Heating and cooling 39 18.9% Beauty/Cosmetics 17 8.3% Retail sales 5 2.4% Cooking 4 1.9% Electricity 3 1.5% Interior design 2 1.0% Confectionary 2 1.0% Secretarial duties 1 0.5% Maintenance 1 0.5% Soap making 1 0.5% Straw handicraft 1 0.5% Calligraphy 1 0.5% Printing 1 0.5% Accounting 1 0.5% Data entry 1 0.5% Electricity / Heating & cooling / Welding 1 0.5% 319 206 162 90 63 31 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 0 50 100 150 200 250 300 350 PTS Technical training IT English Language Career Counseling Other Types of activity (as reported by respondent) N % 118 Electricity / Heating & cooling / Welding / Pneumatics 1 0.5% Graphic design / Confectionary 1 0.5% Type of activity – other N Type of activity – other N Retail 6 Confectionary 1 Volunteering 3 Wall painting 1 Graphic design 2 Other 1 "Friends of the Police" seminar 2 Operating heavy machinery 1 Project management 1 First aid 1 Secretarial training 1 Communication skills 1 Bookkeeping 1 Secretarial duties / Arranging flowers 1 Training on photovoltaic cells 1 Volunteering / Photography 1 Professional health and safety 1 Painting / Planting 1 Accessories 1 Electricity / Vehicle maintenance 1 Training Categories 119 Current Status 0% 5% 10% 15% 20% 25% 30% 35% 40% 0 20 40 60 80 100 120 140 12345 Number of trainings Number of trainings taken by participants Total number of trainings = 850*; mean number of trainings = 2.4 ; median number of trainings = 2 137 73 9 51 5 19 58 PTS + Technical Training + Other PTS + Other Only Technical training + Other only PTS and technical training only Other only (IT, English and Career Counseling) Technical training only PTS Only Trainings (n=352) 120 Activity/Status by Gender Section 3: Return to Education 167 158 18 9 I am not studying and unemployed I am working I am enrolled in an education program I am studying and working Current status (n=352) 88% 72% 18% 53% 16% 5% 96% 39% 42% 31% 22% 8% PTS Technical Training IT English Career Counseling Other Q1. Which of the following activities did you participate in? (Total male = 239; Total female = 113) Male Female 4% 58% 3% 35% 8% 17% 1% 74% I am enrolled in an education program I am working I am studying and working I am not studying and unemployed Q2. Which of the following applies to your current status? (Total male = 239; Total female = 113) Male Female 121 This section is only applicable to respondents who reported they are “enrolled in an education program” and “studying and working” (n=27) Education Program Education Program by Gender 14 5.0 4 2 11 University Formal education (school) Vocational training College Internship Other Q3. Which of the following educational programs are you currently enrolled in? (total=27) 66 3 12 Less than a month - this school year 1- 3 months 3- 6 months More than year Q5. When did you return to the educational program? (total=27) 122 4 3 1 8 1 0 17 11 0 6 11 10 5 4 1 14 2 1 27 Formal education Vocational training Internship University College Other Total Q3. Which of the following educational programs are you currently enrolled in? (total (male = 17; total female = 10) Male Female Total 4 5 1 7 17 2 1 2 5 10 66 3 12 27 Less than a month - this school year 1- 3 months 3- 6 months More than year Total Q5. When did you return to the educational program? (total male = 17; total female = 10) Male Female Total 123 Return to Education Q4_1. What encouraged you to enroll in the educational program? My own decision 6 I aspire/want to complete my education 2 A neighbor of mine (encouraged me) 1 I didn't find work, so I needed to take another course in order to find work 1 I study in the same field as my work 1 I was waiting for the opportunity to study 1 Q6. What was the type of support or intervention that was implemented by Youth for the Future that had the greatest influence on your decision to return to school? Type of support N (multiple response) PTS 14 Technical training 2 IT 1 None 7 Q6. What kind of IYF support had the greatest influence on your return to education? Type of support Was a result of IYF support PTS 11 Technical training 2 IT 0 None 2 Total 15 Return to Education by Gender Yes, 56% No, 44% Q4. Was your return to the educational program as a result of the support and training provided by Youth for the Future? (n=27) 124 Return to Education by Gender 10 5 15 7 5 12 17 10 27 Male Female Total Q4. Was your return to the educational program as a result of the support and training provided by Youth for the Future? (Males= 17; Females = 10) Yes No Total 8 2 0 7 17 6 0 1 4 10 14 2 1 11 27 PTS Technical Training IT None Total Q6. What was the type of support or intervention that was implemented by Youth for the Future that had the greatest influence on your decision to return to school? (Males = 17; Females = 10) Male Female Total 0 0 1 1 0 5 7 1 2 0 0 1 1 5 1 2 1 1 1 6 12 A neighbor of mine (encouraged me) I aspire/want to complete my education I didn't find work, so I needed to take another course in order to find work I study in the same field as my work I was waiting for the opportunity to study My own decision Total Q4_1. What encouraged you to enroll in the educational program? (Males= 7; Females = 5) Total Female Male 125 Section 4: Labor Market This section is only applicable to respondents who reported they are “working” and “studying and working” (n=167) 15 10 25 2 0 2 17 10 27 Male Female Total Q7. Do you plan to continue your education? (Males = 17; Females = 10) Yes I don't know / Not sure Total 0 5 7 0 5 0 1 2 17 1 2 4 2 2 1 0 0 10 1 7 11 2 7 1 1 2 27 I like my specialization and would like to continue in it In order to find a good job In order to obtain a certificate It is beneficial for my future To develop myself / To learn new things To reach a goal To find a better job No reason Total Q7. Why yes? (Males = 17; Females = 10) Total Female Male 126 Employment Sector (as reported by respondent) Employment Sector by Gender IYF Support 63 41 16 12 10 7 5 5 4 2 1 1 Tourism Sales / Retail Service Industry Construction Military Maintenance Government / Public… Education Financial Media Oil and gas Number Sector of Employment (n=167) 24% 40% 8% 1% 6% 3% 1% 0% 8% 3% 5% 1% 25% 20% 20% 15% 5% 5% 5% 5% 0% 0% 0% 0% Sales / Retail Tourism Service Education Construction Government/ Public Sector Financial Media Industry Maintenance Military Oil and gas Q8. In which sector do you work? (Males = 147; Females = 20) Male Female 127 Duration of Employment and Intention to Continue Yes, 29% No, 71% Q9. Did you find this job opportunity as a result of the support from Y4F? (n=167) 51% 47% 10% 6% 6% 4% 4% PTS Technical training English language Career counseling Other IT None Q10. What was the support provided by Y4F that had the biggest influence on your job search? (n=49) 51.3% 31.1% 8.4% 3.4% 2.5% 0.8% 0.8% 0.8% 0.8% My self / My individual effort Through friends / family / relatives Through contacts Labour office Family-owned business Internet Through another VTC The CBO found me a job but I went into the military Through another CBO Q9_1. What helped you find this job opportunity? (n=118) 128 IYF Support by Gender 0% 10% 20% 30% 40% 50% 60% 70% Less than a month 1- 3 months 3- 6 months More than a year Q9 - Did you find work as a result of IYF? X Q11. Duration in current position? (n=167) Total Not as a result of IYF Found work as a result of IYF 0% 10% 20% 30% 40% 50% 60% 70% Plans to continue in position Does not plan to continue Don't know / Not sure Q9 - Did you find work as a result of IYF? X Q12. Do you plan to continue in this position? (n=167) Total Not as a result of IYF Found work as a result of IYF 129 Employment Duration by Gender 30% 70% 20% 80% Yes No Q9. Did you find this job opportunity as a result of the support from Y4F? (Males = 147; Females = 20) Male Female 32% 29% 1% 3% 3% 3% 40% 45% 10% 5% 5% 0% 5% 50% PTS Technical training IT training English training Career counselling Market Research Other Q10. What was the support provided by Y4F that had the biggest influence on your job search? (Males= 147; Females= 20) Male Femal 51% 32% 3% 1% 9% 3% 1% 0% 0% 50% 25% 6% 0% 6% 0% 0% 6% 6% My self / My individual effort Through friends / family / relatives Labour office Internet Through contacts Family-owned business Through another VTC The CBO found me a job at a hotel but I chose to go into… Through another CBO Q9_1. What helped you find this job opportunity? (Males = 103; Female = 16) Female Male 130 Section 5: Evaluation of PTS This section is only applicable to respondents who reported that they took the PTS training (n=319). 10% 13% 19% 58% 20% 20% 15% 45% Less than a month 1- 3 months 3- 6 months More than a year Q11. Since when have you been employed in this position? (Males = 147; Females = 20) Male Female 83 49 15 147 99 2 20 92 58 17 167 Yes No Don't know / Not sure Total Q12. Do you plan to continue in this position? (Males = 147; Females = 20) Male Female Total 131 Evaluation of PTS related to Gender Evaluation of PTS 7% 93% 5% 95% No Yes Q13. Did you benefit from participating in PTS? (Males = 210; Females = 109) Male Female 1 33 4 8 4 2 0 22 11 00 Attended similar trainings The training topics don't interest me Inadequate training period Weak trainers Other Topics don't suit the context Weak training topics Q14. Can you tell me why? (Males = 15; Females = 5) Male Female 132 Evaluation of PTS by Gender PTS Dimensions Q16. What were the factors from PTS that had a positive impact on you personally?* Very useful, 47% Useful, 39% Useful to some extent, 10% Not useful, 4% (No answer), 0% Q15. How do you evaluate the benefits from participating in PTS? (n=319) 42% 40% 12% 5% 0% 57% 35% 6% 3% 0% Very useful Useful Useful to some extent Not useful (No answer) Q15. How do you evaluate the benefits from participating in PTS? (Males = 210; Females = 109) Male Female 133 Dimension Yes / Answered in the positive Relationship with employers 47.0% Relationship with colleagues 86.8% Productivity 87.8% Constructive thinking skills 88.1% Creativity 89.7% Relationship with the community 89.3% Conflict management skills 91.8% Motivation 91.8% Commitment level 91.8% Relationship with parents 93.1% Entrepreneurial spirit 93.1% Assertiveness 93.4% Adaptability to change 94.4% Quest for self-improvement 94.4% Responsibility 94.4% Communication skills 94.7% Self-confidence 94.7% *These factors were randomized in order to eliminate first-mention bias PTS Dimensions by Gender Q16. What were the factors from PTS that had a positive impact on you personally?* Male Female Relationships with Employers 61% 20% Relationships with Colleagues 91% 79% Productivity 85% 93% Constructive thinking skills 84% 95% Creativity 86% 94% Relationships with Community 90% 89% Motivation 90% 94% Conflict management skills 90% 96% Commitment level 90% 94% 134 Relationships with Parents 92% 94% Entrepreneurial spirit 92% 95% Assertiveness 91% 97% Responsibility 93% 97% Adaptability to Change 93% 97% Quest for Self-improvement 93% 97% Communication Skills 94% 96% Self Confidence 94% 96% N 210 109 These factors were randomized in order to eliminate first-mention bias Section 6: CBO Evaluation This section is applicable to all respondents (n=352) CBO Services CBO Services n=78 IT center / Computer lab 28.2% Activities hall 15.4% Library 12.8% Other 6.4% Activities hall and IT/Computer lab 3.8% Computer lab / Library 3.8% Creativity room / workshop 3.8% Creativity room 2.6% No, 78% Yes, 22% Q17. Have you benefited from the youth support provided by your CBO through IYF? (n=352) 135 I benefited, but forgot from what 2.6% IT/Computer lab / Creativity room 2.6% IT/Computer lab / Library / Activities room 2.6% IT/Computer lab / Library / Creativity room 2.6% A session on accessories 1.3% Activities hall / Library / Creativity room 1.3% Activities hall and library 1.3% Communication skills session 1.3% Community service (for example, cleaning the mosque) 1.3% Soap-making course 1.3% Tailoring course in the activities hall 1.3% The COB supported me in getting a Cambridge Diploma 1.3% We took training sessions in the center 1.3% Youth forum 1.3% CBO Services by Gender Evaluation of CBO Services 83% 17% 67% 33% No Yes Q17. Have you benefited from the youth support provided by CBOs? (Males = 239; Females = 113) Male Female 136 Evaluation of CBO Services by Gender Very Useful, 38% Useful, 53% Useful to some extent, 6% Not useful, 3% Q18. What is your evaluation of those friendly services provided by your CBO? 44% 44% 10% 2% 32% 62% 3% 3% Very Useful Useful Useful to some extent Not useful Q18. What is your evaluation of those friendly services provided by your CBO? (Males = 41; Females = 37) Male Female 137 Evaluation of Local Employment Offices by Gender Reason for not using labor office Male Female I didn't know about it 33% 40% I am studying / I want to finish my education first 6% 14% I was expecting to find work through the CBO directly 1% 8% I live far away 3% 5% 1 1 1 0 4 2 16 2 1 2 1 1 1 1 1 0 1 1 0 1 3 0 0 0 1 4 4 17 1 0 5 0 1 0 0 0 1 0 0 1 0 2 A safe space for creativity Because it is a humanitarian act Discussion sessions about youth issues I benefited a lot from the training sessions I benefited from the computer lab / access to the internet I enjoy reading / Reading books and magazines I gained new skills / knowledge I gained new skills / knowledge (including communication skills) I got to know new people through the library I improved my computer skills I learned how to help others and serve my community I used to use it to search for jobs It was useful to the unemployed No benefit from it Others benefited from it, but not me There is no distraction or noise in the library; it gives me… Training sessions We benefited somewhat We didn't have access to it We held meetings to coordinate our activities, such as school… We used to spend our free time there Q18. Why? Female Male 66% 34% 71% 29% No Yes Q19. Have you resorted to the local employment office? (Males = 239; Females = 113) Male Female 138 Because I found work / Was already working 28% 5% I didn't need it 4% 5% I have been busy 6% 4% No particular reason; I will go 1% 4% I got engaged / married; I might not work 0% 4% Personal reasons: I had family issues to deal with; I didn't feel like it 3% 4% Because they don't offer good jobs 1% 3% I don't have a certificate for the training; I am waiting for it / I didn't complete the training 2% 3% My parents don't want me to go 0% 3% Not useful 5% 1% Because the I gained employment through the CBO immediately 5% 0% It needs "wasta" / I don't trust it 1% 0% N 158 80 Evaluation of Labor Offices Q20.1 – Why? Very Useful Useful Useful to some extent Not useful Total It found me a job 50% 24% 0% 0% 11% It didn't find me a job 0% 0% 42% 74% 45% Very Useful, 11% Useful, 22% Useful to some extent, 17% Not useful, 51% Q20. What is your evaluation of the services provided by those offices? (n=114) 139 The job that it found for me was unsuitable 0% 8% 16% 9% 9% It helps people find jobs 33% 56% 5% 0% 17% It is not transparent 0% 0% 0% 5% 3% They prefer experience and/or university degrees 0% 0% 11% 2% 3% They were very helpful 17% 4% 0% 0% 3% No suitable jobs 0% 0% 5% 3% 3% I obtained training through them 0% 4% 0% 0% 1% Limited number of opportunities 0% 0% 11% 3% 4% Poor reception of clients / citizen 0% 0% 0% 2% 1% Other 0% 0% 5% 2% 2% I applied through it and am waiting for a response 0% 4% 5% 0% 2% n 12 25 19 58 114 Evaluation of Labor Offices by Gender Section 7: Requested Crosstabs Status by Gender Q2. Which of the following applies to your current status? I am enrolled in an education program I am working I am studying and working I am not studying and unemployed Total Male 9 139 8 83 239 11% 23% 16% 49% 9% 18% 18% 55% Very Useful Useful Useful to some extent Not useful Q20. What is your evaluation of the services provided by those offices? (Males = 81; Females = 33) Male Female 140 Female 9 19 1 84 113 18 158 9 167 352 Status by Gender and Governorate Governorate Gender I am enrolled in an education program I am working I am studying and working I am not studying and unemployed Total Irbid Male 1 15 1 6 23 Female 2 2 0 2 6 Total 3 17 1 8 29 Ajloun Female 1 1 Total 1 1 Mafraq Male 1 6 3 6 16 Female 0 6 0 16 22 Total 1 12 3 22 38 Balqa Male 1 15 1 10 27 Female 2 3 0 16 21 Total 3 18 1 26 48 Amman Male 4 65 2 29 100 Female 4 6 1 29 40 Total 8 71 3 58 140 Zarqa Male 2 31 1 16 50 Female 1 1 0 10 12 Total 3 32 1 26 62 Madaba Male 4 1 5 Female 0 3 3 Total 4 4 8 Tafilah Male 2 9 11 Female 1 0 1 Total 3 9 12 Ma'an Male 1 6 7 Female 0 7 7 Total 1 13 14 141 Status by Training I am enrolled in an education program I am working I am studying and working I am not studying and unemployed Total Those who took PTS only 5 11 3 30 49 Those who took PTS and Technical Training only 4 22 2 21 49 142 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523