DISCLAIMER: This publication was produced at the request of the United States Agency for International Development (USAID). It was produced by Integrated International on behalf of the USAID Monitoring, Evaluation, and Learning Activity, implemented by The Kaizen Company. The evaluation was conducted by Henrik Schou Hansen (Team Leader), Farah Mahafzah (Evaluation and Gender TVET Specialist), Sari Hweitat (Private Sector Employment Specialist) Ayoub Al Nmour (Statistical Research Specialist), Leen Refai (Evaluation Specialist), and Zeid Qiblawi (Researcher). The contents of this report are the sole responsibility of The Kaizen Company, LLC and do not necessarily reflect the views of USAID or the United States Government. USAID MONITORING, EVALUATION, AND LEARNING ACTIVITY TRAINING FOR EMPLOYMENT ACTIVITY (TEA) END OF PROJECT EVALUATION REPORT November 29, 2021 ii | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV ABSTRACT The purpose of the Training for Employment Activity (TEA) was to train and employ youth in Jordan by upscaling an employment platform based on updated information about labor market needs, direct private sector links, and the ability to train job seekers for available vacancies. USAID selected three implementing partners, Luminus Technical University College, Education for Employment- Jordan, and Technical Vocational Training Academy, for the Activity. Using a mixed-methods approach, including primary and secondary qualitative and quantitative data collection through desk review, key informant interviews, group interviews, and survey of participants and employers, the evaluation was intended to inform how existing and future economic development programs could utilize lessons learned from TEA to incorporate successful approaches to meeting private sector demand for skilled and semi-skilled labor. The evaluation found that all four demographic targets (women, education level, Syrian refugees and marginalized groups, and poverty) were met or almost met by IPs. Similarly, some participants faced barriers in participation in the training and especially in joining the labor market. Nevertheless, the evaluation found that TEA was relevant to both the beneficiaries as well as the employers. Of the beneficiaries surveyed, almost a third were unemployed at the time of the survey. The men in employment had high sector and job mobility whereas the women less so. Finally, although the implementing partners did not believe this Activity could be conducted without donor support, more than 40% of the employers said they would pay for training job seekers under the right circumstances. iii | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV TABLE OF CONTENTS ABSTRACT ii TABLE OF CONTENTS iii LIST OF ACRONYMS iv EXECUTIVE SUMMARY v BACKGROUND AND PURPOSE v EVALUATION QUESTIONS, METHODOLOGY v FINDINGS & CONCLUSIONS vi RECOMMENDATIONS viii 1. EVALUATION PURPOSE & QUESTIONS 1 1.1 EVALUATION PURPOSE 1 1.2 EVALUATION QUESTIONS 1 2. PROJECT BACKGROUND 2 2.1 CONTEXT 2 2.2 OVERVIEW OF TRAINING FOR EMPLOYMENT ACTIVITY 3 2.3 SCOPE OF WORK AND IMPLEMENTATION CONDITIONS 4 3. EVALUATION METHODS & LIMITATIONS 6 3.1 DESK REVIEW 6 3.2 DATA COLLECTION 7 3.3 SEMI-STRUCTURED KEY INFORMANT INTERVIEWS (KII) 9 3.4 GROUP INTERVIEWS (GI) 9 3.5 LIMITATIONS 9 4. FINDINGS AND CONCLUSIONS 10 4.1 EVALUATION QUESTION 1 11 4.2 EVALUATION QUESTION 2 16 4.3 EVALUATION QUESTION 3 21 4.4 EVALUATION QUESTION 4 29 4.5 EVALUATION QUESTION 5 31 4.6 EVALUATION QUESTION 6 34 5. RECOMMENDATIONS 37 ANNEXES 39 ANNEX 1: STATEMENT OF WORK 40 ANNEX 2: INCEPTION REPORT 57 ANNEX 3: ANALYSIS DESIGN AND METHODS 96 ANNEX 4: INFORMATION SOURCES 102 ANNEX 5: DATA COLLECTION TOOLS 106 ANNEX 6: EVALUATION TEAM MEMBERS 119 iv | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV LIST OF ACRONYMS AMELP Activity Monitoring, Evaluation, and Learning Plan CAQA Center of Accreditation and Quality Assurance CBO Community Based Organizations DOS Department of Statistics FGD Focus Group Discussion FTE Full-time employment FY Fiscal Year GI Group Interview GoJ Government of Jordan ILO International Labour Organization IP Implementing Partner EFE-Jordan Education for Employment -Jordan KII Key Informant Interview LOP Life of Project LTUC Luminus Technical University College NGO Non-Governmental Organization OJT On-the-job Training SOW Statement of Work TEA Training for Employment Activity TOC Theory of Change TVET Technical Vocational Education and Training TVTA Technical Vocational Training Academy UNHCR United Nations High Commission for Refugees UNRWA United Nations Relief and Works Agency for Palestine Refugees in the Near East USAID United States Agency for International Development v | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV EXECUTIVE SUMMARY BACKGROUND AND PURPOSE Jordan’s population more than doubled from 2004 to 2019, reaching over ten million people. With nearly two-thirds of the population under the age of 30, the ability of young people to access entry￾level jobs and gain life and work skills is critical to Jordan’s long-term economic growth and security. However, youth in Jordan face many challenges, including limited job opportunities and an education system that does not provide them with the skills and knowledge that match the needs of the labor market. Over a third of youth between 15 and 24 are unemployed, which hinders the experience needed for future employment and limits opportunities for social, economic, and political participation. The purpose of the Training for Employment Activity (TEA) was to train and employ youth in Jordan by upscaling and sustaining an employment platform based on timely information about labor market needs, direct links with the private sector, and the ability to train job seekers for available vacancies. By removing supply-side labor market barriers and responding to market demand for an improved labor force, which can simultaneously propel private sector development and expand access to employment, TEA contributed to USAID/Jordan’s Development Objective 1: “Broad-based Inclusive Economic Development Accelerated” and particularly Intermediate Result 1.2: “Economic opportunities increased including for women and vulnerable groups.” This evaluation was intended to learn from this modality of using three commercial contracts to match labor supply with demand that may inform USAID’s future interventions relative to workforce development. This report presents the findings, conclusions, and recommendations of the team commissioned by USAID to conduct an end-of-project evaluation of the Training for Employment Activity, implemented by three implementing partners (IPs): Education for Employment - Jordan (EFE￾Jordan) – contract number 72027218C00003; Luminus Technical University College (LTUC) – contract number 72027218C00004; and The Technical Vocational Training Academy (TVTA) – contract number 72027218C00005. EVALUATION QUESTIONS, METHODOLOGY The evaluation team was requested to answer the following evaluation questions: 1. To what extent did TEA achieve its intended outcomes (i.e., beneficiaries trained by gender, nationality, education level, and those from poverty pockets or marginalized groups)? How was TEA inclusive in reaching target groups of beneficiaries (e.g., Jordanian and Syrian, men and women, those who did not pass Tawjihi, those with limited financial resources or from poverty pockets, and marginalized groups) residing in Jordan? What are the key barriers to the participation of women and other marginalized groups? 2. To what extent were TEA interventions broadly relevant to the needs of target beneficiaries and local employers? More specifically, were the training courses relevant to local market needs, considering the perceptions of both youth and employers? 3. What were the rates of employment and retention for TEA trainees by gender, nationality, financial status, and sector of training – both within fixed periods (e.g., at least three months, at least six months) and as trends over time? Are there differences in employment and retention by category? What factors are likely driving any observed differences? 4. How likely are TEA interventions to be sustained over time? Why is this the case? 5. What are the key lessons learned about TEA design and implementation successes and challenges which can be applied to future workforce development interventions? vi | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV 6. What are the key approaches, methods, or interventions that can better contribute to improving employment and retention rates? The evaluation used a mixed-methods design that collected both qualitative and quantitative data to address the evaluation questions, including document review, key informant interviews (KIIs), group interviews (GI), and surveys of participants and employers. Data collection took place between June 30 and August 5, 2021, and consisted of: • The USAID and IPs’ document review to better understand approaches to implementation. • Phone surveys with TEA trainees (398) and their employers (293) to better understand results reported through TEA performance indicator data and to inform evaluation questions. • Key informant interviews (KIIs) to capture perspectives and activities of each informant category, revolving around the evaluation questions relevant to each stakeholder. • Group interviews (GIs) with TEA beneficiaries to explore in-depth qualitative issues arising from the quantitative data gathered from surveys. Analysis utilized constant comparative analysis (grounded theory) to identify themes and categories based on emerging trends in the data. Analysis included triangulation, thematic analysis, statistical analysis, and gender analysis among sources of qualitative and quantitative data, to deliver higher validity to the findings. The evaluation utilized patterns in responses and the major themes that emerged and related them to the evaluation questions to draw conclusions and recommendations. FINDINGS & CONCLUSIONS Question 1: While TEA IPs made significant efforts to reach the target groups of beneficiaries (e.g., Jordanian and Syrian, men and women, those who did not pass Tawjihi, those with limited financial resources or from poverty pockets, and marginalized groups), they met their quotas with varying success. While the combined efforts of the three IPs fulfilled or almost fulfilled the four intended outcomes, EFE-Jordan was the only IP to fulfill or come close to fulfilling its sex, refugee, marginalized groups and Tawjihi targets, while LTUC fulfilled or came close to fulfilling all but its marginalized groups target, and TVTA only fulfilled its sex target. In the aggregate, however, the combined TEA IPs met targets for women’s inclusion and came within two and half percentage points of fulfilling the refugee, education and the marginalized groups, and poverty targets. In exploring barriers to participation in training as well as employment, the majority of respondents (93.5% of surveyed trainees) said they did not face significant barriers. Whereas a relatively small proportion of the participants found barriers in attending the training (6.5% of surveyed trainees), a much more significant proportion faced obstacles when seeking employment (25.6%). The most commonly cited obstacle for attending the training was transportation, whereas the lack of employment opportunities was the most significant barrier for employment. Of women surveyed, 35.3% said they faced barriers in finding employment which was more than twice that of men surveyed. While women’s barriers to training and employment participation related mainly to personal, cultural and transportation issues, these barriers did not prevent the IPs from fulfilling their targets of women’s participation. Similarly, shortfalls in reaching Syrian refugee targets are related to limited sector alignment with sectors of work permit permission for Syrians, as well as competition from other donor-supported social networks at odds with Syrians seeking formal or long-term work. Question 2: While the IPs took different approaches to adapt their training to the needs of local employers, most conducted a market assessment, used knowledge of the market, and/or consulted with employers to develop training programs. TVTA and LTUC developed tailor-made courses for vii | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV two employers, founded on existing courses, while EFE-Jordan designed tailor-made courses based on market needs assessment for the digital economy. The training institutions that conducted the training were selected based on their geographical distribution to ensure access to training in all locations. TVTA identified geographical demand for specific employees and conducted training to fulfill these demands. To ensure commitment from the employers, TVTA and EFE-Jordan asked prospective employers to interview the trainees before they started their training. Overall, 69.7% of the employers were consulted on their labor needs, and 55% of the employees said that the training prepared them for the labor market to a great extent. An analysis of the tracking sheets shared by TEA of the full beneficiary trainee population indicated that 37.6% of all trainees found employment. Furthermore, 67% of the surveyed beneficiaries found that the soft skills training had been helpful in their job to a great extent. Of the surveyed employers, it was found that 51.7% found the skill level of the trainees satisfactory to a great extent and 40.4% to a medium extent. Question 3: An analysis of the participants surveyed found that 67.6% were employed at the time of the survey. It was found that women were more than twice as likely to be unemployed than men (16.7% vs. 34.8%) and that a considerable proportion of women trained (12%) had left the labor market and were no longer seeking employment. Men were more likely to seek new employment to improve their salary or working conditions, whereas women were more likely to stay with the same employer. General job mobility was relatively high, with 25.6% moving from the first job they got after training. When asked why they stayed in the same position, the most common answer was that 1) they liked the job; 2) the employer was conveniently located; and 3) the salary was good. However, when surveying the trainees who had moved jobs, a low wage was the second most common explanation for men, although it was not mentioned by women. The most common reason not to stay employed was that jobs were not available, which may have become more prevalent with the COVID pandemic’s influence on the economy. In the high poverty incidence governorates, unemployment was higher (more than 50%), but all surveyed were looking for employment. Among Syrians, none stayed in the same job, and more than half were unemployed but not seeking employment. Most of the trainees were trained in traditionally female-dominated sectors, such as the garment industry or beauty and hair. Incidentally, these were also the sectors that had the highest proportion of unemployed trainees. In the data sample, there were 97 trainees from the garment industry; 59 were unemployed at the time of the evaluation. Looking at employer/employee retention, most trainees seem to have remained employed with the same employer after graduation. Employer/employee retention was particularly evident in money exchange, where up to 85% of trainees remained employed with the same employer that they were placed with, which could be indicative of a lower number of similar employers and therefore increased employer-loyalty, the training received, as well as the work environment/salary. In sectors such as beauty and hair, automotive repair, medical and health service, administrative office work, the majority (76%-90%) of trainees remained employed in the sector, but not necessarily with the same employer, perhaps due to the transferability of skills in demand and the freelance nature of the sector. Question 4: From the KIIs with the IPs, there was a general agreement that the interventions under TEA could only be implemented with support from a donor. All three IPs had already implemented similar interventions for other donors, and there was not much pressure to go out and find alternative funding. While IPs highlighted that financial sustainability of TEA interventions can only be supported with donor support, 40% of employers interviewed indicated some willingness to pay for training, demonstrating more private sector willingness to pay for this kind of training than the IPs realized. Of viii | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV employers unwilling to pay, 83.7% said they did not have the budget for this kind of training. When asked under what circumstances they would pay, 44.2% of employers said that it would depend on the cost, 43.3% said if tailor-made courses for what the company needed could be taught. However, even with private sector interest, the IPs will most likely continue to concentrate on donor-funding. Question 5: TEA IPs in aggregate were successful in recruiting, engaging, and employing marginalized groups, especially women. Efforts made to address cultural barriers faced by women among their families were helpful to the extent that the TEA trainee population exceeded its female target with each IP. However, TEA IP recruitment outcomes were not the same, and not all hit the required sub￾population targets. It was found that the targeting of beneficiaries was largely left to the individual IP which created some issues in the interpretation of targets. An additional challenge was that the poverty target was combined with marginalized groups which diluted the poverty focus of the Activity. The evaluation found no indication that the IPs targeted poverty pockets. Defining and adhering to a poverty target might have added more attention to this target and improved the targeting by the IPs. It was found that ensuring that skills developed are market-driven was critical to success of the Activity. Collaboration with employers also proved to be a successful approach to employment and retention, and this was achieved based on both needs assessment, and employer consultation/collaboration through approaches such as customized training, trainee interview/matching, and consultation on needed skill sets. TEA IPs had success in working with large employers. However, there is a need to constantly examine market and sectoral trends, to establish continuing demand, to assess skill sets needed, and to ensure an adaptable approach to both long-term and freelance/gig labor markets. Question 6: Central to TEA and any other similar training program is the labor market assessment conducted to identify possible areas for employment and training. The three IPs took very different approaches in assessing the labor market. LTUC opted to use an assessment conducted previously, and EFE-Jordan conducted a mini-assessment. TVTA carried out a running assessment where it identified opportunities based on geographical supply shortages. A clear definition of what is required from a labor market assessment would support a more uniform approach in the future. A key approach that USAID could utilize in future programing is a focus on more flexibility in employment options. The Activity focused on vocational courses and therefore left little room for alternative employment options such as self-employment, informal part-time work or job-sharing (all options that appeal to women). KII discussions with women showed that they were seeking alternative working arrangements such as informal part-time work, job-sharing and home-based working arrangements. As the labor market and workplace evolve with both the influence of the COVID pandemic on labor movements from formal to informal as well as the currently unregulated digital economy in Jordan, there is a need to have a more flexible definition of employment. Home-based, informal, freelance and “gig” and self-employment do not neatly fit with the criteria for employment used by TEA,1 potentially inadvertently discounting both gig/freelance and women’s informal employment and encouraging IPs to focus on persons and jobs that more easily fit TEA criteria. RECOMMENDATIONS 1 Indicator 1.2.a notes “Only employment lasting three months or more is counted under this indicator.” Indicator 1.2.1.a. requires proof of employment in the “form of a contract or agreement letter that includes employment period, duties and benefits (salaries, bonuses, or commissions). The document should clearly state that named employer will employ a skilled beneficiary in a specific job.” These criteria do not align with home-based, freelance, informal, digital and self-employment. ix | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV 1. Consider building synergies between IPs and replicating proven curricula across IPs, where possible, especially in relation to soft skills. 2. Continue to target marginalized groups such as women, with an increased focus on better targeting of poverty pockets. Continue to utilize effective approaches to local outreach and address cultural barriers among families to encourage female labor force participation across sectors. Consider a differentiated approach to recruitment of Syrians that better aligns with work permits and coordinates with refugee social safety nets, as well as persons from poverty pockets by defining and monitoring poverty targets. Where successes are found, consider replicating among other IPs. 3. Should future programming engage multiple implementers, ensure that recruitment implementation criterium be equally enforced for all IPs to ensure a level playing field when promoting employment opportunities and recruiting trainees, in order to ensure attainment of subpopulation targets. Consider extending contractual target-meeting requirements to recruitment targets, in the same manner as the employment targets. Consider improved oversight and unification of database tracking among IPs. 4. Continue consultation and collaboration with private sector employers at the earliest stages. Consider requiring customization of all training in consultation with employers and trainee interviews as a best practice to inform curricula, skills in demand, and sectoral and geographic market demand to maximize employment and retention outcomes. 5. In future activities, a USAID labor market assessment should ideally clearly define the supply side shortages in the labor market and challenges in fulfilling these. To mitigate differentiated approaches by IPs in the future, ensure that market assessments are more clearly defined, and standards are met, and ensure that such assessment of market needs, trends, and fluctuations is adapted into training curricula, employer engagement, and job placement. 6. The Activity focused on providing long-term employment in the labor market. However, some of the trainees, especially women, had difficulties in this kind of employment. For future activities, explore the differences between the formal and informal markets, the opportunities for entrepreneurship or self-employment, and digital and freelance markets. For women, consider an alternate form of employment criteria, in recognition of the gender roles women take on that may prevent them from being available for long-term employment under TEA criteria. Consider creating opportunities for flexible employment and internships in order to support improved integration of women into the labor force in a country with female labor market participation below 15%. Similarly, consider instituting a micro-loan program that could help women establish home-based enterprises based on skills acquired at TEA IPs, in line with global evidence in technical vocational education and training (TVET). 7. As a strategy for regular job placement, continue engaging large and sectoral employers with the same training needs (such as garments). As a strategy for increased reach outside of Amman and outside of large employers, consider systematically mapping employers by sector, geography, and the number of employers in order to better inform job placement and regional market demand. 8. To ensure higher rates of initial employment retention, consider instituting stipend and transportation costs, as well as on-the-job training to mitigate challenges related to employees not staying employed. To ensure continued tracking of the trainees, this stipend payment could be tied to a mobile payment platform that meant that only trainees that kept or updated their phone numbers could continue to receive this payment. 9. In future iterations of TVET programming, consider tracking promotion of trainees to inform and assess training courses that are linked to performance (promotion, retention, salary grade) of graduates, which in turn could support assessment of the quality of implementation. Similarly consider utilizing tracking of dropouts/unemployed to identify areas of improvement. 1 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV 1. EVALUATION PURPOSE & QUESTIONS 1.1 EVALUATION PURPOSE This report presents the findings of the end-of-project evaluation of the Training for Employment Activity (TEA). TEA was implemented by three implementing partners (IPs): • Education for Employment - Jordan (EFE-Jordan) – contract number 72027218C00003; • Luminus Technical University College (LTUC) – contract number 72027218C00004; and • The Technical Vocational Training Academy (TVTA) – contract number 72027218C00005. USAID intends to use the findings of this evaluation to ascertain TEA's effectiveness as it relates to beneficiaries' and employers' engagement, training courses, skills development, employment, and employment retention. The best practices and lessons learned through the evaluation are intended to help USAID learn from this modality of using three commercial contracts to match labor supply with demand that may inform future workforce development interventions in Jordan, particularly related to TEA's approaches and methods. In addition, it is envisioned that the findings may be helpful to relevant entities in the Government of Jordan (GoJ) and bilateral and multilateral development partners working in the area of youth, livelihoods, and vocational skills development. 1.2 EVALUATION QUESTIONS Following the TEA Evaluation Statement of Work (SOW), the evaluation sought to answer the evaluation questions listed below (with clarification in italic2): 1. To what extent did TEA achieve its intended outcomes (i.e., beneficiaries trained by gender, nationality, education level, and those from poverty pockets or marginalized groups)? 1.1. How was TEA inclusive in reaching target groups of beneficiaries (e.g., Jordanian and Syrian, men and women, those who did not pass Tawjihi, those with limited financial resources or from poverty pockets, and marginalized groups) residing in Jordan? This question intends to describe the various approaches that IPs took to reach the different target groups (e.g., outreach, accessibility of training) 1.2. What are the key barriers to the participation of women and other marginalized groups? "Participation" may include training as well as employment and retention. 2. To what extent are TEA interventions broadly relevant to the needs of target beneficiaries and local employers? 2.1. More specifically, were the training courses relevant to local market needs, considering the perceptions of both youth and employers? These questions seek to compare the needs identified through IP market analyses with the training offered (e.g., required skill level, soft skills) and identify and compare the reasons given by youth and employers for employment retention vs. leaving employment. 3. What are the rates of employment and retention for TEA beneficiaries by gender, nationality, financial status, and sector of training – both within fixed periods (e.g., at least three months, at least six months) and as trends over time? Are there differences in 2 Consultation was conducted with USAID on June 7, 2021 to clarify evaluation questions. 2 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV employment and retention by category? What factors are likely driving any observed differences? The purpose of this is to consolidate information on 3-month and 6-month employment retention collected through IP tracking sheets to enable trend analysis and multivariate analysis. 4. How likely are TEA interventions to be sustained over time? Why is this the case? This question attempts to identify whether TEA's approaches are recognized as having sufficient value to employers or GoJ for them to provide financial support to continue to fund similar interventions. 5. What are the key lessons learned about TEA design and implementation successes and challenges which can be applied to future workforce development interventions? 6. What are the key approaches, methods, or interventions that can better contribute to improving employment and retention rates? 2. PROJECT BACKGROUND 2.1 CONTEXT Figure 1. Female and Youth (15-24 year) unemployment and female labor market participation From 2004 to 2019, Jordan's population more than doubled, reaching over ten million people. With nearly two-thirds of the population under the age of 30, the ability of young people to access entry￾level jobs and gain life and work skills is critical to Jordan's long-term economic growth and security. However, youth in Jordan face many challenges, including limited job opportunities and an education system that does not provide them with the skills and knowledge that match the needs of the current labor market. As seen in Figure 1,3 Jordan has had youth unemployment above 30 percent over the last two decades. With an economic downturn since 2016, unemployment in general and youth unemployment, in particular, has gotten worse, peaking during the COVID-19 pandemic, when youth 3 International Labour Organization, ILOSTAT database. Data retrieved on June 15, 2021 29.3% 32.4% 31.6% 28.5% 30.0% 28.5% 35.7% 37.3% 12.6% 12.9% 12.5% 15.1% 15.3% 13.9% 14.6% 21.0% 20.8% 25.9% 19.9% 22.6% 27.2% 24.1% 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0% 35.0% 40.0% 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 Youth Unemployment Femal labor market participation Female unemployment 3 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV unemployment reached an unprecedented 50 percent.4 Jordan also has one of the lowest female labor market participation rates in the world. According to statistics from the ILO,5 the female labor participation rate in Jordan has consistently been below 15 percent (only Yemen and Iraq have a lower female labor market participation) and Jordan is ranked 133 out of 156 countries in female economic participation and opportunity according to the World Economic Forum.6 Moreover, even with most women out of the labor market, around a quarter of the women in the labor market were unemployed. With high youth unemployment, low female labor market participation, and the relatively high rate of female unemployment, a whole subsection of society has been economically excluded. Therefore, vocational training should play a vital role in empowering youth, especially females, and supporting them to become more independent, self-reliant, and resilient, thus enhancing their economic inclusion in society. TEA tried to build this resiliency, independence, and self-reliance in young people by bridging the gap between training in vocational, technical, and higher education institutions and the actual needs of the labor market in Jordan. 2.2 OVERVIEW OF TRAINING FOR EMPLOYMENT ACTIVITY The purpose of TEA was to train and employ youth in Jordan by upscaling and sustaining an employment platform based on updated information about labor market needs, direct links with the private sector, and the ability to train job seekers for available vacancies. Table 1. TEA Implementing partners Implementing Partner Contract Amount Contract Period Service Contracted Luminous Technical University College (LTUC) $4,672,423.00 June 2018 – June 2021 2,250 student semesters of career and technical education* (1074 youth) Technical Vocational Training Academy (TVTA) $4,770,000.00 July 2018 – July 2021 2,500 youth trained Education for Employment - Jordan (EFE-Jordan) $4,061,576.00 July 2018 – July 2021 2,500 youth trained * Due to LTUC's status as a college, the contract states semester hours; however, USAID agreed that this would translate to 1,074 youth. TEA THEORY OF CHANGE (TOC): Expanding access to employment and responding to market demand for an improved labor force will directly contribute to Development Objective 1: "Broad-based Inclusive Economic Development Accelerated" by removing supply-side labor market barriers to private sector development and growth. Sustainable employment within the private sector can be achieved by equipping youth with skills and by sparking employer willingness to hire trained youth. The TOC for TEA is further expanded through the logical framework7 of the Activity. As seen in Figure 2, the Log Frame has three distinctive 4 World Bank, "Jordan Economic Monitor - Uncertain and Long Trail Ahead", Spring 2021, p.11 5 International Labour Organization, ILOSTAT database. Data retrieved on June 15, 2021 6 World Economic Forum, "Global Gender Gap Report 2021", p. 31 7 The logical framework is recreated from the Log Frame provided in the SOW 4 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV tracks of interventions, one being training-related, the second being employment-related, and finally, the third dealing with the tracking interventions. Figure 1. Logical Framework for the Training for Employment Activity 2.3 SCOPE OF WORK AND IMPLEMENTATION CONDITIONS TEA conducted its work through three Jordanian IPs. For funding level, contract periods, and target number of participants of the three IPs, please refer to Table 1. According to TEA documents, each IP was to work according to the same Scope of Work, conditions, tasks, and requirements: • The semester hours will include both short-term and long-term programs; • Beneficiaries are students with limited financial resources and above-average academic standings; • At least 50% of beneficiaries are students who did not pass Tawjihi; • At least 30% of beneficiaries are females; • At least 70% of beneficiaries are from poverty pockets and marginalized groups residing in Jordan; • Target students: 80% Jordanians and 20% Syrians. In addition, the contracts stipulated that IPs meet the following conditions: • At least 80% of the beneficiary students will have secured employment, guaranteed by the offerors from their own funding. • At least 60% of the beneficiary students will still have secured employment after six months of employment. 5 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV • The offeror must be a training facility recognized by the Ministry of Higher Education or the Ministry of Labor. • The offeror must be able to issue certificates to the beneficiaries at the end of the program. The IPs were required to conduct the following tasks: • Assess labor market needs, develop employer partnerships, and secure job commitments. • Customize training program design and curriculum to meet employer partner needs. • Recruit & select student participants. • Deliver career and technical education training programs. • Match training program graduates with job opportunities. • Track beneficiary job placement and retention. The payment modalities for TEA represent a departure from typical USAID contracting practices as payment was performance-based. The payment modalities incentivized IPs to achieve higher-level outcomes (rather than only meet lower-level outputs), allowing them the freedom to utilize their existing relationships within local communities and exercise and modify approaches based on their experience. For each beneficiary, the IP was paid: • 50% payment upon providing training; • 25% payment upon placing the student in a job; and • 25% payment after six months of employment retention. Although all three IPs had the same SOW, each pursued a somewhat different approach to achieving the objectives of TEA. Table 2 below shows the differences and similarities in their approaches. This information on the different implementation approaches was collected through KIIs with implementing partners and through their progress reporting. Table 2. Implementing partners' implementation of the SOW TVTA EFE-Jordan LTUC Target geography Amman, Aqaba, Balqa, Irbid, Madaba, Mafraq, and Zarqa Ajloun, Amman, Aqaba, Irbid, Jerash, Tafilah, and Zarqa Amman, Balqa, Irbid, Jerash, Madaba, and Zarqa Target beneficiary selection criteria 1. Syrian refugees unable to complete education 2. Disadvantaged and unemployed Jordanian youth 3. College and university graduates seeking jobs 4. 30% female participation 1. 50% of beneficiaries did not pass Tawjihi 2. Age 18-30 3. Limited financial resources 4. Unemployed in the formal economy for at least two months 5. 30% female participation 1. 50% of beneficiaries did not pass Tawjihi 2. At least 30% of beneficiaries females 3. At least 70% of beneficiaries from poverty pockets 4. 20% of beneficiaries Syrian Employer engagement approach TVTA is known in the automotive industry. Visited all significant employers in the geographic target area to determine the labor need. They customized training courses for certain employers (e.g., Sameh Mall/Kareem Hypermarket). They provided soft skills training for newly recruited employees. EFE-Jordan conducted a mini￾market assessment before training. Reached out to employers and signed support letters. Engaged employers in materials development. Engaged employers in one-to￾one interviews before course enrollment. Engaged employers in mini-job fairs to link them with the training graduates. LTUC relied on mass recruitment. However, with COVID-19, they focused on one-to-one interviews. They used previous labor market assessment in identifying market demand. Visited employers to identify possible labor needs and demands for training (hospitality was added due to market needs). Is currently conducting a new national labor market assessment. 6 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV TVTA EFE-Jordan LTUC Sectors Teacher rehabilitation, automotive, silver crafting, hospitality, office, hair & beauty, cell phone repair, garment, retail, money exchange Garment, retail, hospitality, hair & beauty, call center, office, sales & customer service Hospitality, retail, HVAC, hair & beauty, garment, hybrid automotive, fashion, medical services Outreach for recruiting participants Community based organizations (CBOs). Social media: WhatsApp and Facebook. Employers' newly hired staff. Job fairs and career days. CBOs. Social media: Facebook. Jordanian Hashemite Fund for Human Development (JOHUD). UNHCR for Syrians. Employers newly hired staff (soft skills). Nongovernmental organizations (NGOs) and CBOs (going through UNRWA, UNHCR to reach Syrians). Social media: Facebook and Instagram. Television. Training in addition to technical Employability, life skills, health & safety, communication for middle management English, career readiness, job search skills, soft skills Two English language levels, life skills, career readiness, computer skills. Training certification Center of Accreditation and Quality Assurance (CAQA) Center of Accreditation and Quality Assurance (CAQA) Center of Accreditation and Quality Assurance (CAQA) Training duration Two weeks to three months. Six months for automotive and garment. One to three months technical training. Two weeks soft skills training. Diploma: two years Long term: three semesters Short term: three to six months of training Training modalities Two weeks soft skills at the training center, followed by on￾the-job training (OJT). Training for middle-management to advocate for a healthy work environment Soft-skills training for two weeks prior to any course. Theoretical as well as practical training. OJT for the garment sector Developed training materials. Contracted training centers to deliver training. Provided existing as well as customized courses (customized for Profix and Classic combined with OJT). Diploma students had OJT with Jordan Electrical Power Company (30 training opportunities for students who may be placed in positions). 3. EVALUATION METHODS & LIMITATIONS The TEA evaluation used a mixed-methods design that collected both qualitative and quantitative data to address the evaluation questions, including document review, key informant interviews (KIIs), focus group discussions (FGDs), group interviews (GI), and surveys of participants and employers. Data collection took place between June 30 and August 5, 2021. 3.1 DESK REVIEW The document review was foundational to developing an understanding of the approaches of the IPs and results achieved prior to data collection. The document review for this evaluation served as the basis for designing a data collection plan, data collection tools, and planning data analysis. Primary documents for this evaluation were as follows: • Activity Monitoring, Evaluation, and Learning Plan (AMELPs) • DevResults indicator data • Participant and employer tracking sheets • IP progress reports • TEA gender analysis 7 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV Several donor and sector reports on workforce development programs and labor market needs were also reviewed. 3.2 DATA COLLECTION Table 3. Survey sample by sex, nationality and geographical location Category Respondents (#) 398 % of Grand Total EFE￾Jordan LTUC TVTA 31% 35% 34% Female 109 61 38 52% Male 14 77 99 48% Jordanian 119 130 134 96% Syrian 3 5 2 3% Other nationalities 1 0 4 1% Ajloun 40 0 0 10% Amman 23 77 96 49% Aqaba 2 1 0 1% Balqa 4 8 1 3% Irbid 2 10 15 7% Jerash 6 2 2 3% Karak 15 1 8 6% Ma'an 0 15 5 5% Madaba 0 2 0 1% Mafraq 0 1 0 0% Tafilah 26 0 0 7% Zarqa 5 21 9 9% Governorates with a high rate of poverty incidence 3.2.1. Data collection from participants – Phone surveys were conducted with TEA graduates to investigate and understand results reported through TEA performance indicator data and to inform the following points in the evaluation questions: • Barriers to participation and employment of women and marginalized groups. • Steps that TEA took to be inclusive of reaching women and marginalized groups. • Relevance of TEA's training to employers relative to their expectations of employment development training, market needs, and employment. • Relevance of TEA’s training to youth needs, relative to their expectations of employment development training, and employment. • Factors contributing to TEA graduates remaining employed, and differences among demographic variables. The sampling method adopted for the survey was Proportional Stratified Random Sampling. The survey sample size was calculated using a simple random sample of the entire TEA beneficiary population. The sample population was then evenly distributed among the three IPs and was further stratified based 8 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV on demographic characteristics proportionate to those observed under the specific IP. This ensured quantitative generalizability for respective results at a confidence level of 95% and a 5% margin of error. The participant surveys were slightly oversampled (increased from 120 to 123-137 per IP, eventually totaling 398 employed beneficiaries with completed surveys) to guarantee specific group representation as follows: • Governorates • At least 30% of sample per IP • Male/Female • Poverty pockets • Refugee graduates This approach enabled a corresponding employer sample with 298 responses from supervisors at the employer organizations, such that no employer supervisor completed more than one survey. For the participant survey sample distribution, please refer to Table 3. 3.2.2. Data collection for employers – Phone surveys were conducted with employers of TEA graduates to inform the following points in the evaluation questions: Table 3. Surveyed employers by Governorates Governorate Total # of surveyed employer respondents Amman 193 Aqaba 5 Balqa 18 Irbid 21 Jerash 2 Karak 1 Ma’an 9 Madaba 3 Mafraq 4 Tafilah 3 Zarqa 39 Total 298 • The extent to which interventions are relevant to local employers. • The employer perceptions of job training programs and the quality of the beneficiaries. • Observation of factors that lead TEA graduates to retain or leave jobs. • The employer willingness/ability to pay to train youth that meets their employment needs; and • Recommendations on how a program like TEA could better meet their needs. The employer's survey sample was drawn to match the sample drawn for the participant survey; the method used was Proportional Stratified Random Sampling. Surveying employers of graduates sampled for the participant survey a) brought uniformity; b) ensured diversity of employers across different parameters in relation to the demographics of their employees; and c) created the conditions for a new, deeper analysis level, in which employer results could be studied in relation to participants' results. For further details on the sampling for the data collection, please refer to Annex 3. 9 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV 3.3 SEMI-STRUCTURED KEY INFORMANT INTERVIEWS (KII) Semi-structured KIIs were conducted to capture perspectives and interventions under the evaluation questions as relevant to each stakeholder. The KII guides were designed to allow a relatively free￾flowing conversation while creating a standardized format to facilitate reliable, comparative analysis and triangulation of data across respondent types and evaluation questions. While interview questions were based on the evaluation's overarching questions, they were formulated to suit the informants and allow for probing as areas of importance and themes arise. Key informant interviews were conducted with USAID staff central to TEA, Ministry staff within the area of vocational training, IP staff from each of the three IPs, and 21 employers who had employed beneficiaries from TEA. For a detailed list of all KIIs conducted, please refer to Annex 4. 3.4 GROUP INTERVIEWS (GI) GIs (with a maximum of 5 attendees) were conducted with participants to explore in-depth qualitative issues arising from the quantitative data gathered from surveys. To ensure this triangulation, the GIs were conducted at the end of the period of surveying. To foster an open discussion, the groups were homogenous; each group was single-sex, from the same IP, and from the same geographic location. The GIs were conducted to inform the following points in the evaluation: • Barriers to participation and employment of women and marginalized groups; • Steps that TEA took to be inclusive of reaching women and marginalized groups; • Relevance of TEA's training to employer relative to their expectations of employment development training and market needs; • Relevance of TEA’s training to youth needs, relative to their expectations of employment development training and employment; and • Factors contributing to TEA graduates remaining employed, and differences among demographic variables. 3.5 LIMITATIONS “The program has been calling me non-stop since 2018. A colleague of mine changed her phone number because of the program follow-up calls. I am sorry but you have to work on that.” - Female employer in the South 3.5.1. Assessment fatigue – Because of the funding disbursement modalities of the TEA, where funding to the IPs was released based not only on the training conducted but also on the beneficiary being employed and consequently remaining employed for six months, an extensive system of beneficiary tracking, and data verification was established. Each IP tracked each beneficiary to ensure they found a job and remained employed for at least six months; this was verified by the USAID COR and/or USAID’s Monitoring, Evaluation, and Learning Activity by contacting a considerable sample of beneficiaries. The sample pool of employed beneficiaries used for this evaluation was relatively small and similar to the sample pool used for ongoing TEA data verification over the last three years. This meant that most of the participants contacted in connection with the TEA evaluation data collection had been contacted for verification several times already. The feedback from a large majority (39 out of 42 group interviewees) of beneficiaries indicated assessment fatigue. 3.5.2. Difficulties with focus group discussions (FGD) – As this was an end-of-project evaluation of an activity that would not be renewed for funding in its current form, the beneficiaries, and employers, one step removed from funding support, did not have any incentive or much free time to participate in FGDs or KIIs. Therefore, it was challenging to get beneficiaries and employers to join FDGs and 10 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV KIIs. A beneficiary or employer would often commit to participating in a KII or GI at a specific time and then would not turn up. Challenges to participation in KIIs or GIs included: • Assessment fatigue. • Work hours and unavailability to commit time to an FGD during working hours. • Carpooling transport arrangements that prevented FGD participation after working hours. These challenges were mitigated by conducting small GIs near the place of work ahead of the working day and after hours, providing a transportation incentive, or conducting KIIs by phone/zoom with those unable to commit to GIs but still willing to be interviewed. Through this mitigation measure, the target sample of 42 was achieved, of which 21 were interviewed in 9 GIs, and 21 in KIIs. 3.5.3. Incomplete final data – The timing of the evaluation coincided with the last quarter of TEA implementation, which limited the availability of Life of Project (LOP) data. Fiscal year (FY) 2021 data as reported in DevResults was incomplete as of the writing of the evaluation report and the IPs had not submitted their final reports to USAID and they were therefore also unavailable to the evaluation team. Analysis of data therefore relied on data drawn from IP tracking sheets shared with the evaluation team in May 2021, which was not LOP data. LOP data, provided to the evaluation team during the final review phase of this report, is shared in Table 5. 4. FINDINGS AND CONCLUSIONS The TEA program targeted 6,074 youth (Jordanians and Syrians) with limited financial resources, aiming to place at least 80% of successful trainees into jobs, where 60% of job-placed youth would sustain their jobs for at least six months. TEA targeted youth between 18-35 years age, with the following quota targets: • At least 50% of beneficiaries are youths who did not pass Tawjihi; • At least 30% of beneficiaries are females; • At least 70% of beneficiaries are from poverty pockets and marginalized groups residing in Jordan; • 80% Jordanians and 20% Syrians. Table 4. Implementing partners, targets and LOP achievements Indicator EFE-Jordan LTUC TVTA Total LOP Target Target LOP Achieved* Target LOP Achieved* Target LOP Achieved* Number Trained (Mission Indicator 1.2.2a) 2,500 2,509 (100.4% of target) 1,074 991 (92.3% of target) 2,500 2,424 (97% of target) 6,074 (97.5% of target achieved) Number Employed at 3 months (Mission Indicator 1.2.a) 2,000 (80% of trained) 919 (45.6% of target) 36.6% of trained) 859 (80% of trained) 250 (29.1% of target) (25.2% of trained) 2,000 (80% of trained) 1,284 (64.2% of target) (53% of trained) 4,859 (50.5% of target achieved) Number Employed at 6 months TEA 1,500 (60% of trained) 359 (23.9% of target) (14.3% of trained) 644 (60% of trained) 193 (30.0% of target) (19.5% of trained) 1,500 (60% of trained) 849 (56.6% of target) (35% of trained) 3,644 (38.4% of target achieved) *Cumulative figures as reported in Final Reports in October 2021. Discrepancies between Final Reports and final DevResults data include: TVTA Number Trained reported in Dev Results is 2,292; EFE-Jordan Number Trained reported in DevResults is 2,694; EFE￾Jordan Number employed at 6 months is 539 in Dev Results. Tracking for youth – All IPs’ AMELPs disaggregated by age, geography (governorate level), sex and nationality. IP tracking sheets shared with the evaluation team confirmed that 98.1% of the persons tracked were youth. Those that fell outside of the youth bracket (1.9%) were largely recruited by 11 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV TVTA (84%), were female (62%), had not passed Tawjihi (28%) and came from governorates with a high incidence of poverty pockets (30%). As these outliers were relatively small vis-a-vis the total population (90 out of 4,639 records), represented groups that are marginalized and perhaps harder to recruit, and fell within the statistical margin of error, the evaluation team presented data from the full beneficiary population throughout the report. The sampling approach for the beneficiary survey represents the full TEA population. Tracking for marginalized groups and poverty pockets – USAID did not define marginalized groups under the IPs’ contractual Scopes of Work, nor did the IPs in their AMELPs. Poverty pockets are defined nationally and tracked by the Department of Statistics (DOS) census data at the district level, but this designation does not correspond to IP geographic tracking, which was designated by USAID and recorded by the IP at the governorate level. All IP AMELPs had a Gender and Vulnerable Groups section, where TVTA focused on women, EFE-Jordan focused on women and refugees-Syrians, and LTUC focused on those who did not pass Tawjihi, women, and those with a household income under 230 JD. While LTUC may have used these standards as criteria for recruitment, there is no evidence in tracking sheets of beneficiaries that met these requirements. No IP tracked poverty or poverty pockets in tracking sheets or DevResults data shared with the evaluation team. As a result, the evaluation team created a proxy for poverty pockets, whereby governorates with the highest incidence of poverty pockets were categorized as poverty pocket governorates. These governorates were Ma’an, Mafraq, and Tafilah. DevResults and Final Reporting – With respect to the figures presented in the Table 5, these figures represent the data reported through Final Reports in October 2021, after the evaluation had been completed. As agreed with USAID in the inception phase, the analysis presented in this evaluation focused on IP tracking sheets shared in May 2021, with 4,639 beneficiary records, as well as the TEA beneficiary and employer surveys sampled from the 4,639 beneficiary records. 4.1 EVALUATION QUESTION 1 To what extent did TEA achieve its intended outcomes (i.e., beneficiaries trained by gender, nationality, education level, and those from poverty pockets or marginalized groups)? • How was TEA inclusive in reaching target groups of beneficiaries (e.g., Jordanian and Syrian, men and women, those who did not pass Tawjihi, those with limited financial resources or from poverty pockets, and marginalized groups) residing in Jordan? • What are the key barriers to the participation of women and other marginalized groups? The targets for the three IPs were clearly defined in the SOW; as stated under section 2.3, targets were: 1. At least 30% of the participants should be women; 2. 20% of the participants should be Syrians; 3. 50% of the participants should not have passed Tawjihi; and 4. 70% of the participants should be from marginalized groups or poverty pockets To evaluate delivery on the above targets, the evaluation team analyzed the entire participant population using the Activity’s tracking sheets. The targets were quite clear, although additional clarification was needed to measure target 4. For marginalized groups, the evaluation defined these as women and refugees. To identify participants from the poverty pockets, the DOS definition was used: "the sub-districts where the percentage of individuals below the absolute poverty line exceeds 25% of the total population of the sub-district". Unfortunately, the tracking data did not allow disaggregation 12 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV to the sub-district level, and it was not possible to determine if the participant came from a poverty pocket. In place of this, the three governorates with the highest incidence of poverty pockets and the highest overall poverty incidence rates, namely Mafraq, Ma'an, and Tafilah, were used by the evaluation to indicate the poverty target. Table 5. The implementing partners and the intended outcomes Gender Refugee Educational level Marginalized and poverty pockets Target 30% women 20% Syrian 50% not passed Tawjihi 70% marginalized or from poverty pocket LTUC 40.7% 16.2% 62.2% 42.2% EFE-Jordan 86.9% 21.3% 46.1% 87.5% TVTA 43.0% 4.2% 23.6% 47.6% Total 67.0% 16.5% 47.7% 67.5% Note: Cells highlighted in light grey depict areas in which targets were met. 4.1.1. Gender – All three IPs managed to enroll more female participants than the target of 30%. However, the percentage of female participants varied considerably among the different IPs; EFE￾Jordan managed to enroll 86.9% women, whereas TVTA had 43.0% women and LTUC had 40.7%. EFE￾Jordan and TVTA conducted information sessions to lessen the family resistance to female enrollment. All IPs used CBOs in their enrollment of women. The most commonly offered training courses – the garment industry sector training and beauty and hair training – are also sectors that traditionally attract more women than men. 4.1.2. Inclusion of refugees – With an enrollment of 21.3% Syrians, only EFE-Jordan managed to fulfill the target of at least 20% Syrian beneficiaries. LTUC fell slightly short with 16.2% Syrian beneficiaries, whereas TVTA only had 4.2%. In their outreach to Syrians, all three IPs used CBOs. However, EFE￾Jordan went through UNHCR to identify CBOs for outreach, and LTUC went through UNHCR and UNWRA, which might account for the higher refugee enrollment among LTUC and EFE-Jordan. 4.1.3. Educational level – With an enrollment of 62.2% of beneficiaries that had not passed Tawjihi, LTUC was the only IP that fulfilled the target of at least 50%, although EFE-Jordan was not far off with 46.1%. TVTA, on the other hand, had 23.6% beneficiaries that had not passed Tawjihi. As shown from Table 2, when asked to name the selection criteria, both LTUC and EFE-Jordan had the Tawjihi criteria on the top of the list. TVTA, on the other hand, had as their third criteria "College and university graduates seeking jobs." Given the latter criteria contradicts the criteria of not having passed Tawjihi, it is not surprising that TVTA scored so low on this target. 4.1.4. Marginalized groups and poverty pockets – At 87.5%, EFE-Jordan was the only IP that fulfilled the target of at least 70% from marginalized groups (women and refugees) and poverty pockets (from governorates with high prevalence of poverty pockets). Both TVTA, with 47.6%, and LTUC, with 42.2%, fell quite far below the target. Of all the beneficiaries in TEA, 15.5% came from the three governorates that were defined as containing the highest incidence of poverty pockets.8 Considering the spillover effect from the first two targets, the marginalized groups and poverty pocket target could 8 Poverty pockets are defined as districts/sub-districts with 25% population or more below the national poverty line. Unfortunately, the IP tracking data did not allow disaggregation to the district/sub-district level, and it was not possible to determine if the participant came from a poverty pocket. In place of this, the three governorates with the highest incidence of poverty pockets and the highest overall poverty incidence rates, namely Mafraq, Ma'an, and Tafilah, were used to identify persons from poverty pockets as a proxy measure. 13 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV only be reached by heavily over-delivering on target one or two, or by heavily targeting the three governorates considered high poverty areas. For example, if an IP had 30% women (fulfilled target one) and 20% refugees, of which half were women (fulfilled target two), the IP would still have to make up 30 percentage points in the three governorates with high poverty incidence. Should all three IPs have chosen this approach, the three governorates would have been heavily over-targeted. 4.1.5. Targeting strategies of the IPs – According to KII and GI findings, LTUC utilized online outreach, primarily through Facebook and Instagram. However, no explicit method was employed to target specific groups online. More traditional outreach was conducted through local CBOs, UNHCR, and the Makani Centers. LTUC used these organizations to help reach marginalized youth living in poverty, as well as women and Syrians. According to LTUC, it was a challenge to collaborate with CBOs as they failed to communicate the agreed-upon message appropriately. LTUC had meetings with parents/partners who voiced concern about their daughters/partners joining the training program in order to overcome possible cultural barriers to female participation. “Cultural barriers were noted for females’ participation in training and employment and EFE-J therefore engaged the family members during the interviews. In the garment course they held a field visit for the trainees and family members to ensure family buy-in.”- EFE-Jordan EFE-Jordan partnered with CBOs and UNHCR to reach youth with limited financial resources through outreach activities. EFE-Jordan utilized UNHCR to get updated contact details on registered Syrians and CBOs due to their knowledge of the local community context. The CBOs targeted youth with limited financial resources who were looking for employment opportunities. EFE-Jordan used Facebook to raise awareness and advertise vocational training opportunities. To overcome cultural barriers for female participation, EFE-Jordan engaged family members when female beneficiaries were invited to interviews upon selection. TVTA partnered with CBOs to reach women and Syrians. Specifically, TVTA partnered with an association in Ma’an to reach marginalized populations and target the automotive, crafts, and hospitality sectors. 4.1.6. General barriers for training – When asked if they faced barriers to participation in the training, 6.5% of the beneficiaries surveyed said they had faced obstacles in attending the training. With 8.1%, EFE-Jordan had the highest proportion of participants saying they faced barriers, followed by LTUC with 8% and TVTA with only 3.7%. Of the 6.5% of respondents facing barriers, 50% reported that transportation was the biggest obstacle to their training participation. This survey finding was verified through both the KIIs with all three IPs and the GIs with beneficiaries. The second biggest reason given was the category "other" with 30% of the respondents choosing that provided an open-ended response. Finally, almost 10% said that an obstacle was training being conducted in English for the hospitality sector-training. Due to the sample size of the participants facing training barriers, it was not statistically significant to study the trends by IP. 4.1.7. General barriers for employment – Of the surveyed participants, 25.6% said they had faced barriers when seeking employment. EFE-Jordan had the highest proportion with 31.7% facing barriers, followed by LTUC with 26.1% and TVTA with 19.7%. Of the barriers identified, the most common was the lack of job opportunities, with 35.7% of the participants facing barriers to employment opportunities, followed by personal circumstances (other) with 31% and transportation with 17.1%. 14 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV 4.1.8. Employment barriers by sector – Looking at barriers faced by different sectors, the evaluation team found that sectors such as the garment sector and food service9 had the highest proportion of employees that stated they faced barriers. For example, Figure 3 shows that of the 97 participants surveyed trained in the garment sector, 32 faced barriers. Of these 32 participants, 12 mentioned a lack of job opportunities as a barrier, five identified motherhood, three got married, four lived too far away from their place of work, four had transportation issues, three named a bad working environment, and one mentioned a variety of barriers under “Other”. Figure 2. Employment barriers by sector (by # of beneficiaries) – Did you face any barriers in seeking employment? As shown in Figure 3, the sectors most prone to employment barriers were the garment industry, hair and beauty, food services, money exchange, and the automotive industry. KII informants said that the garment sector was known to be a demanding working environment. “After my experience in the garment sector, I would never ever work in a factory again. The workload was unbelievable, and I have kids at home waiting for me." – Female, South. Reports on a bad working environment became so prominent that TVTA developed training to make the supervisors in this sector more sensitive to the quality of the working environment. After conducting this training in one of the major garment factories, TVTA mentioned that there were fewer complaints about poor-quality working environment. Figure 3. Barriers faced by participants when seeking employment (multiple responses possible10) 9 While other sectors may have demonstrated equivalent or in some cases higher proportions of employees facing barriers, the sample size was insufficient to draw sectoral conclusions. 10 Male n=31 with 36 responses; Female n=71 with 93 responses; Total n=102 with 129 responses 35.7% 50.0% 30.1% 31.0% 22.2% 34.4% 17.1% 16.7% 17.2% 9.3% 13.0% 7.0% 11.1% 5.4% Total Male Female No job opportunities Other Transportation issues Motherhood or pregnancy Bad work environment 15 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV 4.1.9. Barriers for women – Although there were no gender differentials in barriers for participation in training, women were twice as likely as men to identify barriers to employment; 34.3% of women said they faced barriers, compared to 16.3% of men. Of the responses from women facing barriers, 30.1% said that the barrier faced was lack of job opportunities, 17.2% said that it was transportation, 13% had issues with motherhood and pregnancy, and 5.4% had issues with a bad working environment. In addition, 34.4% stated that they were facing other barriers, covering a wide range of personal circumstances. In the KIIs, all three IPs stated that female employment was challenged by the need to ensure spousal or parental approval for beneficiaries to join the labor market. To alleviate some of the resistance from parents, spouses, or other family members, EFE-Jordan and TVTA conducted information sessions for the families where many of these concerns could be addressed. 4.1.10. Barriers for refugees – Due to the small sample size, it was impossible to draw conclusions on the barriers faced by Syrian refugees, as seen from the perspective of the refugees. However, during the KIIs, the IPs identified the obstacles they faced in attracting and retaining Syrian refugees in training and later employment. All three IPs stated that Syrians were challenging to attract due to the crowded donor-financed vocational training market. Often there would be multiple offers of training available, and the Syrian refugees would choose the most attractive offer, frequently one offering a stipend during the training. As TEA only offered transportation costs, it made it more difficult to attract Syrian refugees. In addition, Syrians had difficulties finding employment due to the limited sector choices in the Jordanian work permit system for Syrian refugees. There was also a general agreement that Syrians were more reluctant to commit to a more extended training course and longer employment commitment. 4.1.11. Conclusion – The outreach strategies utilized to reach the intended target population subgroups varied among the different IPs. All three IPs worked with CBOs with local knowledge to reach youth living in poverty and other marginalized groups, and two of the three IPs (EFE-Jordan, LTUC) used social media to reach youth around Jordan, although this approach was, to a large extent, not targeted. The same two IPs also contacted UNHCR and UNRWA to identify refugees. While all IPs were based in Amman, LTUC had a campus in Irbid, and TVTA established a separate office in the South to ensure better outreach. While TEA IPs made significant efforts to reach demographic quotas in beneficiary recruitment, they met with varying success. While the combined efforts of the three IPs met or nearly met the four intended outcomes, EFE-Jordan was the only IP to meet or come close to meeting its gender, refugee, marginalized groups and Tawjihi targets, while LTUC met or came close to meeting all but its marginalized groups target, and TVTA only met its sex target. In the aggregate, however, the combined TEA IPs met the sex target and came within two and half percentage points of meeting the refugee, education, and the marginalized groups and poverty targets. In exploring barriers to participation in training as well as employment, the majority of respondents (93.5% of beneficiaries surveyed) said they did not face significant barriers to participation. Whereas a small proportion of the participants faced barriers in attending the training (6.5% of surveyed beneficiaries), a much more significant proportion faced obstacles when seeking employment (25.6%). The most commonly cited obstacle to attending the training was transportation, whereas the lack of employment opportunities was identified as the most significant barrier to employment. Other barriers to employment participation were transportation and motherhood. 16 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV The sectors with the highest proportion of employees facing barriers to participation were the garment and food services sector; 35.3% of women surveyed said they faced barriers in seeking employment which was more than twice that of men surveyed. A majority of the women facing barriers were trained in the garment sector, where the employment level of the beneficiaries surveyed was less than 40%. Of participants surveyed that faced obstacles in the garment sector, most faced a lack of job opportunities, followed by motherhood and marriage. Syrian refugee beneficiaries were challenging to attract due to the competition from other vocational training programs, and they faced more challenges entering the labor market due to the Jordanian work permit system. In addition, the Syrian refugees themselves faced barriers due to general lack of confidence in refugees’ willingness to remain in long-term employment, and limits to employment opportunities due to restrictions in the Jordanian work permit system. While barriers to participation related mainly to women as a result of personal, cultural and transportation issues, these barriers did not prevent the IPs from meeting their gender targets. Similarly, shortfalls in reaching Syrian refugee targets were related to limited sectors of work permit permission for Syrians, as well as competition from other donor-supported social networks at odds with Syrians seeking formal or long-term work. With TVTA, the shortfall in beneficiaries not passing Tawjihi, was due in part to their contradicting criteria for recruitment of college graduates versus students not passing Tawjihi. Success in reaching inclusion targets was due to strong local outreach, interface with families to assuage cultural challenges to women working, geographic proximity of training locations, and alignment of jobs available with geographies of marginalized people. Sectoral alignment for women and refugees also seemed to influence levels of inclusion, as both Syrian and women faced work permit and/or cultural limitations on sectors of work. 4.2 EVALUATION QUESTION 2 To what extent are TEA interventions broadly relevant to the needs of target beneficiaries and local employers? More specifically, were the training courses relevant to local market needs, considering the perceptions of both youth and employers? The IPs chose different approaches to adapt their training to the needs of local employers. KIIs with the IPs and document review provided the following information. “Luminus visited employers to see what they wanted. We added a component to the hospitality training, for example, following an employer visit.” – LTUC 4.2.1. Luminus Technical University College – LTUC relied on previous experience to map out market labor needs. According to a case study report, LTUC had previously conducted similar projects, including vocational training programs with USAID, UNDP, and UNICEF. As such, LTUC said that they did not conduct a new labor market assessment for this Activity, given their existing knowledge of the labor market. In addition, LTUC created the Employment Hub in 2017 to engage with local employers through regular visits, making it possible to adapt their courses to fit potential market needs. “We went to beauty salons and asked them how many employees they wanted and we trained accordingly. - TVTA “The employer interviewed the candidate before registering them. We made sure they got recruited. That was the main goal.” – TVTA Based on employer visits, LTUC built partnerships with the private sector and enhanced its core curriculum across programs. Tapping into their partnerships with the private sector, LTUC added 17 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV core courses to its programs to give students a good grounding in transferable skills. Courses emphasized: English language training to equip students to succeed in a global workplace; and soft skills including effective communication, responsibility, goal-setting, and teamwork based on the Passport to Success course developed by the International Youth Foundation. These were previously complimentary courses that became mandatory for all beneficiaries selected under TEA. In addition, LTUC offered two tailor-made training courses for Profix, a maintenance services facility, and Classic Fashion, a large garment sector employer. Otherwise, it provided only the courses already being taught at LTUC. 4.2.2. Education for Employment - Jordan – EFE-Jordan conducted mini-assessments based on desk review to identify sectors where there was a need for qualified employees. Specifically for TEA, EFE-Jordan delivered customized courses in areas such as Android programing and digital marketing, based on labor market needs. In addition, the training centers that EFE-Jordan partnered with were selected based on their locations (not concentrated in the central region) to ensure ease of access for participants and respond to different geographic needs. EFE-Jordan achieved employer engagement through its Corporate Department, which consulted employers to identify areas for training and determine the required skill levels. According to KIIs, EFE￾Jordan developed training materials based on employer-identified needs and allowed employers to review and provide feedback that integrated into the material before sharing with training centers. For example, responding to one need identified, EFE-Jordan said they offered mandatory soft skill training once beneficiaries were selected, an area employers flagged as lacking among youth hires in Jordan. Figure 5. Percent of employers respondents confirming they were consulted on their labor needs 4.2.3. Technical Vocational Training Academy – TVTA selected sectors based on targeted geographic demands with gender-based employment taken into account. For example, the beauty and hospitality sectors courses were conducted based on beauty labor demand in the North and hospitality labor needs in the South. TVTA relied on market studies focusing on general labor needs but mostly on its own information about youth unemployment. TVTA asked employers to interview students before their training to ensure employer engagement. All courses had on-the-job training, with a 3-month probation period, and during this period, TVTA provided transportation incentives. 88.9% 72.2% 57.9% 69.7% 11.1% 27.6% 42.1% 30.3% EFE-J LTUC TVTA Total Yes No 18 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV 4.2.4. Relevance to beneficiaries and employers – Noting that IPs reached out to employers and potential beneficiaries through different strategies, the following question became how effective the approaches of the IPs were in preparing the beneficiaries for the labor market and making sure that the employers got the skilled labor that they needed. Figure 5 depicts that 69.7% of the employers were consulted on their labor needs; the lowest consultation rate was TVTA, where 42.1% of the employers had not been consulted. This was followed by LTUC, where 27.6% had not been consulted, and finally, EFE￾Jordan, where 11.1% had not been consulted on their labor needs. When asked to what extent the training had prepared them for the labor market, 55% of the beneficiaries said to a great extent, and 33.2% said to a medium extent. Here again, the approach of EFE-Jordan seems to have been most successful as 65% of the beneficiaries said that they were prepared for the labor market to a great extent. This was followed by TVTA, where 57.7% of the beneficiaries were prepared to a great extent, and finally, LTUC, where it was 43.5% 4.2.5. Full population relevance – When looking at the relevance of the training to the beneficiaries, the evaluation team returned to the TOC of the Activity. The TOC stated that the Activity aimed to remove supply-side labor market issues by equipping youth with skills and sparking employers' willingness to employ these trained youth. An indicator of this success would be to monitor the total number of beneficiaries placed in employment. However, the placement rate was difficult to track as the employment status for the beneficiaries was often left blank in the tracking sheets without an indication of why. However, if the evaluation team were to assume the lack of information in the employment column of the Activity tracking sheets indicated the beneficiary's lack of employment placement, the total number of beneficiaries employed would be 1,743 out of 4,639 beneficiaries, an employment placement rate of 37.6%. This was also on par with the reported LOP data from TVTA, with 45% of those trained placed in jobs for 6 months. At the time of evaluation report writing, EFE-Jordan and LTUC had not reported LOP achievements. As an indicator of training relevance, an alternative to the overall employment placement rate, would be the beneficiary dropout rates. The dropout rate could be interpreted as a lack of relevancy perceived by the beneficiaries. However, based on the available data, the evaluation team noted dropouts were not registered on the tracking sheets. Lack of certification could have been a possible way to identify dropouts. Still, this category was largely left unpopulated (more likely because the IPs determined that an area of training did not warrant certification) and therefore cannot be used to identify possible dropouts. Figure 6. Were the soft skills appropriate to your work 19 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV 4.2.6. Employee relevance – When exploring the beneficiaries' view on how appropriate the training was, one area the evaluation examined was the soft skills relevance to the labor market. All beneficiaries received training in soft skills before being employed. Each IP developed its own soft skills courses, and the soft skills taught, therefore, varied. For some of the IPs, like EFE-Jordan, the soft skill was an important part of their philosophy. Their "Education for Employment" is implemented across the Middle East, and they have a soft skills program called "finding a job is a job" that provides the backbone for their soft skills training. For all IPs, the soft skills courses normally covered skills needed to get a job (e.g., job application process, CV formulation, communication) and keep a job (e.g., punctuality, discipline, and organization, negotiation) as well as general skills such as English language training. “The short training didn’t cover all the skills we needed.” – Male, Jordanian, Hospitality, South When asked if the soft skills were appropriate to their work, 67% of beneficiaries surveyed said that they were appropriate to a great extent, and 26.3% said to a medium extent. Only 6.7% of the beneficiaries said that the soft skills were relevant to a poor extent or not at all. The majority of beneficiaries across all IPs found the soft skills to be relevant. However, during the GIs with the beneficiaries, they said that they would like to see the prioritization change in the training. Common feedback in the GIs was that the training should have been longer and focused more on technical skills. This feedback was confirmed by the KIIs conducted with employers who said that more focus should be placed on on-the-job training. Figure 7. Employer satisfaction with the skill level of trainees 5.3% 4.4% 4.8% 5.8% 29.7% 41.3% 20.3% 26.3% 64.0% 54.0% 70.7% 67.0% Total LTUC EFE - J TVTA Not at all To a poor extent To a medium extent To a great extent 20 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV 4.2.7. Employer relevance – Figure 7 shows that 51.7% of employers said they were satisfied with the skill level of the beneficiaries to a great extent, and 40.4% were satisfied to a medium extent. EFE￾Jordan had the highest overall satisfaction with 77.8% of employers saying they were satisfied with the skill level of the employees to a great extent, followed by LTUC, with 50.9% of the employers expressing satisfaction, and finally TVTA, where 26.3% of employees said they were satisfied to a great extent. However, a relatively high proportion of the employers replied that they were satisfied to a poor extent with a low satisfaction rate of 15.8% for TVTA and of 8.1% for LTUC.KII data show that the employers were satisfied with the beneficiaries to a great extent. However, it is also clear that their expectations might not have been very high. For many of the beneficiaries' positions, on-the-job training was more important than the technical training received through the training institutes. However, there was an appreciation from the employers especially of the soft skills training. For two of the IPs (EFE-Jordan and TVTA), big employers chose to send newly employed staff to the TEA training for them to receive the soft skills training. Figure 8. Did the training prepare the participants for the labor market 15.8% 6.3% 6.7% 57.9% 41.1% 22.2% 40.4% 26.3% 50.9% 77.8% 51.7% TVTA LTUC EFE - J Total Not at all To a poor extent To a medium extent To a great extent 21 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV 4.2.8. Conclusion – While the IPs took different approaches to adapt their training to the needs of the local employers, most conducted a market assessment, used knowledge of the market, and/or consulted with employers to develop relevant training. Some IPs such as TVTA and LTUC developed tailor-made courses for two employers, founded on existing courses, while EFE-Jordan designed tailor￾made courses in Android and digital marketing based on market needs assessment for the digital economy. TVTA selected the training institutions that conducted the training based on their geographical distribution to ensure access to training in all locations and identified geographical demand for specific employees and conducted training to fulfill these demands. To ensure commitment from the employers, TVTA and EFE-Jordan asked prospective employers to interview the beneficiaries before they started the training. Overall, 69.7% of the employers were consulted on their labor needs, and 55% of the employees said that the training prepared them for the labor market to a great extent. An analysis of the IP tracking sheets of the full beneficiary population indicated that 37.6% of all beneficiaries found employment. TEA IP Final Reporting demonstrated that 41.4% of those trained were employed at 3 months, and 23.5% of those trained were employed at 6 months. No data could be found for course dropouts. Furthermore, 67% of the participants stated that the soft skills training had been helpful in their job to a great extent. Of the employers, it was found that 51.7% found the skill level of the beneficiaries satisfactory to a great extent and 40.4% to a medium extent. As a result, it can be concluded that TEA interventions were relevant to the needs of beneficiaries to a good extent, and to local employers to a large extent. 4.3 EVALUATION QUESTION 3 What are the rates of employment and retention for TEA beneficiaries by gender, nationality, financial status, and sector of training – both within fixed periods (e.g., at least three months, at least six months) and as trends over time? Are there differences in employment and retention by category? What factors are likely driving any observed differences? Figure 9. Employment of surveyed participants 13.0% 8.8% 8.1% 10.0% 42.03% 32.12% 24.39% 33.20% 43.48% 57.66% 65.04% 55.03% LTUC TVTA EFE - J Total Not at all To a poor extent To a medium extent To a great extent 22 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV 4.3.1. Employee retention and employment – Central to the theory of change for TEA was linking the beneficiaries with employment. Figure 9 shows overall employment situation of the participants surveyed. This data presents several interesting findings regarding the participants' involvement with the labor market. First it shows that male participants have remained employed to a high degree whereas women have lower rate of employment retention, with 83% of males being employed compared to 53.2% of women. However, it also shows that women tend to stay with the same employer, while men move around to find better employment. Almost half of the employed men surveyed (37.7%) moved to a different employer whereas less than a quarter of the women (14.5%) who moved to work for a different employer. Finally, it also shows that women might leave the labor market altogether whereas men hardly ever do so. Of the surveyed men, 16.7% had registered as unemployed, of which 0.5% had stopped looking for employment. As for the women, 46.8% said they were unemployed, 34.8% were still looking, and 12% registered as unemployed but not looking for work. This loss of labor market engagement can result from the barriers women face, such as competition from household work, marriage, pregnancy, and resistance from family members, parents, or spouses. This was confirmed in the KIIs conducted with EFE-Jordan, where they said that they had seen females dropping out from employment, a finding based on the six-month follow-up conducted with beneficiaries. All three IPs said that the reasons females were not remaining employed were likely due to marriage and pregnancy. Further, it was indicated that women preferred home-based employment. For example, EFE-Jordan stated, “women who graduate from garment courses generally work to save up for a sewing machine to start their own home-based business.” Figure 10. Why participants remained in the same job 16.2% 34.8% 25.9% 0.5% 12.0% 6.5% 45.6% 38.7% 42.0% 37.7% 14.5% 25.6% Male Female Male Female Employed Unemployed Total Unemployed, looking Unemployed, not looking Employed same place Employed different place 23 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV 4.3.2. Employment and sector mobility – Figure 9 shows that 45.6% of the men and 38.7% of the women remained employed in the same job they got after their training under TEA. As can be seen in Figure 10, when asked why they remained in the same position, 45.5% said it was because they liked the job, 27.5% said that it was because the job was easy to get to, 24.2% said it was the salary, and only 2.8% noted that it was because it was the only job they could find. From these findings, it is clear that TEA has been able to create employment. Figure 9 shows that 14.5% of women and 37.7% of the men said they were employed, but no longer employed with the same employer they were placed with following training, so 25.6% of the respondents had moved to work with a different employer. A large proportion (46.8%) of the women were not employed. However, of the women employed, 72.7% stayed with the same employer and 27.3% moved to a different employer. For the men, 54.7% stayed with the same employer and 45.3% moved to a different employer. Figure 11. Reasons given for not staying in the sector 45.5% 27.5% 24.2% 2.8% I like the job It is convenient to get to The salary Only job I could find 24 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV Figure 11 shows that the most common reason for having changed the sector of employment is because jobs were not available within the sector, as cited by 42.9% of the respondents that changed sectors. This may indicate that the labor market assessment made in connection with providing the training did not adequately assess demand or that the sector selected has been oversaturated in connection with the introduction of the new beneficiaries. Specific to gender, it can be seen from Figure 9 and 11 that women were disproportionally trained in sectors where jobs were not available; indicated by the high female unemployment (Figure 9) as well as the high proportion of female respondents using the lack of employment as an explanation for not staying in sector, as indicated in Figure 11. The next most frequently cited reason to move employment sectors was that the salary was not good enough, and this was only observed among male respondents. This would reconfirm the earlier finding that men shop around to improve their employment situation, and 28.9% of males said the reason they moved jobs was that their salary was not good enough. This point was confirmed in KIIs with LTUC, where they said that it is unlikely for beneficiaries to stay in a job if offered higher salaries. “Retaining employees is difficult because some beneficiaries would be offered a better salary elsewhere.” This finding regarding retention was also corroborated with graduates interviewed where salary was said to play a role in youth retention, for example, “I retained employment because of the good salary that was offered to me.” – Female, South Further, some employers highlighted a mismatch between the expectation of beneficiaries and the actual job being offered as an impediment to retention. One employer said the following, “the graduates had very different expectations when they joined us. They expected the job to be easier, shorter shifts, and a higher salary.” – Country Club. IPs also highlighted this mismatch, with LTUC stating, “youth are said to expect to be placed in managerial roles and at times have a different perception of what the job entails.” To mitigate this, LTUC said that they held two rounds of interviews with beneficiaries to manage expectations. 31.6% 81.8% 42.9% 28.9% 22.4% 23.7% 9.1% 20.4% 7.9% 6.1% 9.1% 4.1% 5.3% 4.1% Male Female Total Work was too physically difficult I do not have the required skills Employer's location is far I found better opportunities Low salaries Didn't find a job 25 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV Figure 12. Employment in poverty governorates and of Syrian refugees 4.3.3. Poverty pockets/governorates – As noted earlier in this report, data were not available to analyze specific poverty pockets as in Jordan these are defined at the sub-district level. Given that the IP tracking sheets and DevResults did not report data below the governorate level, the evaluation team chose to designate three governorates (namely Mafraq, Ma'an, and Tafilah) as the poverty pockets. These three governorates had the highest incidence of poverty and highest number of poverty pockets according to the DOS 2020 census. Figure 12 shows that unemployment among participants in the high poverty governorates (55.3%) was higher than the unemployment of the surveyed participants as a whole (25.9%) (shown in Figure 9). It is also clear that there was a need for employment in these governorates as 55.3% of respondents were unemployed and looking for a job. Only 2.1% were unemployed but not looking for a job, whereas 34.0% were employed with the same employer they started with after the training. Only 8.5% had moved to a different employer. This could indicate that the labor market in the high poverty-incidence governorates was less dynamic than the rest of the country and that participants were less likely to change jobs. 4.3.4. Syrian Refugees – Due to the small sample size for Syrian refugees (n=10), any findings in the isolated sample for this population group were not statistically significant. However, it still provided some indications of differences in labor market behavior for this population group. Figure 12 shows that Syrian refugees had a very different labor market interaction than the Jordanian population. Of the Syrians surveyed, 50% said they were unemployed but not looking for employment, and only 10% were looking for employment. The 40% that were still employed all worked for a different employer than they started with after the training. KIIs with the IPs and employers indicated that Syrians are often unwilling to stay in long-term employment and often prefer seasonal employment due to their situation. Additionally, because of the Jordanian work permit system, Syrian refugees were limited in the sectors in which they can be employed. From the limited data available from the data collection, it was clear this restriction has a considerable influence on their interaction with the labor market. This was also confirmed through KIIs with employers where some said that they did not hire Syrians because of labor law restrictions. Others said they prefer to hire Jordanians as they have better retention rates, especially because they struggle to find employment elsewhere. 34.0% 8.5% 40.0% 55.3% 10.0% 2.1% 50.0% Poverty Governorates Syrian Refugees Unemployed not looking Unemployed looking Employed different employer Employed same employer 26 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV 4.3.5. Sector employment – The IPs provided training over a wide array of sectors. However, as depicted in Figure 13, some sectors had more beneficiaries than others. While training in the garment industry, money exchange, hair and beauty, and sales and customer service had many beneficiaries, courses such as silver crafting and handcrafts and fashion design had fewer beneficiaries. Sectors that traditionally attracted women have the most significant number of beneficiaries. However, these sectors, such as the garment industry and hair and beauty, also had a significant number of beneficiaries not employed. In the sample of participants surveyed, 97 were trained in the garment industry and 60.8% were not employed at the time of survey. In hair and beauty, 45 in the sample were trained of which 35.6% were unemployed at the time of the survey. More traditional male-dominated sectors such as automobile and motorbike repair, on the other hand, had a low rate of unemployment, with only 9.4% of 32 not employed at the time of survey. Figure 4. Employment by sector (# of participants) In looking at those who remain employed, three types of situations were examined: 1) the beneficiaries that have remained with the same employer after graduation; 2) beneficiaries that have moved to a different employer but in the same sector; and 3) beneficiaries that are employed by a different employer in a different sector (making the training less useful/relevant to their current employment). Figure 14 shows that the majority of beneficiaries that were employed in the garment industry (87%) remained employed by the employer where they got their job after graduating; 11% had moved to a different employer but remain in the garment industry, and only 3% have moved to another sector. 4.3.6. Employment retention - In hair and beauty, another sizeable female-oriented sector, the vast majority remained in the sector (90%), but most (59%) changed employers, and 10% were employed by a different employer in a different sector. This lower employer retention was most likely due to the nature of the beauty sector where most workers have a loose affiliation with an employer and often only work on a freelance basis. In KIIs with LTUC, it was noted that the beauty sector has relatively low employee retention as salons do not register employees for social security, offer low salaries and offer a relatively poor work environment. As both the garment and the beauty sector had a high number of unemployed, the relatively high retention may have also signified those beneficiaries 27 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV who had secured a job held on to it. These characteristics were also observed in money exchange (85%), teacher rehabilitation (46%), and medical and health services (58%) where beneficiaries generally retained employment with the same employer. With food service and hospitality, about half of the beneficiaries remained with the original employer, but the other half had left the sector – likely as a result of the COVID pandemic and its impact on the food and hospitality sector. Some more male-dominated sectors, on the other hand, seemed to have less employer and sector retention. Looking at the automotive and motorcycle repair sector, only 24% remained with the same employer where they were originally placed; 52% had a different employer but remained in the same sector, whereas 24% had moved to another sector. The mobility in this sector was most likely because of opportunity (repair shops were plentiful). By contrast, employee mobility was much lower in money exchange, where 85% remained with their original employer, 10% had switched employers but remained in the same sector (likely due to fewer alternative employers), while 5% had switched sectors. Similarly, 80% of beneficiaries placed in call centers had remained with the same employer, while 20% had switched employers but remained in the same sector. When asked what kind of beneficiary was most likely to remain employed long-term, the employers surveyed employing beneficiaries from TVTA reported that nationality was the most important explanation (45.1% responded that Jordanian men were most likely to remain employed and 21.5% responded Jordanian women). For employers having beneficiaries from LTUC, nationality, sex and prior working experience were almost equally important (35% responded previous working experience and 30% responded Jordanian men), and for employers having beneficiaries from EFE￾Jordan, prior working experience was the most critical factor. Figure 14. Employee retention by sector 4.3.7. Conclusion – An analysis of the participants surveyed found that 67.6% were employed at the time of the survey. It was found that women were more than twice as likely to be unemployed than 28 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV men (16.7% vs. 34.8%) and that a considerable proportion of women (12%) had left the labor market and were no longer seeking employment. Men were more likely to seek new employment to improve their salary or working conditions, whereas women were more likely to stay with the same employer. General job mobility was relatively high, with 25.6% moving from the job they got after the training. When asked why they stayed in the same position, the most common answer was that 1) they liked the job; 2) the employer was conveniently located; and 3) that the salary was good. However, when surveying people who moved jobs, a low wage was the second most common explanation for men, although it did not come up for women. The most common reason not to stay employed was that jobs were not available. This reason may have become more prevalent in light of the COVID pandemic’s influence on the economy. In the high poverty incidence governorates, unemployment was higher (more than 50%), but everyone was still looking for employment. Job mobility was also lower. Among Syrian refugees, none stayed in the same job, and more than half were unemployed but not looking for employment. Figure 15. Employer perception of reasons for retention Most of the beneficiaries were trained in traditionally female-dominated sectors, such as the garment industry or beauty and hair. Incidentally, these were also the sectors that had the highest number of unemployed beneficiaries. Looking at employer/employee retention, most beneficiaries seem to have remained employed with the same employer they came to just after graduation. Employer/employee retention was particularly evident in money exchange, where up to 85% of beneficiaries remained employed with the same employer as they were placed with, which may reflect the lower number of similar employers in the beneficiaries’ vicinity, and the training received, as well as the work environment/salary. In sectors such as beauty and hair, automotive repair, medical and health service, administrative office work, the majority (76% to 90%) of beneficiaries remained employed in the sector, but not necessarily with the same employer, which may reflect the transferability of skills in demand, and the freelance nature of the sector, as well as the relatively larger number of sector￾relevant employers in the beneficiaries’ vicinity. Sectors such as food service and hospitality, had low sector retention perhaps as a result of the COVID pandemic. Mobility within a sector and its freelance/long-term nature, as well as the number of similar employers in the vicinity are likely causes of employer retention or lack thereof. 29 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV 4.4 EVALUATION QUESTION 4 How likely are TEA interventions to be sustained over time? Why is this the case? 4.4.1. The sustainability of an activity is always difficult to assess. When conducting an activity such as TEA, there is always a hope that the long-term impact of the activity goes beyond the immediate impact of providing training to a significant number of youths, and that these benefits are valued by the end users, such as the training institutes or employers. From the KIIs with the IP, there was a general agreement that the interventions under TEA could be implemented only with support from a donor. All three IPs have already implemented similar interventions for other donors, and there was not much pressure to go out and find alternative funding. One of the IPs said, “The only way we can ensure sustainability is if USAID or other donors keep funding us.” and another IP said, “The private sector won’t pay for training programs. They would rather hire employees with experience or fast learners. Now, after COVID, they can barely afford minimum wages.” However, two of the IPs indicated that the TEA activity supported them in designing accredited and demand-driven training curricula that could help them establish increased credibility for additional donor funding. Figure 5. Employer willingness to pay for training 4.4.2. Employer willingness to pay – Figure 16 shows the willingness of the employers surveyed to pay for similar training interventions. While the majority (59.7%) of employers noted they would not be willing to pay for training, 40.3% of the employers surveyed said they were willing to pay for training if the circumstances were right. When asked why they were unwilling to pay, 83.7% said there was no funding for this kind of training, and 10.7% said it was because there was no benefit for the company. Of the employers willing to pay, 44.2% said that it would depend on the cost, 43.3% said they would pay if the training could be tailor-made for what the company needed, and 10% said they were willing to pay if it would be more cost-efficient than training done in-house. There seems to be more private sector willingness to pay for this kind of training than the IPs realized. 30 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV Figure 6. Perspective of the employers on what gave them the confidence to employ a graduate from TEA To advance the case for private sector funded sustainability, the evaluation team also explored why employers would employ a trainee from the TEA implemented training. The results were relatively uniform across the different IPs. The most important factor in employing beneficiaries was that the employers were impressed with the interviews conducted. The interviews were an integral part of preparing the beneficiaries for the labor market. EFE-Jordan, for example, held two rounds of one-on￾one interviews to inform the beneficiaries about the reality of the job. During the second interview, employers were invited to interview the beneficiaries. When selected, the beneficiaries were requested to sign commitment letters requiring a six-month commitment. The second most common factor was the certification given to the beneficiaries. Thirdly came the reputation of the training institute followed by employers’ previous experience with the training institute. Figure 18. Why beneficiaries joined TEA and to what extent TEA helped finding employment As can be seen in Figure 18, the participants joined TEA for the job placement (30% of responses) and for the certification (28% of the responses). Of beneficiary respondents, 79% found that TEA had contributed to finding them employment (to a great or medium extent). So, for the employees, the areas identified as the most useful were also central to the strategy of TEA, such as providing certification and getting the employers involved early in the selection process through interviews. And for the beneficiaries, job placement, along with certification, were the most valuable aspects of TEA. They found that TEA had contributed to them finding employment to a great or medium extent. 4.4.3. Student tracking at LTUC – A component of TEA was to establish a sustainable student tracking Information Systems at LTUC following international standards for educational management. The system established can track students enrolled at the school, any interaction with potential future students and all information on potential employers for the students. Although only operational relatively late during the TEA the information system can provide a tracking system for LTUC for the 31 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV years to come. Under TEA, an IT system was also installed at TVTA to improve their ability to provide student follow-up and tracking. 4.4.4. Conclusion - While IPs highlight that financial sustainability of TEA interventions can only be supported with donor support, 40% of employers interviewed indicated some willingness to pay for training, demonstrating more private sector willingness to pay for this kind of training than the IPs realized. However, even with the private sector interest, the IPs will most likely continue to concentrate on donor-funded programs as vocational training livelihood-oriented funding is currently plentiful in Jordan with the majority being donor and government-funded with stipend support to beneficiaries during training. While employee-financed certification may be an approach to also consider, this seem unlikely due to fact that the target beneficiaries of TEA were supposed to come from marginalized groups and poverty pockets, and therefore do not have funding for self-financing of training. Therefore, sustainability through private-sector-funding or individual-financing of this kind of training is unlikely. 4.5 EVALUATION QUESTION 5 What are the key lessons learned about TEA design and implementation successes and challenges which can be applied to future workforce development interventions? 4.5.1. The intended outcomes of the Activity – The IPs were tasked with providing training to a pre-agreed number of beneficiaries and ensuring that at least 80% were employed for at least three months and 60% were employed for more than six months. The funding disbursement of TEA was directly tied to the IPs fulfilling these milestones. However, the IPs were also tasked with hitting intended targets for participant distribution, although this was not connected to the funding disbursement. The targets expected to be met by the IPs were that at least 30% of the participants should be women, 20% should be Syrian refugees, 50% should not have passed Tawjihi, and 70% should be from marginalized groups and poverty pockets. As the targeting criteria were not clearly defined, the three IPs opted for quite different approaches. All three IPs targeted women and managed to fulfill the target of 30% women. However, when it came to the target of educational level, the IPs took different approaches. Two of the IPs had the lack of Tawjihi as a clear target, and one of the IPs (TVTA) turned this target on its head and had "college and university graduates seeking jobs" as a criterion. They also pursued different approaches in targeting Syrians, where TVTA used the same targeting strategy for refugees as for the rest of the participants, whereas the two other IPs utilized UNHCR in their targeting of refugees. The differences in participant targeting resulted in TVTA underperforming on the education, the refugees, and the marginalized group's targets. However, by under-performing on the participant targets, TVTA got a participant population that was more uniform and easier to train and employ. They, therefore, also performed well when it came to employment and retention (both relevant targets but also targets that had influence on the funding disbursement). 4.5.2. Poverty target and marginalized groups – One participant targeting criteria was that 70% of the participants should come from marginalized groups or poverty pockets. In Jordan, a poverty pocket is defined as a sub-district where more than 25% of the population lives below the poverty line. It is not clear from the data or the KIIs if the IPs paid attention to target the districts with higher poverty incidence. Marginalized groups cover other participants that have already been targeted under output targets one and two (women and refugees). Keeping this in mind, it was a high target to have 70% of the 32 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV participants from this group, considering that the target for women was 30% and 20% for Syrian refugees (which would also include Syrian women already counted). 4.5.3. Participant tracking – A very innovative component of TEA was participant tracking and monitoring. That this has been implemented across three IPs is impressive. However, when consulting the tracking sheets, it was often found that information was not entered in some of the categories. Ideally, as much of the information as possible in the tracking sheets should be multiple-choice without the possibility to leave a "cell" without information. For example, information such as employment/employer or training certification is a clear multiple-choice option. It should, therefore, always have information, even if to say that the beneficiary was unemployed or uncertified. Due to lack of complete data such as dates of employment, the tracker databases shared did not identify duration of employment for many records. As pointed out earlier, the participant tracking also resulted in considerable assessment fatigue. When talking to the TEA participants, it was found that a majority complained about being called repeatedly to be asked about their employment situation. Understandably, some tracking of the participants was necessary, especially in this Activity where the funding was tied to the IPs being responsible for having their beneficiaries employed for more than six months. However, the participants should have been informed before joining the training that this form of monitoring was a part of the approach. In future programming, USAID and the IPs should explore alternative and less intrusive monitoring options, such as self-reporting through online forms or SMS (this could be combined with contacting the participant by phone if the information was not provided within a deadline), and a call count sheet should be kept for each participant to ensure that the same participants are not repeatedly contacted. When monitored for a reasonable period, measures should be taken to ensure that the participant is taken out of the monitoring pool. 4.5.4. Matching skills with demand – The TEA IP survey data on employment, employer/employee retention, and sectoral retention demonstrated that ensuring that skills developed are market-driven was critical to success. This was achieved based on both needs assessment, and employer consultation/collaboration through approaches such as customized training, beneficiary interview/matching, and consultation on needed skill sets. Food service, automotive, money exchange, sales and retail are where beneficiaries had the lowest rates of unemployment after training, as well as high rates of retention with the initial employer. Other areas such as beauty and hair, call centers, medical and health, automotive, and money exchange had high rates of retention in the sector, but not necessarily with the initial employer, which speaks to both the demand for transferable skills, as well as the more short-term nature of the sector (gig or freelance - oriented rather than long-term employment oriented). While it is clear by sectoral retention rates that these skills are in demand, there may be a need to revisit the definition of job retention based on the nature of the jobs in each sector. As the fourth industrial revolution enables more digital, freelance and gig employment, long-term traditional jobs may be less available or less in demand. As for future programming, it is critical to continue to require market assessments for skills in demand, not only to inform skill-building courses, but also to inform the type of employment that graduates can expect to find upon graduation, potential retention rates, and how to navigate an increasingly gig-oriented sector based on building transferable skills. 4.5.5. Demand side jobs identification – Working with larger employers or chains over multiple geographies was a key to successful placement, employment and retention for those IPs that partnered with groups such as Sameh Mall, Kareem Hypermarkets, Profix, Classic Fashion, and Al Alawneh Exchange. Once employer buy-in had been secured through both reputation of training provider and 33 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV caliber of graduate, beneficiary recruitment and job placement became regular and cyclical with these large employers. Similarly, a sectoral approach to generating demand could enable targeting of multiple employers with similar needs, such as within the garment industry. These approaches to training and job placement build on economies of scale. A mapping of employers by both sector and geography may better enable IPs to build such economies of scale. 4.5.6. Sectoral alignment for marginalized groups – Mapping employers by sector and geography could also result in optimal recruitment, placement, and retention for target subpopulations. As demonstrated with the TEA data, each population subgroup had unique needs and thus required a differentiated approach to recruitment. For example, women may need their cultural/family concerns and flextime needs to be addressed in recruitment and job placement, while Syrians may need sectoral alignment and coordination with agencies offering stipends/support to ensure retention in training. At the same time, such a mapping could clearly delineate poverty pockets, through which recruitment following TEA criteria could support improved targeting and recruitment of persons from poverty pockets, where rates of unemployment are high, even among trained graduates. 4.5.7. Conclusion – TEA IPs in aggregate were successful in recruiting, engaging, and employing marginalized groups, especially women. Efforts made to address cultural barriers faced by women among their families, were helpful to the extent that the TEA beneficiary population exceeded its female target with each IP. Recruitment efforts focused on engaging local groups such as CBOs, and local outreach with the establishment of field offices in the South (TVTA) and delivering training in local areas. However, in utilizing different approaches to recruitment of beneficiaries while adhering to key criteria, the TEA IPs recruitment outcomes were not the same, and not all hit the required sub-population targets. The lessons learned from this is that the implementation criterium must be the same for all IPs to ensure a level playing field when promoting employment opportunities. In addition, if USAID intends to encourage a specific participant distribution, this must be monitored in connection with the implementation and, if the special funding arrangements from this Activity are to be repeated, have similar enforcement as the employment targets. The biggest shortfall in population subgroup targeting was among recruits from poverty pockets, which was not geographically tracked in accordance with national definitions of poverty pockets. While the target for marginalized groups were almost met as a result of recruitment of women and refugees, a lesson learned is to avoid repeating the same participant groups in multiple targets and instead focus on a dedicated poverty target. Having a reliable poverty target that was not geographically bound might have added more attention to this target and made it more relevant in the targeting by the IPs. In its current form, there was no indication in the KIIs or the data that the IPs targeted poverty pockets, which is an area that needs to be strengthened in future programs. Collaboration with employers also proved to be a successful approach to employment and retention. This was achieved based on both needs assessment, and employer consultation/collaboration through approaches such as customized training, beneficiary interview/matching, and consultation on needed skill sets. TEA IPs had success in working with large employers or sectors with large employment needs, such as in the garment industry. TEA data on high rates of sectoral retention of beneficiaries demonstrate the importance of such approaches, but simultaneously demonstrate that the job market is not uniform and that some sectors are less uniform, are freelance in nature, and likely require a different approach. There is a need to constantly examine market and sectoral trends, to establish continued demand, to assess skill sets that are needed, and to ensure an adaptable approach to both long-term and freelance/gig labor markets. 34 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV While participant tracking is key to observation of outcomes, assessment fatigue was prevalent among TEA beneficiaries. Efforts to reduce assessment fatigue such as informing participants of the need for frequent monitoring calls, alternative and less intrusive monitoring options could be considered. 4.6 EVALUATION QUESTION 6 What are the key approaches, methods, or interventions that can better contribute to improving employment and retention rates? “I wish TEA would have provided me with equipment and raw materials so I can start crafting jewelries in my home.” – Female, South "Financial support from USAID to get my own sewing machine would have been appreciated, so I could start working from home." – Female, South 4.6.1. Employment flexibility – In 2007, when LTUC conducted a gender assessment, four recommendations emerged: 1) Work closely with the government and private sectors to provide more supportive working environments for women, i.e., create part-time jobs for them, job sharing, working remotely, and encouraging home-based businesses; 2) Integrate gender issues and cultural communication skills into the career readiness training (to start focusing on gender-sensitive issues); 3) Introduce parent involvement sessions and encourage parents to attend to provide them with awareness about the labor market and the importance of women's roles in the working environment; and 4) Increase the internship opportunities for females during the training to provide them with the chance to experience how the work environment operates before their graduation. While points 2 and 3 had been adopted to a great extent, as evidenced in this evaluation, there were still opportunities to better improve flexible employment and internships in order to support improved integration of women into the force in a country with female labor market participation below 15%. Due to the responsibility with home duties, frequent feedback from female participants, especially from those trained for the garment sector or handicrafts, was a request for support from TEA IPs for a micro-loan program that could help them establish home-based enterprises. Globally and in Jordan, many of the other donor-funded vocational training programs offer an entrepreneurship component, and it is unclear if the feedback from women came due to their having heard about these other programs. “We observed that women prefer home-based jobs. For example in Garment sector, one of the beneficiaries worked to save-up for a sewing machine. After she did that, she resigned and started her own home-based business. Unfortunately, self-employment was not counted.” – TEA IP As discussed earlier, TEA's funding disbursement structure depended on the IPs' delivery on training and short-term (3 months) and long-term (6 months) employment retention. Employment under TEA requires proof of employment in the “form of a contract or agreement letter that includes employment period, duties and benefits (salaries, bonuses, or commissions). The document should clearly state that named employer will employ a skilled beneficiary in a specific job.” 11 This definition 11 Indicator 1.2.a notes “Only employment lasting three months or more is counted under this indicator.” TEA Indicator 1.2.1.a. requires proof of employment in the “form of a contract or agreement letter that includes employment period, duties and benefits (salaries, bonuses, or commissions). The document should clearly state that named employer will employ a skilled beneficiary in a specific job.” These criteria do not align with home-based, freelance, informal, digital and self￾employment. 35 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV challenges IPs to respond to evolving conditions of the job market and to be responsive to demands in the informal sector and the digital economy, as well as self-employment through entrepreneurship. The KIIs with the IPs confirmed that there was a need to have a more flexible definition of employment. All the IPs talked about the need for the possibility of self-employment. They also all mentioned that due to the economic crises after COVID and lack of formal employment opportunities, many people turned to informal employment, noting that the TEA training interventions were equally applicable in the informal sector, and there should be a way to tap into this part of the labor market, a point that was raised in KIIs conducted with LTUC. One example of more informal employment terms was the beauty sector, where employment was seasonal and often on individual contractor terms. Employees would be called when needed for specific assignments such as weddings or “gigs.” Employment on these terms did not fit with the definition used by TEA and could therefore not be counted as jobs held for 3 to 6 months with a contract or agreement letter that included the employment period, duties and benefits (e.g., salaries, bonuses, or commission).12 This potentially created a bias towards encouraging IPs to focus on persons and jobs that more easily fit these criteria, which may discount or exclude employment that was home-based, freelance, informal, digital, or self-employed, noted as preferred by women. Inclusion of these types of jobs may have supported increased employment, particularly for women. This issue was brought to the forefront with the COVID-19 pandemic and the consequent shift in the labor market to increased informal, gig, home-based, and platform employment.13 LTUC felt that the focus on traditional vocational training, at times, did not keep up with evolving new market demands. LTUC would have liked to amend their curricula to match recent market trends to respond particularly to social media and filmmaking, industries that are currently growing. However, this was not possible with the focus on more traditional vocational training approaches. The labor market is constantly in flux, and activities such as TEA must follow the market trends. Should USAID in the future implement similar activities in Jordan, they should strongly consider conducting a study of female labor market integration to identify best practices that would be included in programming. Alternative working arrangements such as gig work and job-sharing, and home-based working arrangements should also be explored to make the implementation approach more flexible and family friendly. New and emerging areas should be included to supplement the traditional vocational training courses. 4.6.2. Early employer involvement – An aspect of the activity that the IPs themselves have focused on was the frequent interaction between employers and the IPs. The idea was that by involving the employers early in the training and employment process, the IPs created commitment from the employers' side. The employers, on the other hand, could influence how the training was conducted to ensure that it was relevant to their needs. The approach taken by EFE-Jordan illustrates this interaction. They conducted two rounds of interviews to make sure the potential candidates had a 12 Mission Indicator 1.2.a notes “Only employment lasting three months or more is counted under this indicator.” TEA Indicator 1.2.1.a. requires proof of employment in the “form of a contract or agreement letter that includes employment period, duties and benefits (salaries, bonuses, or commissions). The document should clearly state that named employer will employ a skilled beneficiary in a specific job.” These criteria do not align with home-based, freelance, informal, digital and self-employment. 13 According to the World Economic Forum, COVID-19 and accelerated technological adoption risks are “creating a ‘double disruption’ in labor markets, with many jobs unlikely to return. At the same time, the overall projections for job creation in the next five years are still higher than those of job losses. New jobs are expected to emerge due to shifting demand patterns.” WEF 2021. “Shaping a new labor market for the post-pandemic economy.” Global Agenda – The Davos Agenda, 2021. WEF website. 36 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV good understanding of what was required. The employers had the option to participate in the second round of interviews, thereby creating a connection between the employer and possible future employees early in the process. However, from KIIs, the evaluation found that most of the employers did not have this option for interaction. Most of the employers interviewed said that they had been asked about their labor needs but that they had not had an influence on the specific training curriculum, nor did they interview candidates early. However, should a similar training program be implemented in the future, it might be an area that can inspire possible IP/employer interaction and cooperation. 4.6.3. Labor market assessment – Central to TEA and any other similar training program is the labor market assessment conducted to identify possible areas for employment and training. As the number of training programs has increased, the donors have not relaxed their requirements for a market assessment for each program implemented. In the case of TEA, the three IPs took very different approaches in assessing the market. LTUC opted to use an assessment conducted earlier, while EFE￾Jordan conducted what they call a ‘mini-assessment’. TVTA had a running assessment where they identified opportunities based on geographical supply shortages. When looking at the employment status of the different sectors, there were considerable differences in the level of unemployment and dropout. Of the beneficiaries surveyed who were trained in the garment sector, only 39% were employed at the time of the survey. The majority of the beneficiaries for the garment sector are women, and they had a 12% incidence of dropping out of the labor market altogether. Additionally, the garment sector is known to have a difficult work environment which might mean that more employees drop out. With these two parameters in mind, having 61% of the beneficiaries trained and then unemployed was high. A labor market assessment does not need to be a major report. But it should ideally be undertaken close to the onset of implementation to ensure that the labor needs identified are current. It should contain a loosely sketched-out indication of possible areas for training, the market demands identified within the area of training, and possible areas of concern within the single sector (such as challenging work environment, long hours, transportation issues, etc.). This information will make it easier to map out ways to mitigate potential challenges the beneficiaries will face when moving from training to employment. 4.6.4. Stipend and transportation cost – The IPs highlighted recruitment difficulties due to the crowded market for vocational training-oriented employment programs. One area that often swayed potential beneficiaries when choosing among competing programs was the availability of stipends during training. Program decisions like adding a stipend should, of course, not just be made to better compete more effectively with other donors. However, the lack of earnings during training might be an obstacle for some beneficiaries, as their families might have depended on their earning potential during that period. By not providing a stipend to the beneficiaries, some of the most vulnerable and marginalized potential beneficiaries might be excluded. Although no stipend was offered, TEA provided the participants with transportation funds. This approach meant that beneficiaries living further away from the training centers did not incur extra costs while attending the training. Some employers also offered the newly employed transportation funds. In some cases, this cost was paid through the TEA during an on-the-job probational training period. The employer then took over the payment when the beneficiary joined as a fully-fledged employee. TVTA said that transportation was critical in retaining youth in employment. “Transportation 37 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV is the number one challenge in Jordan, so if that problem is solved, 50% of the employment problems are resolved.” 4.6.5. On-the-job training and job shadowing/internships – As noted by the beneficiaries interviewed, on￾the-job training was identified as one of the most desirable methods to improve skill-building and improved placement. In line with global best practices, on-the-job training better prepares beneficiaries for the job as they receive firsthand experience in performing tasks related to the role from a coworker or supervisor. This experiential training is often used in jobs that involve hands-on work or the use of specialized equipment, software or machinery, and can increase productivity and efficiency in specific industries. It is also an effective approach for labor market entrants. From the employer's perspective, on-the-job training/internships can shorten the learning curve of a new hire and can also potentially increase retention by ensuring that job expectations are experienced and understood before hiring. 4.6.6. Tracking promotion – As long-term relationships are built between training institution and employer, and as beneficiaries are hired on a regular basis, there may be an opportunity to track the promotion of beneficiaries. If promotions were to be tracked, this data could serve to inform training courses linked to the performance (promotion, retention, salary grade) of graduates, which in turn could support advertisements and course design/replication and showcase stellar performers. At the same time, as the pool of employed beneficiaries grows, these employed beneficiaries could serve as recruiters and ambassadors in their sectors, which may be capitalized on by the technical vocational education and training (TVET) training institutions. 5. RECOMMENDATIONS 1. Consider building synergies between IPs and replicating proven curricula across IPs, where possible, especially in relation to soft skills. 2. Continue to target marginalized groups such as women, with an increased focus on better targeting of poverty pockets. Continue to utilize effective approaches to local outreach and address cultural barriers among families to encourage female labor force participation across sectors. Consider a differentiated approach to recruitment of Syrians that better aligns with work permits and coordinates with refugee social safety nets, as well as persons from poverty pockets by defining and monitoring poverty targets. Where successes are found, consider replicating them among other IPs. 3. Should future programming engage multiple implementers, ensure that recruitment implementation criterium be equally enforced for all IPs to ensure a level playing field when promoting employment opportunities and recruiting trainees, in order to ensure attainment of subpopulation targets. Consider extending contractual target-meeting requirements to recruitment targets, in the same manner as the employment targets. Consider improved oversight and unification of database tracking among IPs, 4. Continue consultation and collaboration with private sector employers at the earliest stages. Consider requiring customization of all training in consultation with employers and trainee interviews as a best practice to inform curricula, skills in demand, and sectoral and geographic market demand to maximize employment and retention outcomes. 5. In future activities, a USAID labor market assessment should ideally clearly define the supply side shortages in the labor market and challenges in fulfilling these. To mitigate differentiated approaches by IPs in the future, ensure that market assessments are more clearly defined, and standards are met, and ensure that such assessment of market needs, trends, and fluctuations is adapted into training curricula, employer engagement, and job placement. 38 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV 6. The Activity focused on providing long-term employment (3-6 months or more) in the labor market. However, some of the trainees, especially women, had difficulties in this kind of employment. For future activities, explore the differences between the formal and informal markets, the opportunities for entrepreneurship or self-employment, and digital and freelance markets. For women, consider an alternate form of employment criteria, in recognition of the gender roles women take on that may prevent them from being available for long-term employment under TEA criteria. Consider creating opportunities for flexible employment and internships in order to support improved integration of women into the labor force in a country with female labor market participation below 15%. Similarly, consider instituting a micro-loan program that could help women establish home-based enterprises based on skills acquired at TEA IPs, in line with global evidence in TVET. 7. As a strategy for regular job placement, continue engaging large and sectoral employers with the same training needs (such as garments). As a strategy for increased reach outside of Amman and outside of large employers, consider systematically mapping employers by sector, geography, and the number of employers in order to better inform job placement and regional market demand. 8. To ensure higher rates of initial employment retention, consider instituting stipend and transportation costs, as well as on-the-job training to mitigate challenges related to employee departure. To ensure continued tracking of the trainees, this stipend payment could be tied to a mobile payment platform so that only trainees that kept the same phone number or provided updated contact information could continue to receive this payment. 9. In future iterations of TVET programming, consider tracking promotion of trainees to inform and assess training courses that are linked to performance (promotion, retention, salary grade) of graduates, which in turn could support assessment of the quality of implementation. Similarly consider utilizing tracking of dropouts/unemployed to identify areas of improvement. 39 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV ANNEXES 40 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV ANNEX 1: STATEMENT OF WORK The entire SOW is included in the PDF version of this report shared with the Mission. 57 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV ANNEX 2: INCEPTION REPORT The entire Inception Report is included in the PDF version of this report shared with the Mission. 96 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV ANNEX 3: ANALYSIS DESIGN AND METHODS Annex 3.1. Evaluation Matrix Evaluation Topic Type of Answer/ Evidence Needed Methods for Data Collection Data Analysis Methods Method Data Source(s) 1. To what extent did TEA achieve its intended outcomes (i.e., beneficiaries trained by gender, nationality, education level, and those from poverty pockets or marginalized groups)? Number of people trained disaggregated by sex, nationality, education level, governorate Doc review 1-TEA indicator data, IP tracking sheets for participants and employers and data verification summary reports 2-USAID staff Frequencies, comparative analysis 1.1 How was TEA inclusive in reaching target groups of beneficiaries (e.g., Jordanian and Syrian, men and women, those who did not pass Tawjihi, those with limited financial resources or from poverty pockets and marginalized groups) residing in Jordan? Special steps taken by IPs to reach and address needs of these marginalized groups (e.g., approaches to outreach, course design, accessibility, geographic area, consideration of cultural norms) KIIs IPs and USAID staff Content analysis, triangulation FGDs, survey Participants 1.2 What are the key barriers to the participation of women and other marginalized groups? Identify barriers KIIs, FGDs, survey IPs, Participants, Employers Content analysis, grounded theory, frequencies Doc review TEA gender analysis, TEA progress reports and data verification summary reports Content analysis 2. To what extent are TEA interventions broadly relevant to the needs of target beneficiaries and local employers? 2.1 More specifically, were the training courses relevant to local market needs considering the perceptions of both youth and employers? Identify needs of target beneficiaries and needs of employers Document review TEA market assessments and other available (online) reports and data on Jordan's TVET sector by GOJ and other development partners Content analysis, triangulation KIIs, FGDs, survey USAID staff, GOJ, Employers, Participants Course characteristics, skill level, training modalities Doc review KIIs, FGDs IPs, Participants Content analysis, comparative analysis Expectations of participants and employers relative to training and Employment KIIs, FGDs, survey Employers, Participants Content analysis, comparative analysis 97 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV Reasons that participants remain with employer/leave employer (relative to training) 3. What are the rates of Employment and retention for TEA beneficiaries by gender, nationality, financial status, and sector of training – both within fixed periods (e.g., at least three months, at least six months) and as trends over time? Number of participants employed for 3 months and 6 months, segregated by demographic variables Doc review Participant tracking sheets Data compilation, content analysis, trend analysis, comparative analysis Survey Employers, Participants Rates of retention for participants trained in specified time periods, separated by demographic variables Doc review Participant tracking sheets 3.1 Are there differences in Employment and retention by category? 3.2 What factors are likely driving any observed differences? Differences in retention by demographic variables and gender Doc review Participant tracking sheets, TEA Progress Reports Factors cited for retention and dropout KIIs, FGDs, survey USAID staff, IPs, Employers, Participants 4. How likely are TEA interventions to be sustained over time? Why is this the case? Stakeholders' stated intention and ability to continue/discontinue KIIs USAID staff, GOJ, IPs, Employers Content analysis, triangulation 5. What are the key lessons learned about TEA design and implementation successes and challenges which can be applied to future workforce development interventions? Identify design and implementation challenges and successes KIIs USAID staff, IPs, Employers Content analysis Steps that were taken that had positive impact on overcoming challenges Recommendations for incorporating workforce training into existing and future EDE programs 98 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV Annex: Sampling 3.2 – Participants' Survey Sampling Approach The sampling method adopted for this survey is Proportional Stratified Random Sampling, in which a simple random sample is calculated for the study population. The total sample will then be split equally across the three IPs. Mutually exclusive strata will be defined and for each stratum, a portion of the third of the total sample size is assigned, reflecting its actual size (%) against the population size for that IP. The adopted methodological framework guarantees quantitative generalizability of respective results at a confidence level of 95% and a margin of error standing at 5%. That given, the total sample size would stand at 312 (104 per IP). Participant surveys will oversample (increase to 107-120per IP) to guarantee representation of certain groups (those who stand lower chances in the randomized draw). Sampling Method Population Size Response Distribution Factor Confidence Level Margin of Error Proportional Stratified Random Sample Total number of the Training for Employment Activity graduates who retained Employment for a minimum of 3 months 50% 95% 5% Sample Size Sample Size (per IP) The sample size n and margin of error E are given by x = Z(c/100)2r(100- r) n = N x/((N￾1)E2 + x) E = Sqrt[(N - n)x/n(N -1)] - N is the population size, r is the target fraction of responses, and Z(c/100) is the critical value for the assumed confidence level. - Assumed response distribution = 50% Sample Size (total)/3 - Due to the pre-determined uniformity of sample sizes across 3 IPs with different population sizes, the actual confidence level may fluctuate, but is guaranteed at 95% as the minimum per IP. Stratification Parameters (conditioned with data availability in the participant tracking files provided by the Monitoring, Evaluation, and Learning Activity) ▪ 3-months employment retention ▪ 6-month employment retention ▪ Male ▪ Female ▪ Governorate Representation deficit correction After the sample is drawn, a proportional complementary addition of sample points will be conducted to guarantee the following: ▪ Proportional representation of graduates from poverty pockets ▪ At least 30% female respondents ▪ Proportional representation of refugee graduates 99 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV Sampling The below tables provide an overview of the numbers of TEA participants, with breakdowns by IP, gender, nationality and Governorates on 3 levels as follows: 1) Beneficiaries: full population of TEA participants as extracted through the compiling and cleaning of all provided tracking sheets 2) Study Population: full population of TEA participants who fit the sampling criteria as per the Inception Report 3) Study Sample: all participants from the Study Population who were selected using an auto￾randomized factor as per the inception report. The Study sample consists of 3 unique sets: a) Primary set: all those selected from the Study Population at the first stage of sampling as per the stratification approach provided in the Inception Report b) Backup set 1: a secondary set of participants randomly drawn from the reminders of the Study Population that were not selected for the primary set, with matching stratification c) Backup set 2: another secondary set of participants randomly drawn from the reminders of the Study Population that were not selected for neither the primary set nor the backup set 1, with matching stratification (some strata for TVTA were less than the first two sets due to the lower numbers of participants) Beneficiaries Category Beneficiaries (#) 4575 % of Grand Total EFE￾Jordan LTUC TVTA 56% 24% 20% Female 2233 440 397 67% Male 338 641 526 33% Jordanian 2020 902 868 83% Syrian 546 174 38 17% Other nationalities 5 5 17 1% Ajloun 362 2 1 8% Amman 845 730 394 43% Aqaba 47 3 14 1% Balqa 78 42 69 4% Irbid 159 81 135 8% Jerash 99 9 5 2% Karak 239 0 49 6% Ma'an 162 1 95 6% Madaba 10 35 46 2% Mafraq 186 26 1 5% Tafilah 235 1 0 5% Zarqa 148 151 114 9% Governorates with high rate of poverty incidence 55 participants did not have IP labels in the tracking sheets. Therefore, these were not calculated in the Grand Total. 100 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV Study Population Category Study Population (#) 2167 % of Grand Total EFE￾Jordan LTUC TVTA 43% 21% 37% Female 832 140 377 62% Male 89 309 420 38% Jordanian 903 439 743 96% Syrian 16 10 38 3% Other nationalities 2 0 16 1% Ajloun 303 0 1 14% Amman 169 330 331 38% Aqaba 9 0 14 1% Balqa 56 14 62 6% Irbid 3 18 134 7% Jerash 50 4 5 3% Karak 140 0 0 6% Ma'an 0 1 95 4% Madaba 1 13 44 3% Mafraq 0 9 1 0% Tafilah 168 1 0 8% Zarqa 22 59 110 9% Governorates with high rate of poverty incidence Study Sample Category Study Sample (#) 986 % of Grand Total EFE￾Jordan LTUC TVTA IP % from Primary Set Total (including oversampling units) 33% 33% 33% IP % from Total Sample (including oversampling units) 37% 31% 32% Female 324 130 132 59% Male 36 178 186 41% Jordanian 343 298 276 93% 101 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV Syrian 16 10 38 6% Other nationalities 1 0 4 1% Ajloun 120 0 0 12% Amman 66 224 141 44% Aqaba 6 0 0 1% Balqa 24 10 15 5% Irbid 18 54 7% Jerash 18 3 2 2% Karak 54 0 0 5% Ma'an 0 1 72 7% Madaba 0 9 0 1% Mafraq 0 3 0 0% Tafilah 66 0 0 7% Zarqa 6 40 34 8% Governorates with high rate of poverty incidence Reached Respondents Category Respondents (#) 398 % of Grand Total EFE￾Jordan LTUC TVTA 31% 35% 34% Female 109 61 38 52% Male 14 77 99 48% Jordanian 119 130 134 96% Syrian 3 5 2 3% Other nationalities 1 0 4 1% Ajloun 40 0 0 10% Amman 23 77 96 49% Aqaba 2 1 0 1% Balqa 4 8 1 3% Irbid 2 10 15 7% Jerash 6 2 2 3% Karak 15 1 8 6% Ma'an 0 15 5 5% Madaba 0 2 0 1% Mafraq 0 1 0 0% Tafilah 26 0 0 7% Zarqa 5 21 9 9% Governorates with high rate of poverty incidence 102 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV ANNEX 4: INFORMATION SOURCES 4.1 INFORMATION SOURCES : BIBLIOGRAPHY Donor, Country and Project Documents • AMELPs • DevResults indicator data • Participant and employer tracking sheets • IP progress reports • The Training for Employment Activity for gender analysis Secondary Sources • International Labour Organization, ILOSTAT database. • World Bank, "Jordan Economic Monitor - Uncertain and Long Trail Ahead", Spring 2021, p.11 • Education for Employment official website • TVTA official website • Luminus Technical University College (LTUC) official website • Department of Statistics Jordan, poverty status report Jordan, 2010 103 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV 4.2 INFORMATION SOURCES: LIST OF INTERVIEWS 4.2.1. Key Informant interview Date Organization Name Title Contact Information July 4 Luminus Eman Nayef Bakheet Project Manager e.altheeb@luminuseducation.com July 7 Luminus Field Team NA NA July 7 Luminus The dean The dean NA July 4 EFE-Jordan Isra'a Awajan Project Manager iawajan@efejordan.org July 4 EFE-Jordan Janan Ababneh MEL POC jababneh@efejordan.org July 7 TVTA Ahmad Aqel COP manager@ata-jo.com July 4 MOL Mai Alyan Policy Director July 11 USAID/EDE Rebecca Accuna Deputy Office Director racuna@usaid.gov July 1 USAID/EDE Haneen Al Rasheed COR halrasheed@usaid.gov July 1 USAID/FMO Bassam Hamdan Voucher Examiner bhamdan@usaid.gov July 5 USAID/AAO Alla' Sawaked Contracts Specialist asawaked@usaid.gov July 5 USAID/ PRO Anna Karmandarian (ex) MEL Backstop akarmandarian@usaid.gov July 5 USID/PRO Kenana Amin MEL Gender Lead kamin@usaid.gov 104 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV 4.2.2. Group interviews and Key informant interviews with beneficiaries Date Region # of attendees (GI) July 11 Central females 2 July 12 Central males 3 July 13 North females Jor 0 July 13 North+Central Syrians 5 July 14 North males Jor 3 July 15 South females 2 July 15 South males 2 July 25 Central females 0 July 27 Central females 2 July 28 South females 3 July 29 North females Jor 1 Date Region # of interviewees (KIIs) July 25 Central females Jor 5 July 27 Central males Jor 1 July 28 North females Jor 3 July 29 North males Jor 4 July 28 South females Jor 4 105 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV 4.2.3. Key Informant Interviews with Employers Date Employer Region IP worked with July 11 Al Anbaat Association South TVTA July 11 Country Club Central EFE-Jordan July 12 Classic Fashion South EFE-Jordan July 13 Rehan Central TVTA July 28 Sameh Mall Central TVTA July 28 Alawneh Exchange Central TVTA July 29 Alawneh Exchange North TVTA July 29 Ocean Fresh Fish Central LTUC July 30 Almasira Textiles North LTUC August 5 Lamsat Asma Beauty Shop South LTUC August 5 Lumi Market / Manaseer North EFE-Jordan 106 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV ANNEX 5: DATA COLLECTION TOOLS USAID KEY INFORMANT INTERVIEW 1. Economic Development and Energy Office (EDE) 2. Program Office (PRO) 3. Financial Management Office (FMO) and Acquisition and Assistance Office (AAO) EDE 1. Discuss reason for departure from traditional design of one prime and multiple subs. What were you trying to achieve that hadn’t been achieved in the past? 2. How were IPs selected? Particular criteria or areas of strength of each? 3. Given that there are three IPs, discuss some of the management and reporting challenges (e.g., time to manage, capacity, reliability of reporting, ability to get big picture of TEA progress as well as individual IP progress and issues) 4. Discuss benefits/challenges of local vs. American IPs (as experienced through TEA) 5. Discuss performance-based payment approach benefits and challenges 6. Sustainability: was it discussed with IPs, how do you envision it working in Jordan? 7. Lessons learned from this design and implementation PRO 1. Provide overview of PRO’s involvement with TEA 2. Contract says # beneficiaries set annually with focus on TVET – done? Why or why not? 3. Contract says beneficiaries have limited financial resources and above-average academic standings – was this just a suggestion or how was this measured? 4. Given that there are three IPs, discuss some of the of the management and reporting challenges (e.g., time to manage, capacity, reliability of reporting, ability to get big picture of TEA progress as well as individual IP progress and issues) 5. Discuss benefits/challenges of local vs. American IPs (as experienced through TEA) 6. Discuss performance-based payment approach benefits and challenges 7. Lessons learned from this design and implementation FMO and AAO 1. Provide overview of FMO/AAO’s involvement with TEA 2. Given that there are three IPs, discuss some of the of the contracting and financial management challenges 3. Discuss performance-based payment approach benefits and challenges 4. Lessons learned from this design and implementation 107 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV MINSTRY OF LABOR KEY INFORMANT INTERVIEW Head of the Directorate of Policies, Strategies and Institutional Performance Development, Eng. Mai Alyan 1. What is your familiarity with TEA or the implementing partners? Do you have any thoughts on the effectiveness of TEA or the workforce development programs conducted by the implementers? 2. What do you see as the main challenges facing employment-oriented training programs in Jordan? 3. In your opinion, what programs have contributed most to workforce development and youth employment, and what factors do you think have been most important to their success? 4. What role do you see for organizations such as LTUC, TVTA, LTUC and others like them in workforce development in Jordan? 5. Beyond donor funding, how do you think workforce development programs could continue? 6. Do you have recommendations for designing future employment-oriented training programs? 108 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV IMPLEMENTING PARTNERS’ KEY INFORMANT INTERVIEW Note: It is anticipated that there will be multiple consultations with various IP staff regarding outreach, training, placement/retention, and management. Only relevant topics will be discussed based on the staff member’s involvement. 1. Please describe your understanding of the target group. Acknowledging that some targeted beneficiaries are hard to reach, can you talk about how you conducted outreach to women, marginalized groups, those who did not pass Tawjihi, those with limited financial resources? (e.g., local partners, type of social media, etc.) Probe: Which were most effective and why? 2. What criteria was used to select beneficiaries? (Probe: ask about age, educational background, unemployment status, financial resources, etc.) Were any of these criteria more highly prioritized for example? 3. Can you talk about some of the barriers that you may have encountered in motivating youth to participate? Do you have observations of how the barriers differed for these groups? For example, transportation, timing, location and duration of the training, doubt that it would lead to employment). 4. Some skills training programs offer financial incentives to participants. Did the lack of financial incentive pose any challenge to getting participants in your program? 5. Can you walk us through the process of how you designed or modified your program to overcome barriers and address the special needs of marginalized groups? (Probe: different literacy levels, gender and nationality? 6. How were your staff sensitized to the special needs of the participants from marginalized groups? (Probe: did you conduct training? Did an outside organization conduct training? Were these trainings useful?) 7. Can you talk about how you established expectations and commitment with participants? (Probe: promise of employment and certification upon completion? Financial incentives? Transportation reimbursement?) 8. Did trained beneficiaries receive certification upon training completion? If so, what type of a certificate were they awarded? (example, CAQA OR Jordanian Teachers Syndicate (JTC). 9. What were the steps taken to ensure that the placement of the different groups of beneficiaries was responsive to and sensitive to family concerns and social norms in the different geographical locations? (Specific to women). Can you speak more about associated challenges? 10. As TEA was intended to meet the requirements of private sector employers, can you talk about how you approached assessing local market needs and requirements in terms of skill levels, number of people needed, pre-identifying employers who would hire? (Probe: Were you involved with any consultations with employers to identify skills required or to determine skill level? Did you pre-identify participant placement? Did you develop your own labor market assessment or use an existing one prior to training curricula development?) 11. Can you speak more about the courses: Were these customized in accordance to employer needs/preference or did you offer existing courses that you had already developed? What was the process you used in designing the curricula, for example did you combine theory and practice, on-the-job-training with employers? Please elaborate. 12. There seems to be a challenge in Jordan with employment retention among youth. Were there some perceptions among youth that you had to overcome and how did you do it? How did you go about understanding and meeting expectations? Was there some kind of agreement or commitment between yourselves and youth to manage expectations? 109 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV 13. Can you speak to more about the perceptions of employers towards workforce training? Were there some employer perceptions that had to be overcome? If so, how was that done? How did you go about understanding and meeting these expectations? 14. What was the process used for placing graduates? (Probe: Did it vary by sector? By employer? Was it based on one-on-one interviews? Or was it by mass recruitment?) 15. Were there any specific challenges relating to placing graduates in jobs? Any noticeable gender challenges? Did these differ by the beneficiary group targeted? Did It vary by sector? By employer? By location? Can you give examples? 16. What are the challenges in retention? Any noticeable differences among groups? 17. Any noticeable difference in challenges for 3- and 6-month retention? 18. Any specific challenges that were faced in tracking employment and retention? 19. Can you speak about the particular challenges for placing Syrians? Were work permits an issue? How was this overcome? 20. Were there any noticeable trends over time with employment rates of graduates (up or down)? (Probe: women, Syrians, Jordanians, sector, geography) 21. To your knowledge what are some of the factors that are likely driving any observed differences in employment and retention rates (across location, sector, gender, nationality)? 22. If USAID weren’t paying for job placement, would the private sector and training providers be able to continue? Do you think employers would be willing to pay? 110 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV EMPLOYERS’ KEY INFORMANT INTERVIEW Basic employer information will include sector and total number of people employed by the business. 1. How many graduates did you hire? 2. How did you find out about TEA? 3. Discuss whether employer was consulted about the hiring needs (Probe: areas of skill, skill level, number of employees you needed) 4. Discuss how expectations (relative to employer relationship with IP, and what employer would get in return) were set. (Probe: was an agreement signed, hiring commitments made, graduate competency)? 5. Discuss whether employer had any reluctance about engaging with the program or hiring graduates. 6. What were your expectations of TEA graduates and to what extent/in what ways were your expectations met/not met? 7. Any observations of the factors that are most significant to TEA graduates leaving/staying employed. (Probe: graduate expectations, job hopping, transportation) 8. Any observations on categories (demographic characteristics) of TEA graduates more likely to stay or leave? 9. Discuss how TEA compares with other job training programs you have experience with. 10. Discuss whether TEA graduates are valuable enough to the business so that the employer would consider paying for/contributing to the cost of training? Is there something that could make it valuable enough to pay? 11. Any recommendations for how TEA or other job training programs could better meet the needs of the business? 111 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV PARTICIPANT SURVEY I- General Information 1. Name (Text) 2. Age group: a. Younger than 18 b. 18-35 c. Older than 35 3. Gender: (Single select) a. Female b. Male 4. Nationality: (Single select) a. Jordanian b. Syrian c. Others…………… (Text) 5. Period of graduation: (Single select) 1. 2018 2. January – July 2019 3. August 2019 – February 202014 4. March – December 2020 5. 2021 6. What motivated you to participate in the training program Check what applies: (Multi select) a. To satisfy family financial need; e. To be productive and to contribute to my family/community b. To get out of the house; c. To become financially independent; f. To acquire some decision-making Power within the household; d. To have a personal sense of purpose; g. To follow on the footsteps of other role models h. To get an accreditation/certification i. Family member wanted me to participate j. My friend was enrolled so he/she encouraged me k. Others………(Text) II- Training 14 On March 17, 2020, the country began a lock-down of non-essential workers due to COVID-19. Movement outside was severely restricted. 112 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV 7- Which center did you / are you attending? (Single select) a. Education for Employment - Jordan (EFE-Jordan) b. Luminus Technical University College (LTUC) c. Technical Vocational Training Academy (TVTA) 8- What course of training did you attend? (Single select) a. Manufacturing b. Wholesale & retail trade, c. Repair of motor vehicles & motorcycles d. Administrative & support service activities e. Construction f. Human health & social work activities g. Accommodation & food service activities (Hospitality) h. Electricity, gas, steam & air￾conditioning supply i. Transportation & storage j. Mining & quarrying k. Agriculture, forestry & fishing l. Information & Communication m. Other service activities, 9- How did the training provider make it attractive to you? (Multi select) a. Provided transportation incentives b. Provided accredited certification c. Short training program d. Job placement opportunities e. Sector/field of training was unique f. The name and reputation of training provider g. I am a refugee and not eligible to participate in training with others <> h. Provided work permit after graduation <> i. Others, ………… (Text) 10- Did you face any barrier in attending training for employment? (Single select) a. Yes << Open 10.1>> b. No << Skip to section III>> 10.1 What are those barrier(s)? (Multi select) a. Culturally for females to get an education in this field b. Didn't pass Tawjihi c. Mixed environment (females and males) d. Got married e. Motherhood or pregnancy f. Parents/Spouse/ Other family member disapproval g. Morning training sessions h. Evening training sessions 113 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV i. I live far away from where the training providers are located j. Transportation issues k. Other family responsibilities l. Others. Specify III- Employment and retention 11- Were you placed in a job after your graduation by the training provider? a. Yes b. No, <> 11.1- How long have you/did you work there? a. Less than 3 months b. 3 months c. 4 to 5 months d. 6 months e. More than 6 months 12- What is your current work employment status now? a. Employed with the same employer where I was placed by training provider b. Employed with a different employer than the one where I was placed c. Unemployed and looking for a job <> d. Unemployed and not looking for a job <> 12.1- Employment type 1? <> a. Formal with contract b. Informal (daily or seasonal work, or just no contract?) 12.2- Employment type 2? <> a. Full time b. Part time 12.3 Same sector/field of graduation? <> a. Yes << Skip to 11.3.1>> b. No << Skip to 11.3.2>> 12.3.1 If yes, for how long? a. Less than 3 months b. 3 months c. 4 to 5 months d. 6 months e. More than 6 months 12.3.2-If no, Why? a. Tasks were hard b. Not confident of my technical skills c. Got married <> d. Motherhood or pregnancy <> e. Parents/Spouse/ Other family member disapproval f. Didn’t find any job in the market 114 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV g. Employers' location is far and high transportation cost or transportation not available h. Low salaries offered i. Others. Specify 13. Did you face any barriers in trying to obtain a job? (Single select) a. Yes << Open 13.1>> b. No << Skip to section IV>> 13.1 What are those barrier(s)? (Multi select) a. Culturally for females to get an education in this field b. Didn't pass Tawjihi c. Mixed environment (females and males) d. Got married e. Motherhood or pregnancy f. Parents/Spouse/ Other family member disapproval g. Morning shifts h. Evening shifts i. I live far away from where the employers are located j. Transportation issues k. Other family responsibilities l. Others. Specify IV- Perception of Youth 5-To a great extent 3- To a medium extent 1- To a poor extent 0- Not at all 14. To what extent did the skills training prepare you for doing your work? 15. To what extent did the program provide you with soft skills that you need for your work? 16. To what extent do you feel that job placement assistance has contributed / will contribute to you finding a job and staying employed 17. To what extent did the program meet your expectations? 115 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV EMPLOYER SURVEY I- General Information 1. Business Name (Text) 2. Sector (Single select): a. Manufacturing b. Wholesale & retail trade, c. Repair of motor vehicles & motorcycles d. Administrative & support service activities e. Construction f. Human health & social work activities g. Accommodation & food service activities (Hospitality) h. Electricity, gas, steam & air-conditioning supply i. Transportation & storage j. Mining & quarrying k. Agriculture, forestry & fishing l. Information & Communication m. Other service activities 3. Location: <> 4. Number of employees: (Number) 5. Legal Form of Business: (Single select) a. LLC b. Privately Held c. Sole Ownership 6. Which organization did they graduate from? (Multiple select) a. LTUC b. TVTA c. EFE d. Don’t know II- Employment 7. How many TEA graduates did you hire? And for how long? <> a. Jordanian male (Number) / Duration b. Jordanian female (Number) / Duration c. Syrian male (Number) / Duration d. Syrian female (Number) / Duration 116 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV 1. 3 months 2. 4-5 months 3. 6 months 4. More than 6 months 5. Multiple hire 8. Prior to hiring, were you consulted about your hiring needs in terms of skill level of employees? (Single select) a. Yes b. No 9. Prior to hiring, were you consulted about the number of employees you needed? (Single select) a. Yes b. No III- Retention 10. For employees who have stayed more than 3 months, what factors do you think contribute most? (Multi select) a. My business offers above minimum wages (220 JODs) b. My business provided Social Security c. My business provided Health insurance d. My business provided formal contract e. My business sponsored work permits for Syrian graduates f. Flexible shifts (Morning and Night) g. I am the only employer within this sector h. My business provides transportation services i. My business provides childcare services j. Others, (Text) 11. If graduates left employment, what do you think are the reasons? (Multi select) a. Transportation challenges b. Unable to provide formal contract c. Unable to provide work permit d. Graduate’s personal situation e. Graduate wanted more money f. Graduate got another job g. Graduate could not perform to required level h. Work hours/conditions too challenging i. None of them left employment j. Others, (Text) 12. Within your company, do you have observations on categories of employees more likely to remain employed? a. Jordanian men b. Jordanian women c. Syrian men d. Syrian women e. Those who passed Tawjihi f. Those who did not pass Tawjihi g. Those with most financial need 117 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV IV- Perception of Employers 13. What factors gave you confidence to hire a TEA graduate? (select all) a. They consulted with me on my needs b. We signed an agreement that I would hire graduates c. I have previous experience with the training organization d. They have a good reputation e. The graduate(s) I hired have certification f. I was impressed with the graduate(s) interviewed g. I didn’t have much confidence – I was just taking a chance 5-To a great extent 3- To a medium extent 1- To a poor extent 0- Not at all 14. To what extent did graduates meet your expectations? 15. To what extent are you satisfied with the competency/ skill level of graduates? 16. To what extent do graduates’ personal skills (e.g., communications, timeliness) satisfy your needs? V. Sustainability and Recommendations 17. Given your experience with TEA, would you be willing to contribute financially for job training for new employees? (select one) a. Yes – it was less costly than training people on-the-job or having people who don’t perform b. Yes – depending on how much the cost is c. Yes - if they would train to my specific requirements d. No – my business cannot afford to pay e. No – there was not enough benefit to my business 18. Given your experience with TEA, do you have any recommendations for how it could be improved? (text) 118 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV FOCUS GROUP DISCUSSION INTERVIEW GUIDE Note: standardized demographic information will be collected from each participant 1. Given that there are a number of job training programs available, what prompted you to participate in this program? How did you hear about it? 2. Are there barriers to your participation that this program overcame (Probe: transportation, participation requirements, cultural) that other employment training programs may not? 3. During the training did you face any challenges? How did you overcome those challenges? Did you get any support? By whom? Please elaborate how. 4. What was your expectation of the program, how expectations were set, were expectations met? 5. Was the training program relevant to your needs? Like materials, skill level, trainer knowledge, trainer sensitivity, easy-to-understand materials, practical training, duration of training? (Please elaborate how with examples.) 6. What was the most satisfying thing about the training? Why? 7. Was the training relevant to what employers seem to want? (Probe: skill trained in, skill level) 8. Talk about how you got placed in a job (e.g., IP had the employer lined-up, went to a placement center, etc.) 9. How do you feel about the process that you went through to get employed? 10. Are you working in the sector in which you were trained? With the same employer? What factors contributed to your remaining with or leaving the original employer? What is your intention for staying with this employer, or within the sector, and why? 11. For those who are looking for a job or who are employed in a different sector than their training, why? 12. Discuss how the employment met expectations, what their expectations were and how expectations were set? 13. Would you recommend this program to others? Why? 14. Do you have any recommendations for how the program could better meet your needs (or others like you)? 119 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV ANNEX 6: EVALUATION TEAM MEMBERS Henrik Schou Hansen (Team Leader) Henrik is a senior Development Practitioner with 25 years of Strategic Planning, Research, Monitoring, Evaluation and Learning experience in in Nepal, Yemen, Thailand, Mexico, India, Turkey and Jordan, having designed, implemented and assessed a multitude of donor-funded projects. Led end of project evaluations for USAID, ILO, UNICEF, UN Women, managing and collaborating with multi-national team members, in leadership roles within evaluations and multi-lateral institutions. Thematic expertise in livelihoods creation and employability among vulnerable populations, local governance, HIV response, and institutional capacity building. Specialized in supporting multilateral institutions such as the UN develop learning products serving strategic aims. Holds Master’s Degree in Public Administration from Aarhus University in Denmark, and Coursework in Evaluating Social Programs from MIT-x in Boston, MA. Fluent Danish, English, Proficient Dutch Comprehension, Basic Spanish. Sari Hweitat (Private Sector SME): Sari is an entrepreneur and business climate-enabling professional with direct experience in product management, innovation, and market trends in Jordan. He regularly works with both employers and job seekers to position and develop technology and trade-focused enterprises within the Jordanian economy, and to create and promote demand for technology-focused careers and employers. Sari has launched and managed several successful startups in Jordan including Aoun, which was named one of Forbes’ “15 Most Promising Startups in the Middle East”. His expertise in the expansion of the private sector in Jordan is regularly sought after, and he has served as a consultant and SME on several development projects in Jordan. Farah Mahafzah (Researcher & TVET Specialist): Farah is an M&E specialist with over 7 years of experience in designing and leading evaluations for multiple organizations in Jordan, such as WFP, USAID-Gender Program, Mercy Corps- Economic Growth pillar, Terre des Hommes, and World University Services of Canada (WUSC). She is an MBA student at American International Theism University with Program Management Development (PMD) certification and has a variety of experience in humanitarian and development projects. During her tenure, she led several assessments including (i) Longitudinal assessments for TVET training and employment interventions; (ii) Labor market assessments; (iii) Rapid assessment on employment situation within the health sector for elderly care givers and administers; and (iv) Situational analysis for National M&E capacities and practices; (v) Digital landscape mapping for Iraq and Jordan; and (vi) Nurseries impact on access to employment for women. Leen Refai (Researcher & Evaluation Specialist): Leen is a MEL Specialist with 4 years’ experience in supporting social cohesion and marginalized youth and gender programming across the MENA region in Jordan, Lebanon, Syria and Iraq. As a MEL Specialist, Ms. Refai has served the OTI/USAID Syria Regional Program to design cluster level evaluations informed by field level data collection, and the UK FCO to develop a systematic monitoring plan to support program performance and track media impact on stabilization. She has trained staff in survey administration, and has developed social cohesion tools and ostracization measures to assess youth resilience and factors contributing to women’s empowerment. She is proficient in use of STAT, QGIS, SPSS, MAXQDA, Kobo. She is currently designing a M&E Framework for Orange Jordan for an EU-funded program which is focused on private sector development & digital & technical skills training programs for women and youth in Jordan. In addition, she is working on an evaluation of gender-based violence (GBV) programming with AECID. She holds a master’s degree in Development Studies from the London School of Economics. Ayoub Al Nmour (Statistical Research Specialist) Ayoub is a research specialist and projects manager, with a particular focus on methodology design and quality assurance for development interventions. He has a diverse projects portfolio with 9 years of hands-on experience across the MENA region. He holds two MA degrees, one in European Studies and International Politics from the Free University of Berlin and one in Human Rights and Human Development from the University of 120 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV Jordan. Ayoub’s software skills include: MS Office (word/Excel/PowerPoint), MS PowerBi, Adobe Creative Suit, Esri ArcGIS, SPSS and STATA. Zeid Qiblawi (Researcher): Zeid is an experienced researcher with a focus on data analytics. His portfolio of work includes data quality assurance and analysis on large scale data collection activities revolving around employment, COVID and social cohesion. Illustrative assignments include EBRD￾funded research on impact of COVID, Mercy Corps Local Community Development Evaluation, and USAID Training for Employment Activity Evaluation. His focus is on data analysis and quality assurance, and utilizes Alchemer, PowerBi, SPSS and STATA. He holds an BA in Sociology from the American University of Beirut; and is a Native Arabic and Native English speaker. 121 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV ANNEX 7: EVALUATION TEAM CONFLICT OF INTEREST DISCLOSURES Appendix III – Non-Disclosure Agreement Template I acknowledge that I have been given access to USAID sensitive information to facilitate the performance of duties assigned to me for compensation. I understand it is my responsibility to safeguard sensitive information disclosed to me, and to refrain from disclosure of sensitive information to persons not requiring access for the performance of official duties. Under the terms of this Agreement, sensitive information is defined to include: ● Any information designated as sensitive but unclassified (SBU), including oral communications, ● Any category of information (including, but not limited to, medical, personnel, financial, investigatory, visa, or law enforcement) which, if released, could result in harm or unfair treatment to any individual or group, or could have a negative impact upon individual privacy, federal programs or foreign relations ● Any information whose improper use or disclosure could adversely affect the ability of the Agency to accomplish its mission ● Any information that is considered proprietary ● Information received through privileged sources ● Non-public information that has not been made available to the general public, such as internal documents or other internal communications ● Source selection information related to any Agency procurement action. I acknowledge receipt of a copy of the Agency Notice of August 15, 2005 and agree to comply fully with the guidelines for non-disclosure of information contained therein, on the same basis as USAID employees. I understand that any breach of this Agreement may result in the termination of my access to sensitive information, which, if such termination effectively negates my ability to perform my assigned duties, may lead to action adverse to me, up to and including termination of my employment on the Task Order. In addition, I understand that the misuse of information requiring protection under the Privacy Act may constitute a violation, or violations, of U.S. criminal law, and that I may be subject to a fine up to USD 5,000.00. I understand that all sensitive information to which I have access or may obtain access is now, and will remain, the property of, or under the control of, the US Government. The conditions and obligations imposed on me by this Agreement will remain in effect during the time I am granted access to sensitive information and at all times thereafter. Signed: Date: 07/15/2021 Henrik Hansen I acknowledge that I have been given access to USAID sensitive information to facilitate the performance of duties assigned to me for compensation. I understand it is my responsibility to 122 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV safeguard sensitive information disclosed to me, and to refrain from disclosure of sensitive information to persons not requiring access for the performance of official duties. Under the terms of this Agreement, sensitive information is defined to include: ● Any information designated as sensitive but unclassified (SBU), including oral communications, ● Any category of information (including, but not limited to, medical, personnel, financial, investigatory, visa, or law enforcement) which, if released, could result in harm or unfair treatment to any individual or group, or could have a negative impact upon individual privacy, federal programs or foreign relations ● Any information whose improper use or disclosure could adversely affect the ability of the Agency to accomplish its mission ● Any information that is considered proprietary ● Information received through privileged sources ● Non-public information that has not been made available to the general public, such as internal documents or other internal communications ● Source selection information related to any Agency procurement action I acknowledge receipt of a copy of the Agency Notice of August 15, 2005 and agree to comply fully with the guidelines for non-disclosure of information contained therein, on the same basis as USAID employees. I understand that any breach of this Agreement may result in the termination of my access to sensitive information, which, if such termination effectively negates my ability to perform my assigned duties, may lead to action adverse to me, up to and including termination of my employment on the Task Order. In addition, I understand that the misuse of information requiring protection under the Privacy Act may constitute a violation, or violations, of U.S. criminal law, and that I may be subject to a fine up to USD 5,000.00. I understand that all sensitive information to which I have access or may obtain access is now, and will remain, the property of, or under the control of, the US Government. The conditions and obligations imposed on me by this Agreement will remain in effect during the time I am granted access to sensitive information and at all times thereafter. Signed: Date: 06/06/2021 Name: Ayoub Nmour 123 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV I acknowledge that I have been given access to USAID sensitive information to facilitate the performance of duties assigned to me for compensation. I understand it is my responsibility to safeguard sensitive information disclosed to me, and to refrain from disclosure of sensitive information to persons not requiring access for the performance of official duties. Under the terms of this Agreement, sensitive information is defined to include: ● Any information designated as sensitive but unclassified (SBU), including oral communications, ● Any category of information (including, but not limited to, medical, personnel, financial, investigatory, visa, or law enforcement) which, if released, could result in harm or unfair treatment to any individual or group, or could have a negative impact upon individual privacy, federal programs or foreign relations ● Any information whose improper use or disclosure could adversely affect the ability of the Agency to accomplish its mission ● Any information that is considered proprietary ● Information received through privileged sources ● Non-public information that has not been made available to the general public, such as internal documents or other internal communications ● Source selection information related to any Agency procurement action I acknowledge receipt of a copy of the Agency Notice of August 15, 2005 and agree to comply fully with the guidelines for non-disclosure of information contained therein, on the same basis as USAID employees. I understand that any breach of this Agreement may result in the termination of my access to sensitive information, which, if such termination effectively negates my ability to perform my assigned duties, may lead to action adverse to me, up to and including termination of my employment on the Task Order. In addition, I understand that the misuse of information requiring protection under the Privacy Act may constitute a violation, or violations, of U.S. criminal law, and that I may be subject to a fine up to USD 5,000.00. I understand that all sensitive information to which I have access or may obtain access is now, and will remain, the property of, or under the control of, the US Government. The conditions and obligations imposed on me by this Agreement will remain in effect during the time I am granted access to sensitive information and at all times thereafter. Name: Sari Hweitat Date: 06/06/2021 124 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV I acknowledge that I have been given access to USAID sensitive information to facilitate the performance of duties assigned to me for compensation. I understand it is my responsibility to safeguard sensitive information disclosed to me, and to refrain from disclosure of sensitive information to persons not requiring access for the performance of official duties. Under the terms of this Agreement, sensitive information is defined to include: ● Any information designated as sensitive but unclassified (SBU), including oral communications, ● Any category of information (including, but not limited to, medical, personnel, financial, investigatory, visa, or law enforcement) which, if released, could result in harm or unfair treatment to any individual or group, or could have a negative impact upon individual privacy, federal programs or foreign relations ● Any information whose improper use or disclosure could adversely affect the ability of the Agency to accomplish its mission ● Any information that is considered proprietary ● Information received through privileged sources ● Non-public information that has not been made available to the general public, such as internal documents or other internal communications ● Source selection information related to any Agency procurement action I acknowledge receipt of a copy of the Agency Notice of August 15, 2005 and agree to comply fully with the guidelines for non-disclosure of information contained therein, on the same basis as USAID employees. I understand that any breach of this Agreement may result in the termination of my access to sensitive information, which, if such termination effectively negates my ability to perform my assigned duties, may lead to action adverse to me, up to and including termination of my employment on the Task Order. In addition, I understand that the misuse of information requiring protection under the Privacy Act may constitute a violation, or violations, of U.S. criminal law, and that I may be subject to a fine up to USD 5,000.00. I understand that all sensitive information to which I have access or may obtain access is now, and will remain, the property of, or under the control of, the US Government. The conditions and obligations imposed on me by this Agreement will remain in effect during the time I am granted access to sensitive information and at all times thereafter. Name: Farah Mahafzah Date: 06/02/2021 125 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV I acknowledge that I have been given access to USAID sensitive information to facilitate the performance of duties assigned to me for compensation. I understand it is my responsibility to safeguard sensitive information disclosed to me, and to refrain from disclosure of sensitive information to persons not requiring access for the performance of official duties. Under the terms of this Agreement, sensitive information is defined to include: ● Any information designated as sensitive but unclassified (SBU), including oral communications, ● Any category of information (including, but not limited to, medical, personnel, financial, investigatory, visa, or law enforcement) which, if released, could result in harm or unfair treatment to any individual or group, or could have a negative impact upon individual privacy, federal programs or foreign relations ● Any information whose improper use or disclosure could adversely affect the ability of the Agency to accomplish its mission ● Any information that is considered proprietary ● Information received through privileged sources ● Non-public information that has not been made available to the general public, such as internal documents or other internal communications ● Source selection information related to any Agency procurement action. I acknowledge receipt of a copy of the Agency Notice of August 15, 2005 and agree to comply fully with the guidelines for non-disclosure of information contained therein, on the same basis as USAID employees. I understand that any breach of this Agreement may result in the termination of my access to sensitive information, which, if such termination effectively negates my ability to perform my assigned duties, may lead to action adverse to me, up to and including termination of my employment on the Task Order. In addition, I understand that the misuse of information requiring protection under the Privacy Act may constitute a violation, or violations, of U.S. criminal law, and that I may be subject to a fine up to USD 5,000.00. I understand that all sensitive information to which I have access or may obtain access is now, and will remain, the property of, or under the control of, the US Government. The conditions and obligations imposed on me by this Agreement will remain in effect. Name: Leen Refai Date: 126 | TRAINING FOR EMPLOYMENT END OF PROJECT EVALUATION REPORT USAID.GOV I acknowledge that I have been given access to USAID sensitive information to facilitate the performance of duties assigned to me for compensation. I understand it is my responsibility to safeguard sensitive information disclosed to me, and to refrain from disclosure of sensitive information to persons not requiring access for the performance of official duties. Under the terms of this Agreement, sensitive information is defined to include: ● Any information designated as sensitive but unclassified (SBU), including oral communications, ● Any category of information (including, but not limited to, medical, personnel, financial, investigatory, visa, or law enforcement) which, if released, could result in harm or unfair treatment to any individual or group, or could have a negative impact upon individual privacy, federal programs or foreign relations ● Any information whose improper use or disclosure could adversely affect the ability of the Agency to accomplish its mission ● Any information that is considered proprietary ● Information received through privileged sources ● Non-public information that has not been made available to the general public, such as internal documents or other internal communications ● Source selection information related to any Agency procurement action. I acknowledge receipt of a copy of the Agency Notice of August 15, 2005 and agree to comply fully with the guidelines for non-disclosure of information contained therein, on the same basis as USAID employees. I understand that any breach of this Agreement may result in the termination of my access to sensitive information, which, if such termination effectively negates my ability to perform my assigned duties, may lead to action adverse to me, up to and including termination of my employment on the Task Order. In addition, I understand that the misuse of information requiring protection under the Privacy Act may constitute a violation, or violations, of U.S. criminal law, and that I may be subject to a fine up to USD 5,000.00. I understand that all sensitive information to which I have access or may obtain access is now, and will remain, the property of, or under the control of, the US Government. The conditions and obligations imposed on me by this Agreement will