KARACHI YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION REPORT (AMANTECH) APRIL 28, 2017 This publication was produced for review by the United States Agency for International Development. It was prepared by Jeffrey Tines, Noman ul Haq Siddiqui, Nadeem Akhtar, Muhammad Sadiq, Tehmina Tanveer, and Syed Zohaib Iqbal Zaidi, Management Systems International, A Tetra Tech Company. KARACHI YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION REPORT (AMANTECH) Contracted under Order No. AID-391-C-15-00004 Performance Management Support Contract DISCLAIMER This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents are the sole responsibility of Management Systems International and do not necessarily reflect the views of USAID or the United States Government. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) iii CONTENTS Acronyms............................................................................................................................v Project Summary .............................................................................................................. 1 Executive Summary.......................................................................................................... 3 Evaluation Purpose and Questions.................................................................................................. 3 Project Background............................................................................................................................. 3 Key Findings and Conclusions.......................................................................................................... 3 Summary Recommendations............................................................................................................. 4 Evaluation Purpose and Questions.................................................................................. 5 Evaluation Questions.......................................................................................................................... 5 Project Background .......................................................................................................... 5 Theory of Change................................................................................................................................ 6 Evaluation Methods and Limitations............................................................................... 6 Data Collection.................................................................................................................................... 6 Evaluation Limitations and Mitigation Measures........................................................................... 7 Findings .............................................................................................................................. 8 Q1: How effective are the project’s targeting, recruiting, and selection strategies? ............................................................................................................................... 8 Q2: How is the program affecting individual trainees?.............................................................14 Q3: How, if at all, has the perception of insecurity changed in targeted towns?...............28 Q4: How are stakeholders involved in the project? .................................................................32 Q5: Is the project scalable and can local resources be leveraged to transition it out of donor dependence? ...............................................................................................34 Conclusions ...................................................................................................................... 36 Recommendations .......................................................................................................... 37 Annexes............................................................................................................................ 38 Annex 1: Evaluation Scope of Work.............................................................................................38 Annex 2: Assignment Work Plan...................................................................................................44 Annex 3: Data Collection Instruments.........................................................................................75 Annex 4: Sampling ...........................................................................................................................165 Annex 5: Composite Marginalization Index ..............................................................................167 Annex 6: Tests of Equality of Distributions ..............................................................................172 Annex 7: Regression Results.........................................................................................................173 Annex 8: Effect of Duration on Labor Force Participation after Training .........................175 Annex 9: Disaggregates by Training Topic ................................................................................176 Annex 10: Conflict of Interest Disclosures...............................................................................178 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) iv LISTS OF TABLES AND FIGURES List of Tables Table 1: Project Summary................................................................................................................................................ 1 Table 2: Interviews Planned and Conducted............................................................................................................... 7 Table 3: Distribution of Trainess by Risk Ranking..................................................................................................... 9 Table 4: Socioeconomic Characteristics of Trainees..............................................................................................12 Table 5: Cost-Effectiveness of Recruting Strategies................................................................................................13 Table 6: Transition of Alumni Between States of Employment............................................................................16 Table 7: Job Retention Rates (Percent) by Industry and Alumni Characteristics............................................25 Table 8: Alumni Who Would Pay for Similar Training in the Future .................................................................34 Table 9: Surveys Required, Planned, and Completed ...........................................................................................165 Table 10: Profile of Survey Respondents.................................................................................................................166 Table 11: Dimensions, Indicators, Definitions, and Weights for the CMI .......................................................168 Table 12: CMI Calculation for All 18 Towns..........................................................................................................170 Table 13: Regression Analysis of Effects of Distance on Recruiting..................................................................173 Table 14: Logit Regression Results for Job Satisfaction........................................................................................173 Table 15: Regression Results for Monthly Income................................................................................................174 Table 17: Number of Observations by Training Topic.........................................................................................176 Table 18: Before and After Training Median Income by Training Topic..........................................................177 List of Figures Figure 1: Amantech Facility and Target Towns of Karachi...................................................................................... 2 Figure 2: Distribution of Trainees by Town – Amantech Risk Analysis.............................................................10 Figure 3: Distribution of Trainees by Town – Marginalization Index..................................................................11 Figure 4: Employment Status Before and After Training........................................................................................16 Figure 5: Change in Labor Force Participation and Employment by Sex ...........................................................17 Figure 6: Average Weekly Working Hours Before and After Training..............................................................19 Figure 7: Median Personal Monthly Income Before and After Training .............................................................20 Figure 8: Job Satisfaction of Employed Alumni .........................................................................................................21 Figure 9: Determinants of Job Satisfaction ................................................................................................................22 Figure 10: Job Placement Support after training.......................................................................................................23 Figure 11: How Training Contributes to Reduced Crime and Violence............................................................27 Figure 12: Survey Respondents’ Contributions to Household Income, Household Affairs, and Neighborhood Wellbeing ............................................................................................................................27 Figure 13: Perceived Level of Crime and Violence..................................................................................................28 Figure 14: Impact of Crime and Violence on Households.....................................................................................29 Figure 15: Perceived Impact of Religious Groups....................................................................................................29 Figure 16: Perceived Impact of Political Parties........................................................................................................30 Figure 17: Perceptions of Income-Generating Opportunities ..............................................................................30 Figure 18: Perceptions of Poverty Level ....................................................................................................................31 Figure 19: Residents’ Satisfaction with Government Services..............................................................................31 Figure 20: Marginalization Index, Location of Training Venue, and Percentage of Applicants by Town ..171 Figure 21: Test Results for Equality of Distributions............................................................................................172 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) v ACRONYMS CAD-CAM Computer-Aided Design and Computer-Aided Manufacturing CAD-CIVIL Computer-Aided Design for Civil Engineering CMI Composite Marginalization Index DO Development Objective MSI Management Systems International Rs. Pakistani Rupees SBTE Sindh Board of Technical Education SITE Sindh Industrial Trading Estate SMS Short Message Service STEVTA Sindh Technical Education and Vocational Training Authority UNDP United Nations Development Program USAID United States Agency for International Development Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 1 PROJECT SUMMARY Table 1 summarizes basic information about the Karachi Youth Employment Project. TABLE 1: PROJECT SUMMARY Title/Field Project/Activity Information Contract/agreement numbers AID-391-A-15-00005 Contracting/agreement officer’s representative (COR/AOR) - Start date January 1, 2015 Completion date December 31, 2017 Location Karachi Implementing partner(s) Aman Institute for Vocational Training – Amantech USAID/Pakistan Mission Strategic Framework objectives addressed Development Objective 3: Increased stability in focus areas Development Objective 2: Improved economic status of focus populations Budget - Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 2 FIGURE 1: AMANTECH FACILITY AND TARGET TOWNS OF KARACHI Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 3 EXECUTIVE SUMMARY Evaluation Purpose and Questions The midterm evaluation of the Amantech-implemented Karachi Youth Employment Project aims to assess whether the project is on track to meet training and employment targets, examine the efficacy of recruitment strategies, gauge stakeholder involvement, and assess the project’s potential for scalability and sustainability. USAID/Pakistan expects to use the evaluation results to guide future stabilization programming, including the design of the second part of Let Girls Learn, a USAID-funded program that seeks opportunities to foster public-private partnerships to advance girls’ education. The evaluation will also help identify potential course corrections before the project ends in January 2018. The evaluation addresses five questions: 1. How effective are the program’s targeting, recruiting, and selection strategies? 2. How is the program affecting individual trainees? 3. How, if at all, has the perception of insecurity changed in targeted neighborhoods? 4. What is the stakeholder involvement in the projects? 5. Is the project scalable and can local resources be leveraged to transition it out of donor dependence? Project Background The project focuses on improving the quality of skilled labor available in Karachi, with a focus on enhancing employment prospects for youth (aged 18–29 years) in marginalized neighborhoods identified as priorities by USAID. The project’s theory of change hypothesizes that training marginalized/at-risk youth from Karachi’s violence-prone areas will improve their employment prospects, reduce violent extremism, and contribute to establishing a foundation for long-term economic growth and prosperity. Trained individuals are expected to become agents of change in their society and contribute to reducing aggregate levels of violence and extremism in the city. The project focuses on vocational training to build the capacities of youth to engage in the job market in strategically selected industries that are important in the project areas. Amantech implements the training in its facility in Korangi Town. Key Findings and Conclusions Amantech did not employ a community-based recruiting strategy that targeted specific towns. Consequently, it recruited only about half of USAID-sponsored students from the most marginalized/at￾risk towns.1 Amantech has only one training facility, located in Korangi. The time and cost associated with travel between Amantech’s training facility in Korangi and beneficiaries’ homes is one of the most important determinants of individuals’—particularly women’s—choices about whether and where to 1 Amantech’s cooperative agreement and work plans did not specify the towns from which it would recruit. They did, however, suggest that it target the towns that its risk analysis identified as the most at-risk. It succeeded in recruiting about half of its students from towns the analysis identified as high risk, and the remainder from towns with medium to low risk. It also recruited about half of its students from the towns the evaluation team’s marginalization index classified as the most marginalized. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 4 engage in training. Having a single training facility therefore limits Amantech’s geographic reach within Karachi, and Amantech may face challenges in substantially engaging students from communities far from Korangi. Training generated more and better jobs measured by a number of indicators. It increased labor force participation and employment rates, facilitated a shift from self-employment towards more stable wage employment for women, and increased median incomes. Technical and technology-focused trainings (e.g., computer-aided design for civil engineering [CAD-CIVIL], electrical and electronics, general electric, vehicle body technician, and stitching machinist) generated larger increases in employment than more traditional and manual skills training and generated higher than average percentage increases in average monthly income. Two of the trainings with the highest alumni employment rates (CAD-CIVIL and electrical and electronics) reported the lowest rates of matching occupations. Together these findings demonstrate the importance of identifying and training for promising employment opportunities and then placing individuals in these occupations, keeping in mind that some skills may transfer to occupations other than those on which the training focuses. Amantech appears to implement its training approach well; it engages with potential employers to identify labor market demands and reportedly prepares trainees relatively well to meet these demands. Although stakeholders see a need to expand vocational training projects to engage more marginalized/at-risk youth, Amantech’s nontargeted recruiting strategy, single training venue, and fee structure limit its potential to scale up the approach, target it to specific populations, and wean it from donor support. Amantech may be able to scale up recruiting from targeted towns at the margins by increasing its engagement with communities to identify local labor market needs and target potential trainees, but this will require deeper engagement with communities and a community-focused recruiting strategy. Summary Recommendations  Because it has only the single training facility in Korangi, Amantech will find it difficult to effectively target residents of neighborhoods throughout greater Karachi for training. If Amantech desires to target specific populations, it could consider, and perhaps pilot, different strategies to recruit and train youth from the most marginalized/at-risk communities. Options include: (1) implementing community-based recruiting strategies, (2) engaging partners with training facilities in the targeted towns, (3) finding some way (e.g., using government or donor support) to compensate individuals for the time and cost of travel to Amantech’s training facility, (4) establishing or supporting satellite vocational training centers in coordination with existing local vocational training centers, and (5) setting up mobile skills development/vocational training units to provide similar vocational training that is certified by the Trade Testing Board in local communities.  To meet employment and income targets, Amantech could work more closely with the private sector to identify required hard and soft skills, train in the practical application of these skills, and provide more consistent and targeted career counseling and job placement services. It might also identify, and focus on, the fields with the greatest potential for employment. For example, the evaluation identifies several fields for which trained labor seems to be in high demand. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 5 EVALUATION PURPOSE AND QUESTIONS The midterm evaluation of the Karachi Youth Workforce Development Project being implemented by Amantech aims to assess whether the project is on track to meet training and employment targets, examine the efficacy of recruitment strategies, gauge stakeholder involvement, and assess the project’s potential for scalability and sustainability. USAID/Pakistan expects to use the evaluation results to guide future stabilization programming, including the design of the second part of Let Girls Learn, a USAID￾funded program that seeks opportunities to foster public-private partnerships to advance girls’ education. The evaluation will also help identify potential course corrections before the project ends in January 2018. Evaluation Questions The evaluation focused on answering the following five questions identified in the scope of work (Annex 1) and assignment work plan (Annex 2): 1. How effective are the program’s targeting, recruiting, and selection strategies? 2. How is the program affecting individual trainees? 3. How, if at all, has the perception of insecurity changed in targeted neighborhoods? 4. What is the stakeholder involvement in the projects? 5. Is the project scalable and can local resources be leveraged to transition it out of donor dependence? PROJECT BACKGROUND Amantech is a subsidiary of the Aman Foundation, a nonprofit that works on a range of development issues. Amantech is the foundation’s flagship vocational training institute and offers a wide variety of vocational training. USAID partnered with Amantech in January 2015 to launch the Karachi Youth Workforce Development Project. The project focuses on training youth and aims to provide 3,600 person years of training over the three￾year life of the project; e.g., a trainee completing a six-month course counts as half a person year towards the target. Amantech established a target of placing 70 percent of alumni. Upon completing their coursework, trainees sit for either the City and Guilds or the Sindh Board of Technical Education accreditation examination. Only trainees who pass an external examination contribute to the training targets. Amantech offers training on a variety of subjects. Trainings vary in duration from short 3-month courses to 18-month trainings. The project has shifted toward shorter courses over time, with most current trainees being placed in six-month trainings. In addition to technical skills, Amantech emphasizes soft skills, such as personal grooming, office etiquette, and language skills. Amantech’s single training facility in Korangi is located near the three towns USAID prioritized in the cooperative agreement, Korangi, Lyari, and Sultanabad, a section of Kiamari Town. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 6 Theory of Change The project’s theory of change hypothesizes that training marginalized/at-risk youth from Karachi’s violence-prone areas will improve their employment prospects, reduce violent extremism, and contribute to establishing a foundation for long-term economic growth and prosperity. Trained individuals are expected to become agents of change in their society and contribute to reduced aggregate levels of violence and extremism in the city. EVALUATION METHODS AND LIMITATIONS The evaluation applies a quasi-experimental design aimed at measuring project outcomes by incorporating both a before and after comparison of alumni and a comparison of the pretraining status of current trainees to the posttraining status of the alumni. The pretraining status of current trainees serves as a counterfactual (i.e., represents what the current status of alumni would have been if they had not participated in training) to compare with the posttraining status of alumni as an alternative method of measuring impact. The ability to draw two comparisons allowed the evaluation team to triangulate changes in the outcome variables and enhanced the reliability of the findings. Data Collection The evaluation used mixed methods to answer the evaluation questions. Primary data collection methods included surveys, semistructured interviews, and group interviews. (See Annex 3 for the data collection instruments.) The evaluation conducted a structured survey of a random sample of alumni and current trainees and administered group interviews to a subset of survey respondents to obtain more detailed information and answer “how” and “why” questions. Finally, the evaluation used key informant interviews with a purposive sample of respondents from government, industry, and Amantech staff to learn their perspectives on the project. The evaluation also analyzed data from a variety of secondary sources. Amantech provided demographic information on trainees (both current and alumni), which served as the sampling frame for the survey and group interviews. To assess the prevalence of crime in Karachi’s towns, the evaluation team compiled and mapped crime reports for Karachi from the Express Tribune newspaper. To provide context and comparisons where applicable, the team also analyzed datasets from the Pakistan Bureau of Statistics. A literature review of the available discourse on the subject provided background information for triangulation. Survey Amantech provided demographic data on trainees, including their town of residence, which served as the sampling frame for the survey. The evaluation team selected a simple random sample of alumni and current trainees, stratified by sex and town of residence, from the lists provided by Amantech. The evaluation team calculated sample sizes that allowed them to detect a 10 percentage point change in the proportion of trainees employed as a result of the training, with an alpha of 0.05. Table 2 indicates the number of planned versus completed surveys. Annex 4 describes the sampling approach in more detail. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 7 TABLE 2: INTERVIEWS PLANNED AND CONDUCTED Respondent Type Number of Surveys Planned Number of Surveys Conducted Current trainees 490 491 Alumni 410 278 Total 900 769 Because of difficulties in obtaining government permissions for fieldwork, the team conducted the interviews at the Amantech training facility. To invite trainees to the interviews, surveyors attempted to contact them at least three times a day for two consecutive days; if they had no response after two days, they marked the case as nonresponse and replaced it. Twenty-five percent of sampled individuals had their mobile phones switched off, and another 11 percent did not answer repeated phone calls, even after they had been sent a Short Message Service (SMS) introducing the survey purpose. Fourteen percent were unable to travel to training institutes for the survey or were unavailable or unreachable for other reasons. Well under 1 percent of sampled individuals refused to take part in the survey. Overall, 50 percent of sampled individuals were available for the survey. Group Interviews The evaluation team conducted group interviews with a subset of survey respondents to acquire more detailed qualitative information and substantiate findings from the survey. The team conducted eight group interviews stratified by sex, training status, and employment status, with up to four participants in each group interview. The team purposively selected participants from the available pool of survey respondents by prioritizing respondents from areas that USAID categorized as marginalized, namely Lyari, Korangi, and Sultanabad. Key Informant Interviews The team conducted 32 key informant interviews with a variety of stakeholders, including representatives of government line departments (3), Amantech staff (3), employers of alumni (5), and experts on Karachi city (5). Secondary Review In addition to collecting primary data using the methods outlined above, the evaluation team studied Amantech’s reports in detail. The team also reviewed relevant academic journal articles and policy documents, and analyzed relevant government statistics on the labor force and employment to draw comparisons. Evaluation Limitations and Mitigation Measures The evaluation team made extensive efforts to ensure the data collected were valid and reliable. This section presents the limitations arising as a result of the evaluation design and field procedures. It also explains the mitigation measures the team took to minimize the impact of these limitations on the reliability and validity of the data collected.  Because of anticipated difficulties in obtaining the necessary government permissions to conduct fieldwork in Karachi, the evaluation team was not able to visit trainees at their homes. The team considered a telephone survey; however, they deemed that option infeasible in view of the Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 8 length of the questionnaire. Instead, they conducted the surveys of trainees at the Amantech training facility. The location of the survey may have caused social desirability bias, with respondents underreporting bad behavior and overreporting good behavior. Data collection instruments were designed to avoid direct personal questions to the extent possible, but some residual potential for social desirability bias remains.  To reduce nonresponse rates, trainees were given Pakistani Rupees (Rs.) 500 for taking part in the survey. Given the low response rate, this amount was perhaps lower than optimal. Although it could be argued that monetary incentives bias responses, Toepoel2 found that they do not bias responses and may instead contribute to more detailed information.  The potential for low response rates became apparent during field testing of the survey instrument. To mitigate this potential issue, the evaluation team provided replacements in addition to the original 900 sampled trainees, stratified by sex and town of residence. Despite this mitigation measure, the response rate was low. The team completed 769 of the planned surveys. The low response rate raised the potential for bias since trainees who came for the survey may be systematically different from those who did not.  The survey asked alumni a set of recall questions about their pretraining status. In some cases, the recall period was longer than one year, so the responses have the potential for some degree of unreliability. To mitigate this, the team limited recall questions to topics that trainees could reasonably be expected to remember. For example, the survey included questions on employment status and personal monthly incomes but not on feelings of insecurity before the training. Another measure to mitigate the effect of recall bias was to train enumerators to encourage respondents to think carefully about the question and use as much time as they needed before answering recall questions. In addition, comparing the pretraining status of current trainees to the pretraining status of alumni validated the reliability of responses to recall questions.  The survey sample was too small to produce statistically significant results for disaggregates by type of training, industry of employment, or sex. A sample sufficient to accommodate analysis at these levels would have been prohibitively expensive and time-consuming. Furthermore, given the difficulty the evaluation team had recruiting survey respondents, it may not have been possible to recruit a sample of sufficient size. FINDINGS Q1: How effective are the project’s targeting, recruiting, and selection strategies? Q1(a): Is Amantech selecting beneficiaries from marginalized/at-risk neighborhoods? The analysis addressed this question by first examining whether the distribution of applicants by town is different from the distribution of individuals selected for USAID-supported training. It then examined 2 Toepoel, Vera. 2012. “Effects of Incentives in Surveys.” Handbook of Survey Methodology for the Social Sciences: 209-223. http://link.springer.com/chapter/10.1007%2F978-1-4614-3876-2_13. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 9 whether beneficiaries (alumni and current trainees selected for USAID-supported training programs) were more likely to come from marginalized/at-risk towns than from other towns.3 A comparison of the distribution of applicants by town to the distribution of individuals selected for USAID-supported training found no significant differences, which suggests that Amantech did not apply specific geographical criteria to select trainees from among the pool of applicants. To determine whether trainees came from marginalized/at-risk towns, the analysis examined the percentage of trainees who came from the most marginalized/at-risk towns using two measures of marginalization/risk: (1) Amantech’s risk rating for towns in Karachi4 and (2) a marginalization index developed by the evaluation team (see Annex 5). Amantech’s records indicate that it has trained 4,461 individuals under the USAID-funded project, virtually all of whom came from 1 of the 18 towns in Karachi. Twenty percent of trainees (50 percent of women and 19 percent of men) came from Korangi, where the Amantech training facility is located. Amantech recruited about half of the USAID-supported trainees from towns considered high risk (based on Amantech’s risk analysis) or most marginalized/at-risk (based on the evaluation team’s marginalization index). Forty-eight percent (53 percent of women and 47 percent of men) came from the seven towns Amantech identified as high risk based on its risk analysis (Figure 2 and Table 3), and 48 percent (87 percent of women and 45 percent of men) came from the five towns the marginalization index ranked as most marginalized/at-risk, i.e., Lyari, Landhi, Orangi, Korangi, and Jamshed (Figure 3 and Table 3). TABLE 3: DISTRIBUTION OF TRAINESS BY RISK RANKING Town Amantech Risk Rating Marginalization Index Rank % of USAID Trainees Lyari Medium 1 2% Landhi Low 2 11% Orangi High 3 6% Korangi High 4 20% Jamshed Not rated 5 9% Baldia Not rated 6 3% Liaquatabad Not rated 7 4% Bin Qasim Not rated 8 2% North Nazimabad High 9 1% Saddar High 10 3% New Karachi High 11 6% Kiamari Not rated 12 2% Shah Faisal High 13 6% Sindh Industrial Trading Estate (SITE) Not rated 14 2% Gulshan High 15 6% Malir Medium 16 10% Gulberg Not rated 17 3% 3 The analysis focuses on towns instead of neighborhoods because neighborhoods are not well defined, making it difficult to consistently identify the neighborhood in which a trainee lived. 4 Amantech’s analysis, contained in the cooperative agreement, ranked towns from highest to lowest risk in the following order: Shah Faisal, New Karachi, Saddar, Orangi, Korangi, North Nazimabad, Gadap, Malir, Gulistan-e-Juhar, and Landhi. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 10 Town Amantech Risk Rating Marginalization Index Rank % of USAID Trainees Gadap Medium 18 1% Cantonment Not rated 2% Sultanabad Not rated 0% Other in Karachi Not rated 0% Out of Karachi Not rated 0% % from high risk High 48% % from medium risk Medium 13% % from low risk Low 11% % from unrated Not rated 27% Rounding discrepancies may lead to slight differences in sums and comparisons across tables and figures. FIGURE 2: DISTRIBUTION OF TRAINEES BY TOWN – AMANTECH RISK ANALYSIS Note: Percentages are not disaggregated by sex. Thus, the 2 percent for women in Korangi represents 2 percent of all trainees, not 2 percent of women trainees. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 11 FIGURE 3: DISTRIBUTION OF TRAINEES BY TOWN – MARGINALIZATION INDEX Note: Percentages are not disaggregated by sex. Thus, the 2 percent for women in Landhi represents 2 percent of all trainees, not 2 percent of women trainees. Amantech’s reports and the evaluation team’s interviews with Amantech staff suggest that the organization did not specifically use the applicants’ home locations as a criterion for selecting them for training. The fact that there is no significant difference between the distributions of applicants and trainees across towns—for men or women—supports this finding.5 In the absence of specific geographic selection criteria, evidence from various sources (see findings for question 2(a)) suggests that the distance between the training facility and trainees’ homes was an important factor in attracting trainees. A regression analysis of the effect of distance from the Amantech training facility in Korangi on the percentage of trainees recruited from various towns found that distance explained 44 percent of the variation in the percentage of trainees recruited from a town.6 Furthermore, the effect of distance was stronger for female than for male applicants (Annex 7). Conclusions: Amantech recruited about half of the men and more than half of the women trainees from the most marginalized/at-risk towns (based on both the marginalization index and Amantech’s risk analysis). Women came disproportionally from Korangi, the location of the Amantech training facility, confirming findings presented later that the distance and cost of travel between a trainee’s home and the training facility were important factors in determining the towns from which Amantech successfully recruited trainees, especially women (see Table 13). 5 A nonparametric Kolgomorov-Smirnov test found no significant difference between the distribution of applicants (men, women, or combined) across towns and the distribution of trainees (alumni and current trainees). See Annex 6 for statistical tests and distribution of applicants and trainees. 6 The analysis used a crude measure of distance between the geographic centers of the towns. A linear regression of the percentage of trainees (alumni and current applicants) on distance from the center of Korangi (the town in which the Amantech facility is located) and a dummy variable indicating the three USAID priority towns explained 44 percent of the variation in percentage of trainees. The estimated coefficient was negative and significant (𝛼 = 0.01) while the coefficient of the USAID priority dummy variable was positive but not significant (𝛼 = 0.13). Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 12 Q1(b): Are the recruiting strategies attracting marginalized/at-risk individuals and is Amantech selecting marginalized/at-risk individuals from among the pool of recruits? This question differs from question 1(a) because it focuses on individual characteristics instead of characteristics of a town. To answer the question, the evaluation team first had to define “marginalized/at-risk” at the individual level. Because there is no standard definition, the analysis used data from the surveys of alumni and current trainees to develop a profile along seven dimensions related to marginalization, which were measured pretraining: household income, poverty score, likelihood of poverty, literacy, educational attainment, employment rate, and percentage employed. It then matched the characteristics of respondents to those of the population of Karachi to determine whether respondents were more or less “marginalized” than the general population. Because Amantech did not collect data on any of the seven marginalization criteria from applicants, it is not possible to answer the first part of the question, i.e., whether recruiting strategies attracted marginalized/at-risk individuals. However, because Amantech does not appear to have applied any particular criteria, other than passing a test, to select trainees from among the pool of applicants, it is reasonable to assume that the “marginalization” status of applicants is similar to that of alumni and current trainees. To answer the second part of the question, i.e., whether Amantech selected marginalized/at-risk individuals from among the pool of recruits, the analysis compared the characteristics of alumni and current trainees at the time they started training to the values of these characteristics for the target population (i.e., youth 18–29 years of age) in Karachi (Table 4). TABLE 4: SOCIOECONOMIC CHARACTERISTICS OF TRAINEES Characteristic Amantech Karachi Aged 18–29 years (%) 93% 21% Literate (%) 99% 90% In labor force (%) 63% 43% Percentage employed (%) 48% 37% Employment rate—accounting for labor force participation (%) 73% 87% Educational attainment Secondary or lower (%) 62% 64% Higher secondary or above (%) 38% 36% Poverty score (mean) 78.3 73.5 Maximum likelihood of being under the poverty line of $1.25/day (average %) 27% 27% Maximum likelihood of being under the national poverty line (average %) 30% 30% Median monthly household income (Rs.) 30,000 32,000 Household income of Rs. 20,000 or below (%) 26% 52%a a. Estimates of annual household income are from the Household Economic Survey conducted by the Pakistan Bureau of Statistics and represent a sample of only 89 households. Conclusions: Table 4 suggests that Amantech effectively attracted the target age group. Results are mixed on whether Amantech’s alumni and current trainees are more or less marginalized than their age Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 13 cohort in urban Karachi based on the other indicators. They are more likely to be in the labor force, more likely to be literate, and less likely to have a monthly household income below Rs. 20,000. However, they are significantly less likely to be employed, have a higher average poverty score, and have a slightly lower median household income. Q1(c): What is the relative cost-effectiveness of the different recruiting strategies? To answer this question, the evaluation team first used data from the surveys of alumni and current trainees to determine which recruiting strategies attracted them to the training opportunity. It then collected data from Amantech on the costs of each recruiting method, estimated the total number of successful applicants (i.e., accepted into the training program) attracted by each method, and calculated the cost per successful applicant by dividing the cost of the method by the number of successful applicants it attracted. The survey of trainees (alumni and current trainees) asked respondents to indicate all the ways they had heard about the training opportunity. Family and friends (i.e., word-of-mouth) emerged as the most frequently mentioned source, with social media and cable television as distant second and third most frequently mentioned, but still ranking well above most other methods. The data suggest that Amantech employed a broad recruiting strategy that relied on a variety of communication channels, and that social media and television were the most effective direct (i.e., not word-of-mouth), methods.7 Overall, however, most trainees heard of the training opportunity through word-of-mouth. Table 5 ranks the recruiting methods from most to least cost-effective. These estimates suggest that, of the direct methods, the most cost-effective were social media; the refer-a-student campaign; SMS; cable television; newspaper ads; pamphlets, brochures, and flyers; and billboards, in that order. The remaining methods entailed substantially higher average cost per trainee.8 TABLE 5: COST-EFFECTIVENESS OF RECRUTING STRATEGIES Recruiting Method Effectiveness of Method (% of Responses)9 Total Cost of Method (Rs.) Number of Successful Applicants Cost per Successful Applicant (Rs.) Social media 15% 170,000 229 742 Refer-a-student campaign 3% 130,000 34 3,842 SMS 3% 300,000 44 6,780 Cable airing production 16% 1,900,000 216 8,795 Newspaper ads 8% 850,000 96 8,826 Pamphlets, brochures, flyers 6% 800,000 65 12,294 Billboard 5% 1,000,000 55 18,295 Promotions at sports events 2% 100,000 3 38,419 Social mobilization 1% 500,000 10 48,024 7 The importance of word-of-mouth as a source of information about the training opportunity does not detract from the importance of direct methods. In fact, since word-of-mouth is derivative of the direct methods, it has a multiplier effect. It was not possible in this research, however, to determine the direct recruiting methods that contributed most to word-of-mouth. 8 The data did not permit gaining a deeper understanding of the contribution of each of the direct methods to word-of-mouth. The ranking of methods by cost-effectiveness implicitly assumes that they contributed equally to word-of-mouth. Because the question asked which methods alumni and current trainees had heard of, and not which were most influential in their decision to apply, the analysis also assumes that the methods were equally influential in attracting successful applicants. 9 Percentage of survey respondents who mentioned each recruiting method. Respondents could identify more than one method. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 14 Recruiting Method Effectiveness of Method (% of Responses)9 Total Cost of Method (Rs.) Number of Successful Applicants Cost per Successful Applicant (Rs.) Door-to-door activity <1% 800,000 10 76,839 Radio show <1% 500,000 3 192,097 Street theaters <1% 1,800,000 3 691,549 Indirectly through family and friends (word-of-mouth) 85% N/A N/A N/A All methods N/A 8,850,000 768a 11,526 / 4,286b a. The number reflects the team’s estimate of the number of recruits that responded to the campaigns for which Amantech provided cost data. b. Amantech spent Rs. 11,526 per individual recruited directly by a method. Factoring in the contribution of the direct methods to word-of-mouth (family and friends) implies that Amantech spent about Rs. 8,850,000 to recruit 2,065 trainees in the relevant period at an average cost of Rs. 4,286 per trainee. Q2: How is the program affecting individual trainees? Q2(a): To what extent are trainees obtaining new or better jobs relative to similar individuals who do not receive training? The evaluation team assessed various job characteristics to determine whether training is helping trainees get “new or better jobs.” The factors include the employment status, labor force participation rate, employment type, contract length and type, income, working hours, and job satisfaction. The team also analyzed the degree to which jobs are relevant to training received, and whether Amantech has provided adequate support in placing alumni in jobs. Taken together, these analyses show how the program affected the employment of beneficiaries. The analysis also explored the relationship between indicators of better jobs and factors such as training type, sex, prior education, and location to gain further insights into how and for whom training was effective. In response to a general question about how training had affected employment, 53 percent of 278 surveyed alumni felt that the training had a positive impact on their employment, with women being more likely than men to believe their employment had improved (62 percent of 73 women compared to 52 percent of 205 men). Sixty-four percent of 171 employed alumni—and 35 percent of 107 unemployed alumni—believed that the training had a positive impact. One alumna explained how improved job prospects from the training had affected her life: “Earlier I couldn’t support my parents, now I can support them financially. Now that I have done this course, there are better chances of acquiring a good job. My life is better now.” – Female alumna, Landhi Town When asked to identify all the ways in which the project had positively affected employment from a defined list of effects, alumni most frequently mentioned that it increased their confidence in finding a job (33 percent of 146 responses), increased their income (28 percent), and improved their skills (26 percent). Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 15 Employment Status10 Employment status reflects the combined effect of employment, unemployment, and labor force participation. The data show that training increased employment and labor force participation rates but that it also increased unemployment rates among men as previously unemployed alumni entered the labor force. Employment: The data suggest that Amantech training increased the percentage of trainees who were employed—i.e., the percentage of respondents who reported having worked in the past seven days.  A comparison of employment status before and after training found that alumni were 31 percent more likely to be employed after training than before. Sixty-three percent of 278 alumni reported being employed in the seven days prior to the survey, compared to 48 percent prior to entering the program. The difference is significant at a p<.05 level.11  To rule out improved overall employment prospects as an explanation for the observed effect of training, the analysis compared the employment status of alumni to that of current trainees who had just entered the program. Forty-three percent of 491 current trainees reported being employed before the training (Figure 4), which suggests that the overall employment situation had not improved and that training contributed to higher employment rates among alumni.12  The analysis also used a regression model to explore whether individuals who were employed prior to training were more likely to obtain jobs after training. The logit model estimated alumni’s posttraining employment status as a function of pretraining employment status, education level, sex, age, marital status, town, and time (months) since completing the training. The analysis found that alumni who were employed prior to training were 3.4 percent more likely to obtain jobs after training than those who were not employed prior to training. It also found that men and older alumni were more likely than women and younger trainees to be employed posttraining. Unemployment and Labor Force Participation: Somewhat counterintuitively, the proportion of alumni that were unemployed—defined as not working but looking for work—increased from 14 percent before training to 17 percent after training. The simultaneous increase in employment and unemployment rates is the result of a higher labor force participation rate—i.e., more unemployed people seeking work. Participating in the labor force means that an individual either is working or has sought work in the past seven days. Sixty-three percent of alumni reported participating in the labor force prior to starting training, a value that increased to 80 percent for the same group after they completed their training. To determine whether observed increases in labor force participation and unemployment were caused by underlying economic conditions rather than the training, the analysis compared the employment and labor force participation status of current trainees—i.e., those who began their training just before the survey and represent what the status of alumni would have been had they not participated in training— with the status of alumni. The data show lower employment, unemployment, and labor force 10 For this question, the evaluation team calculated the percentage of respondents who were employed, unemployed, and not in the labor force based on the number of all respondents who were asked the question. Individuals were designated as employed if they had a job or business or claimed to have done work for pay, profit, or family gain for at least one hour in the seven-day period in question. If they were not employed, respondents were considered unemployed if they sought work in the same seven-day period. All others were designated as outside the labor force, and they were not excluded from the calculation. Therefore, the three categories always equal 100 percent. 11 t=-34.1, p=.0001 12 The pretraining employment rates are roughly consistent with publicly available figures. The 2014–15 Labor Force Survey found that 39 percent of people 20–24 years old, and 47 percent of those 25–29 year old, in urban Sindh province were employed. http://www.pbs.gov.pk/sites/default/files//Labour%20Force/publications/lfs2014_15/t18-pak.pdf Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 16 participation rates among current trainees than among alumni (Figure 4), which suggests that alumni would have been less likely to be employed or participating in the labor force and more likely to be unemployed if they had not participated in the training. Eight alumni who participated in group interviews reported being unemployed; the most common explanation (three of the eight) was that they could not supply the references requested by prospective employers. Together, these findings suggest that Amantech training did motivate more beneficiaries to seek employment, but that a lack of prior experience may have been a barrier. Table 6 provides a more nuanced look at how training has affected employment status and labor force participation. The table shows the percentage of alumni who transitioned from the status listed in the rows prior to training to the status listed in the columns after training. Of the 48 percent who were employed prior to training, most (78 percent) were employed after training, and only 9 percent had left the labor force. Similarly, 55 percent of those who were unemployed prior to training reported finding employment after training, and 62 percent of those who were not in the labor force prior to training had joined the labor force, either through employment or by seeking employment, after training. The transition matrix provides additional evidence that training had a greater effect on employment status for those who were either unemployed or not in the labor force prior to training than for those who were employed. FIGURE 4: EMPLOYMENT STATUS BEFORE AND AFTER TRAINING TABLE 6: TRANSITION OF ALUMNI BETWEEN STATES OF EMPLOYMENT Post-Training Employment State Employed (%) Unemployed (%) Not in Labor Force (%) Pre￾Training Status (%) Pre-Training Employment State Employed (%) 78% 13% 9% 48% Unemployed (%) 55% 35% 10% 14% Not in Labor Force (%) 46% 16% 38% 37% Post-Training Status (%) 63% 17% 20% A transition matrix shows the percentage of respondents who transitioned from the pretraining employment state (row) to the posttraining employment state (column). Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 17 Disaggregating the analysis by sex reveals some interesting differences in employment, unemployment, and labor force participation. While both men and women reported increased labor force participation and employment, the increase in labor force participation among men more than offset increased employment leading to an increase in the unemployment rate (Figure 5). However, women were less likely than men to begin participating in the labor force after training, so their unemployment rates actually declined after training. FIGURE 5: CHANGE IN LABOR FORCE PARTICIPATION AND EMPLOYMENT BY SEX Qualitative data sources also linked training to increased labor force participation. During group interviews, the evaluation team asked alumni what they thought they would be doing if they had not taken the training course. Half of 16 alumni group interview participants claimed they would likely be engaging in an activity that would preclude them from the labor force, such as continuing their studies or simply staying home. To determine whether increased labor force participation reflects a fundamental change in attitudes toward employment rather than temporary optimism following training, the analysis regressed labor force participation on time (months) since completing the training (and its square), pretraining labor force participation, sex, age, marital status, education, and town. The logit analysis found that labor force participation did not decline with the duration of unemployment after training—at least during the relatively short durations captured in the data. In fact, it found only pretraining labor force participation and sex to be significant determinants of posttraining labor force participation. Alumni who were participating in the labor force before training were 5.6 percent more likely than those who were not in the labor force prior to training to participate in the labor force after training. Similarly, men were 3.6 percent more likely to participate in the labor force after training than their female counterparts (Annex 8, Table 16). Disaggregated by age, older trainees (30 years old and above) were significantly more likely than younger trainees (18–29 years old) to be employed before or after training. However, training improved employment rates and prospects for younger trainees far more than for older trainees. Employment rates among older trainees remained virtually unchanged before and after training at about 74 percent, while younger trainees reported a statistically significant increase in employment rates from 45 percent to 62 percent. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 18 Training in more advanced electrical and computer skills appear to have had a greater impact on employment opportunities than training in more traditional and less technology-driven vocational skills.13 Most notably, employment rates increased among alumni of CAD-CIVIL training (from 33 percent to 67 percent of 21 respondents) and general electric training (from 56 percent to 84 percent of 17 respondents), and declined among alumni of automobile and plumbing training. 14 Labor force participation rates increased by at least 10 percentage points for alumni of all training types except plumbing. These results, however, are based on relatively small numbers of respondents (Annex 9) and are not statistically significant. Conclusions: Training increased labor force participation and employment rates relative to what they would have been without training. For men, the increase in labor force participation more than offset increases in employment, leading to an increase in unemployment rates as well. Alumni who were more likely to be unemployed at the start of the program (youth and women) registered greater gains in employment than those who were employed prior to training implying that training is more effective—in terms of employment—for those with less experience in the job market. Employment Type The survey data revealed no major shift in employment type for men before and after training—80 percent reported that they were paid employees before training compared to 81 percent after training. Women, however, shifted away from self-employment (34 percent of 30 employed women alumni before training compared to 21 percent of 42 after training) toward paid employment (66 percent before training to 79 percent after). The comparisons between employment types for alumni posttraining and current trainees (the counterfactual) corroborate the results of the before and after comparison for alumni. Men from either group are about as likely to be paid employees (80 percent of 99 posttraining alumni compared to 82 percent of 215 current trainees). Training does not seem to have affected the proportion of alumni engaged as wage employees but may have shifted some from unpaid family workers to formal paid employment. The proportion of wage employees was similar for all employed alumni before and after training: 77 percent of 129 employed alumni were paid employees before training, and 80 percent of 171 employed alumni were after. Meanwhile, the percentage of alumni working as unpaid family workers declined from 4 percent before training to 2 percent after training. Comparing the 2 percent of posttraining alumni in unpaid employment to the 9 percent of current trainees who worked as unpaid family workers before training (the counterfactual) suggests an even larger impact. Employed alumni were more likely to have either a written or verbal contract with their employer after training than before. The percentage of employed alumni with written contracts increased from 23 percent before training to 36 percent after training, while the percentage with verbal contracts increased from 3 percent to 12 percent. The 13 percentage point difference for written contracts is statistically significant.15 Comparing the 36 percent posttraining result for alumni with the 12 percent of employed current trainees with an employment contract (the counterfactual) suggests that the effect of training may have been as high as 24 percentage points. Having a contract, i.e., a written or verbal permanent employment agreement as opposed to casual or part-time work, did not appear to affect job satisfaction; the proportion of alumni who were satisfied 13 The survey of alumni collected more than 10 responses on only 9 training courses—stitching machinist, refrigeration and air conditioning, CAD-CIVIL, plumbing, general electric, computer-aided design and manufacturing (CAD-CAM), mechanical, automobile, and carpentry. In this report, analyses disaggregated by training consider only these training areas. See Annex 9 for a table of observations by training. 14 All survey results are weighted to reflect the population of trainees. Therefore, the percentages reported may not always seem consistent with the number of responses or respondents. 15 t=-1.85, p=0.033 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 19 with their jobs was about the same for those with and without a contract—53 percent of 48 individuals and 57 percent of 90 individuals, respectively.16 This was true for both written and verbal contracts, as roughly the same proportion of alumni with written (54 percent of 26 alumni) and verbal (55 percent of 22 alumni) contracts were satisfied with their jobs. Working Hours Working more hours is another indicator of more and better employment. A comparison of employed alumni’s before and after training weekly hours of work revealed an average increase of three hours worked per week for men and seven hours per week for women (Figure 6). The comparison between posttraining alumni and current trainees (the counterfactual) also suggests that training increased average hours worked for men (by four hours per week). Changes between before and after training work hours were not statistically significant for men or women17 or for both sexes combined.18 FIGURE 6: AVERAGE WEEKLY WORKING HOURS BEFORE AND AFTER TRAINING The evaluation team was not able to collect any data from female current trainees. Income The analysis considers only personal income from employment and excludes individuals with no income. Estimates of median income therefore exclude respondents who are not employed. Based on a comparison of before and after training median monthly income, training increased incomes for both men and women. Men’s median monthly personal income increased by 10 percent (from $78.43 to $86.60)19 while women’s income increased by 19 percent, from $39.22 to $46.69 (Figure 7). The increase in men’s median income is statistically significant. The larger percentage increase for women is probably also meaningful, but the relatively small number of observations makes the difference statistically undetectable.20 The comparison between posttraining income of alumni and pretraining 16 t=0.143, p=0.444 17 Men t=-0.48, p=0.633 Women t=-1.29, p=0.202 18 t=-0.74, p=0.458 19 Pakistan rupees were converted to U.S. dollars based on the 102 rupees per dollar average month-end exchange rates between January 2014 and December 2016 reported by the State Bank of Pakistan. 20 The survey did not provide sufficient detail about when respondents were working to rigorously convert nominal income to real income. Pakistan experienced 7 percent inflation between January 2015 and December 2016 (the approximate period of time between the start of the project and the evaluation). The increases in income are greater than 7 percent, so they more than offset inflation and thus represent changes in real income. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 20 income of current trainees (the counterfactual) suggests even larger increases in income relative to what it would have been without training, but because the income of current trainees is not a rigorous counterfactual, the before and after comparison is probably a better measure of the effects of training. Disaggregated by type of training, the number of observations is too small to detect significant changes in median income (see Table 18 in Annex 9). The data suggest, however, that alumni of plumbing and carpentry training enjoyed the greatest improvements in income—66 percent and 38 percent, respectively (see Table 17 in Annex 9). Because it is difficult to accurately measure income, estimates of the magnitude and direction of changes in median income are rough at best, especially for small disaggregates such as by training topic. FIGURE 7: MEDIAN PERSONAL MONTHLY INCOME BEFORE AND AFTER TRAINING Note: “n=x/y” denotes the number of observations on men/women. The evaluation team was not able to collect any data from women current trainees. The analysis employed a regression model to examine the determinants of posttraining income among employed alumni (Annex 7). The analysis concluded that being employed prior to training, sex (male), and age all had positive and significant effects on income while marital status (unmarried) depressed income. Job Satisfaction Satisfaction with a job is an indicator of overall job quality; it may incorporate dimensions of salary, work environment, consistency with interests or lifestyle, and proximity to home. Men reported slightly higher levels of job satisfaction after training than before, while women reported lower levels, although none of the differences were statistically significant (Figure 8).21 Alumni of three training areas reported increases of more than 25 percentage points in job satisfaction after training: automobile (47 to 84 percent), CAD-CIVIL (36 to 63 percent), and plumbing (36 to 62 percent). Alumni of stitching machinist training, however, reported lower job satisfaction rates after training (65 to 59 percent). Alumni of the electrical and electronics training had the lowest job satisfaction rate before and after training (15 and 27 percent, respectively). The data are not sufficiently detailed or of sufficient quantity to determine the causes of differences in satisfaction. Furthermore, 21 t=-0.356, p=0.361 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 21 some factors that affect satisfaction, e.g., proximity to home and work environment, have little to do with the training per se. FIGURE 8: JOB SATISFACTION OF EMPLOYED ALUMNI When asked to explain why they were satisfied or dissatisfied with their jobs, respondents most frequently mentioned salary, opportunities for career growth, and proximity to home (Figure 9 ). Women were much more likely than men to mention a job’s proximity to their homes as a determinant of job satisfaction (41 percent of women compared to 28 percent of men). Men more often mentioned opportunities for career growth (21 percent of men compared to 6 percent of women). This may indicate that men are more concerned with long-term considerations when assessing employment while women care more about access to their jobs given their limited mobility. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 22 FIGURE 9: DETERMINANTS OF JOB SATISFACTION Group interview participants cited additional characteristics of a good job. Fifty-three percent of 32 participants noted a high salary, while 38 percent mentioned a good work environment and facilities. However, location of—and transportation to—the workplace was still a consideration for 28 percent of participants (13 percent of men and 44 percent of women). As one noted: “A good job is given to the one with good qualities. The work environment should be good, where you can adjust well. I lived nearby so I could easily join Amantech but my workplace was too far and it was inconvenient for me to travel that far. Public transport took a lot of time.” – Female alumna, Korangi Town A logit regression of an indicator of being satisfied with a job (an aggregate of highly satisfied and satisfied) found no statistically significant relationship between job satisfaction and sex, age, marital status, income, whether the job matched training, or having a written or verbal contract. Having completed a secondary or intermediate education, however, had a positive impact on the likelihood of being satisfied with a job. See Annex 7 for the regression results. Job Placement and Retention Amantech did not consistently support alumni after training—38 percent of alumni reported receiving no support finding a job after the training. Those who did obtain support reported that Amantech placed them directly in a job, provided assistance finding a job, or helped with the application process (Figure 10). Women were more likely than men to have received support (81 percent compared to 54 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 23 percent), but the evaluation found no evidence to explain this disparity. Only 3 percent of alumni reported receiving career counseling, and 1 percent were placed in an apprenticeship. FIGURE 10: JOB PLACEMENT SUPPORT AFTER TRAINING The group interviews provided more nuanced information than the survey about posttraining job placement support. Twelve of the 16 alumni participants reported receiving some sort of support in obtaining employment after training. This included job offers and references to prospective employers. However, many of the job opportunities did not pan out because salaries were too low (four respondents), prospective employers did not follow up (two respondents), or the jobs were too far from the candidates’ homes or in unsafe neighborhoods (one respondent). The following quote illustrates one alumna’s experience. “Vietiem and Amantech had a contract in place that after three months they will offer us a job. However, they told us that they will hire us on piece-rate, which I think is very unfair. One bundle has four pieces and they were giving 50 rupees per bundle. Based on this, we would hardly be able to earn 100 rupees per day and that is too little.” – Female alumna, Korangi Town Another alumnus suggested that the training did not prepare students adequately. “The head of the refrigeration and air conditioning department offered me a job at the institute as an instructor. However, the salary package they offered was not acceptable to me so I didn’t accept this offer. The placement cell at the institute is very ineffective. They invite interview panels from factories. The interview questions cover both theory and practice. The students perform well in theory but only a few students pass the practical part. To make its students competent, the institute needs to improve its teaching methods in practical areas. Moreover, it should send students to factories and industries to gain some practical experience as it happens in engineering universities.” – Male alumnus, town unknown Job retention rates were relatively high among employed alumni. Seventy-eight percent of alumni who have had a job since they completed training (either a new job or the same job they had prior to starting Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 24 the training) are still in the first job they obtained. Alumni employed in retail trade reported the highest retention rates (89 percent), while those employed in sales, maintenance, and repair of automobiles and motorcycles reported the lowest retention rates (71 percent). The data provide little firm explanation for the small difference in retention rates by industry (Table 7). Women were as likely as men to have retained their first job. Similarly, differences in retention by characteristics such as a job that matches training, self-employment versus paid employment, and contract type were generally small and never statistically significant. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 25 TABLE 7: JOB RETENTION RATES (PERCENT) BY INDUSTRY AND ALUMNI CHARACTERISTICS Industry Overall Sex Job Matches Training Employment Type Contract Type Men Women Yes No Self Paid None Written Verbal Manufacture of textiles 84.0% 81.8% 86.4% 80.1% 92.4% 100.0% 82.5% 84.8% 66.3% 87.2% Sale, maintenance, and repair of motor vehicles and motorcycles; retail sale of automotive fuel 71.2% 71.2% n.a. 56.6% 91.8% n.a. 71.2% 82.8% 67.8% n.a. Retail trade, except of motor vehicles and motorcycles; repair of personal and household goods 88.9% 88.9% n.a. 73.7% 95.2% 79.9% 93.8% 92.3% n.a. 100.0% Education 82.4% 100.0% 77.8% n.a. 82.4% 71.8% 88.2% 91.9% Others 73.1% 74.9% n.a. 71.2% 75.2% 86.4% 71.4% 69.6% 81.4% 39.3% Total 78.0% 77.8% 79.1% 72.7% 83.7% 84.4% 77.3% 80.2% 76.1% 65.4% n.a. means “no data” Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 26 Relevance of Jobs to Training “A good job is one that matches your skill set and allows you to earn a reasonable salary, preferably within an established institution.” – Male current trainee, Malir Town Slightly more than half of employed alumni (52 percent of 171 respondents) were working in occupations that matched their training course. By sex, 54 percent of women and 51 percent of men were working in jobs matched to their training. Employed alumni most frequently reported finding matching jobs after training on general electric (74 percent of 13 respondents), carpentry (72 percent of 11 respondents), stitching machinist (63 percent of 55 respondents), and refrigeration and air conditioning (63 percent of 22 respondents). After controlling for other variables, alumni employed in occupations that matched their training earned $9.80 per month more than alumni employed in occupations that did not match their training.22 The evaluation team interviewed four industry leaders who have experience hiring graduates of Amantech. Two believed that graduates of the program have better skills than other recruits. Another stated that because Amantech graduates usually come from disadvantaged communities, he prefers other recruits. The fourth industry leader noted that: “Generally speaking, their performance is good. However, they are not motivated to remain with the company for a long period of time.” Project beneficiaries were aware of the importance of both hard and soft skills in obtaining employment. Fifty-nine percent of 32 group interview participants said that interpersonal/communication skills or proper behavior were skills that employers in their industry look for when hiring. Fifty-six percent discussed having certified skills or a proper education as a factor in being hired. In spite of providing beneficiaries with a variety of skills, the program is unable to overcome what group interview participants felt was the greatest challenge for young people to obtain desirable jobs. Fifty-nine percent of the 32 participants said that nepotism and bribes were the greatest challenge. This kind of corruption in the broader job market of Karachi is likely a major impediment for trainees who come from the most marginalized areas of the city. A group interview participant expressed this view: “People get exploited, as employers ask for bribes. It’s difficult to gain employment without giving a bribe. Secondly, the nepotistic culture is another major challenge.” – Male current trainee, Landhi Town Q2(b): What effect is the program having on keeping youth away from violent groups or activities? The surveys of alumni and current trainees also asked respondents if they thought vocational training— for youth such as themselves—would deter youth from participating in violent extremism or crime, and 90 percent of 769 respondents said they believed that it would. The survey asked respondents to explain why, and the most frequent responses were by changing attitudes, increasing employment, and leading youth to use their time more productively. At the aggregate level, for all towns, 90 percent of 769 respondents thought that training and employment would reduce crime and violent extremism. When asked to explain how, 41 percent attributed the anticipated reduction to employment generally, 33 percent thought that youth would have less time available for “negative activities”, and 25 percent 22 The analysis employed an ordinary least squares model to regress posttraining monthly personal income on a binary variable indicating that employment matched training, pretraining employment status, sex, age, marital status, education level, and neighborhood. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 27 believed training would change attitudes. Figure 11 shows the percentage of respondents who gave each reason disaggregated by town. FIGURE 11: HOW TRAINING CONTRIBUTES TO REDUCED CRIME AND VIOLENCE The survey asked respondents whether they contributed to household income, general household affairs, and the wellbeing of their neighborhoods in any way, in order to assess whether these positive outcomes could be associated with the project. Figure 12 illustrates contributions disaggregated by sex. The only significant change between before and after training was in contributions to household income, where both men and women reported making a greater contribution. Otherwise, men were more likely than women to report contributing to household affairs and the wellbeing of their neighborhoods. FIGURE 12: SURVEY RESPONDENTS’ CONTRIBUTIONS TO HOUSEHOLD INCOME, HOUSEHOLD AFFAIRS, AND NEIGHBORHOOD WELLBEING Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 28 Q3: How, if at all, has the perception of insecurity changed in targeted towns? In the absence of a baseline for this question, change in perception of insecurity cannot be measured over time. Instead, the evaluation discusses the current state of perceptions about insecurity in the targeted towns, and this can serve as a baseline for future evaluations. This analysis also could inform decisions on selecting towns in Karachi for similar future activities. This section presents data from the surveys of alumni and current trainees on various dimensions of security, such as the perceived level of crime and violence, its effect on households, the effects of religious and political groups, and perceptions of employment opportunities and poverty. The analysis for question 2(b) found that an overwhelming majority of respondents believed that training reduced the likelihood of young people engaging in criminal or violent activities. All five experts interviewed for the evaluation believed crime in Lyari, Korangi, and Sultanabad had decreased over time, and four attributed the change to the ongoing Rangers operation in the city. Experts were unable to predict whether the lull in violence would be permanent. Respondents reported higher levels of crime and violence in Lyari, Landhi, and Korangi than in other towns (Figure 13) but believed that the impact of crime and violence on their households was greater in Baldia, Lyari, Landhi, and Korangi (Figure 14). FIGURE 13: PERCEIVED LEVEL OF CRIME AND VIOLENCE Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 29 FIGURE 14: IMPACT OF CRIME AND VIOLENCE ON HOUSEHOLDS The survey also asked about the impact of religious groups and political parties on the respondents or their households. No clear pattern emerged on the perceived effect of religious groups. About half of all respondents saw them as having a positive effect, with most of the rest reporting no effect (Figure 15). Similarly, most respondents in all towns reported that political parties had no effect on their households. Respondents from Lyari and Baldia were somewhat more likely to report a negative effect, but they do not stand out substantially from other towns in this regard (Figure 16). FIGURE 15: PERCEIVED IMPACT OF RELIGIOUS GROUPS Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 30 FIGURE 16: PERCEIVED IMPACT OF POLITICAL PARTIES Respondents were not optimistic about the employment prospects in any of the towns. Less than 16 percent believed there were many employment opportunities in their towns. The distribution of the remaining responses across “None” and “Few” varied somewhat by town, with respondents from Lyari and Saddar seeing the least opportunity and those from Korangi seeing the most (Figure 17). Responses revealed no clear pattern in perceived levels of poverty (Figure 18). FIGURE 17: PERCEPTIONS OF INCOME-GENERATING OPPORTUNITIES Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 31 FIGURE 18: PERCEPTIONS OF POVERTY LEVEL The surveys also asked respondents to rate the effectiveness of government services in their neighborhoods. Government services referred to municipal services such as water, sanitation, education, and health. Respondents from Saddar were the most dissatisfied with government services, while those from Baldia were most satisfied (Figure 19). FIGURE 19: RESIDENTS’ SATISFACTION WITH GOVERNMENT SERVICES Conclusions: The indicators of insecurity measured in the survey suggest that all of the towns are relatively insecure. While differences are not stark, Korangi and Lyari, two of the three towns targeted by USAID, rank among the least secure towns on several key indicators. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 32 Q4: How are stakeholders involved in the project? Amantech follows a very different approach than United Nations Development Program (UNDP) 23. One main difference is that Amantech manages the entire process itself, with relatively minor involvement of stakeholders such as communities and private sector employers. The findings in this section are based on seven in-depth interviews with Amantech staff (2) and industry representatives (5) in addition to survey data collected from alumni. Q4(a): What was the community’s involvement in planning and implementing this project? Based on interviews, the evaluation team found little evidence that Amantech actively involved the community in planning or implementing the project. The five industry representatives noted that community members, e.g., alumni and elders, facilitated recruiting by talking about their experiences (alumni) and identifying potential candidates for training (elders). For example, a USAID representative mentioned having attended two neighborhood events in which alumni met with employers to discuss their concerns related to training and employment. These events constitute a level of community feedback on implementation. Both Amantech representatives the team interviewed confirmed that Amantech has not involved the community in planning or implementing the project and instead relied on its own staff and the staff of the Aman Foundation for mobilization, recruiting, and marketing activities. However, they explained that Amantech is planning to engage social mobilizers in the future to help recruit youth from marginalized communities. The social mobilizers will use identification and recruitment methods such as awareness￾raising campaigns, motivational talks, and visual displays. These community-based identification and recruitment approaches complement the existing approaches that Amantech uses, such as social media, television, and newspaper advertisements. Q4(b): What was the private sector’s involvement in planning and implementing this project? Amantech works closely with the private sector across a comprehensive range of vocational training and employment-related services, including recruitment, training, job matching, job placement, and follow-up services. For recruitment, private sector enterprises contact Amantech when they need semiskilled and skilled workers to fill existing vacancies. In response, Amantech refers graduates to these employers as potential candidates. Amantech’s Job Placement Unit has developed job vacancy and job seeker databases that contain descriptions of potential employment opportunities provided by the private sector and profiles of job seekers. The Job Placement Unit uses these databases to match and refer potential candidates to private sector employers. Amantech also works closely with the private sector to ensure that trainings meet the needs of potential employers and comply with national standards, such as those established by the Trade Testing Board or international standards established by City and Guilds. All Amantech trainings are accredited based on national or international standards. Two of the five industry representatives interviewed explicitly mentioned that employers augment Amantech’s basic training in a trade area with in-house, specialized training. One of the industry representatives stated: “There is always a gap between the skills that trainees receive from the training institution and the skills they need to work at this company. Amantech graduates 23 The UNDP implemented a similar USAID-supported training program in Karachi. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 33 understand that they need to learn new skills and have the capacity to learn such skill sets because their basic skill sets are good.” Lastly, Amantech collaborates with numerous private sector enterprises on apprenticeship schemes. All of the private sector representatives the evaluation team interviewed praised Amantech for matching graduates to private sector jobs and providing strong job placement and follow-up services. Amantech staff explained that they work closely with the private sector to match trainees to jobs, by striving to understand the needs of each employer before they refer their graduates. Amantech also follows up with enterprises that hire Amantech alumni to find out how its graduates are performing on the job. The evaluation team was not able to learn the results of these follow-ups. Amantech continues to provide follow-up services to its graduates at least three months after job placement. Amantech classifies alumni as employed only when they have been employed for at least three months after graduating. The follow-up services provided by Amantech include contacting graduates and their employers to assess whether both the graduate and the employer are happy with the job placement. Q4(c): What was the involvement of other key stakeholders in planning and implementing this project? Amantech maintains a cursory relationship with the government, mainly because it is registered as a vocational training institute with the Sindh Technical Education and Vocational Training Authority (STEVTA). Amantech staff explained that, for all practical purposes, this is the extent of its involvement with STEVTA. Although it does not represent meaningful involvement in planning or implementation, Amantech also hires various private sector entities to develop its media and awareness-raising campaigns, which include television and newspaper advertisements as well as social media networks. Q4(d): What, if any, cost-sharing has the project benefitted from? The only cost-sharing that occurs with Amantech’s vocational training is the fee that trainees pay. According to trainees, the fee ranged from Rs. 1,000 to 30,000 depending on the nature and duration of the course. Private sector employers have not contributed to any of the costs associated with the training. Reflecting on this, one of the Amantech staff members stated: “In a few places, we do collaborate with industry, for example some of our labs were made with the help of industry, and in textile courses we have a link with one of the industry enterprises. However, we cannot specifically associate these contributions from the private sector to this USAID project.” When the evaluation team asked alumni if they would be willing to pay for similar training in the future, about two-thirds said they would (Table 8). This demonstrates a level of satisfaction with the courses provided by Amantech. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 34 TABLE 8: ALUMNI WHO WOULD PAY FOR SIMILAR TRAINING IN THE FUTURE Alumni Respondents Count Yes (%) No (%) Not Sure (%) In the labor force? Yes 186 66% 32% 3% No 44 66% 22% 13% Total 230 66% 30% 4% Employed? Yes 148 67% 30% 3% No 82 62% 30% 8% Total 230 66% 30% 4% Matched to job in which trained? Yes 75 66% 30% 3% No 73 68% 30% 2% Total 148 67% 30% 3% Q5: Is the project scalable and can local resources be leveraged to transition it out of donor dependence? Q5(a): To what extent are project activities scalable? The evaluation team discussed the potential for scaling up Amantech’s project directly with seven respondents, including two Amantech staff members and five industry representatives. Three of the seven respondents believed that a project of this nature should be expanded to other locations. They explained that many young trainees who reside in different marginalized/at-risk communities across Karachi face difficulties in accessing Amantech’s single training center in Korangi due to the distance, and proposed various solutions. One of the industry representatives suggested: “More communities in Karachi should be targeted than just Korangi, Lyari, and Sultanabad. The SITE industrial area should be contacted for job placements of youth from Lyari and Sultanabad.” However, while recognizing the need for other centers, an Amantech representative stated the organization’s limitations: “Replicate the project in other areas because there is a need, but USAID would have to find other implementing partners because Amantech does not have centers in other areas.” Private sector respondents spoke highly of the quality of vocational training Amantech provided but noted that the trainees they hire often receive further specialized on-the-job training, in addition to the basic training received from Amantech. However, industry representatives explained that there is a shortage of skilled workers in many occupational areas, so if Amantech could train a larger number of individuals using its demand-driven approach and subsequently refer them to the private sector, the collaboration would result in more gainful employment for the youth. Private sector respondents also noted that there is a need to improve trainees’ soft skills to ensure that they possess the right attitude and work ethic in addition to basic technical skills. Employers emphasized that Amantech should consider approaches to training and placing youth closer to their residences. The private sector respondents believed that by decentralizing training activities, a larger number of marginalized/at-risk youth would benefit. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 35 In conclusion, a scale-up of Amantech’s approach is limited by the fact that Amantech has only one training facility, and youth who reside in areas far from the facility find it time-consuming and costly to commute long distances. The solution is either to partner with other training centers or to provide incentives to trainees to attend the main Amantech facility. Q5(b): Could the trainings continue without USAID funding? If yes, how? If no, why not? The evaluation team discussed sustainability of project training and placement activities with two Amantech staff members and five independent experts. The Amantech representatives said that they will continue to provide vocational training in the same occupational areas, even if USAID funding is no longer available. However, the target population of this project—i.e., youth from marginalized/at-risk communities—would not continue to be the primary target group since Amantech’s mandate is to provide high quality, demand-driven vocational training to any interested individuals. The independent experts suggested that Amantech needs to strengthen the engagement of government and the private sector in its activities so that a larger number of youth from marginalized/at-risk communities gain access to the vocational training that Amantech offers. The experts believe that it is a responsibility of both government and the private sector to design and implement a vocational training approach similar to Amantech’s but one that is locally based and more accessible to youth in marginalized/at-risk communities. Amantech could help in the design of such a model, which would be available at the local community level. Government officials stated that the best way to sustain a vocational training system that both meets Amantech’s quality standards and targets youth from marginalized/at-risk communities is to design an alternative model. This model should not be dependent on a single central training institution. Instead, training and job placement should take place at the local level, closer to the marginalized communities. Even though such a community-based model may not be suitable for Amantech to implement, Amantech could at least support it technically. In conclusion, the sustainability of a project similar to Amantech’s Karachi Youth Workforce Development Project is questionable. However, if Amantech were to provide its vocational training and employment services through a more local, community-based approach, then a larger number of youth from marginalized/at-risk communities could potentially benefit. A local, community-based model could prove to be successful if it embraces the values and principles of Amantech and implements activities that are consistent with current Amantech standards. Prospects for sustainability may also be more promising if trainees are willing to pay for the training. A regression analysis reveals that alumni who were employed after training and who came from relatively wealthy households were more willing to pay for future training than those who were not employed or came from relatively poor families. Similarly, alumni who were unmarried or older than 29 at the time of the survey were more willing to pay for future training than married and younger alumni.24 These results suggest that programs may be more self-supporting if they can attract students with a greater ability to pay (e.g., from wealthier households, older, or without the financial obligations of a family) and provide training in occupations with good employment prospects. 24 The dependent variable was a binary variable that took the value of 1 if alumni were willing to pay for future training and 0 otherwise. Independent variables included posttraining employment status, household monthly income, alumni personal characteristics (age, gender, education), and neighborhood. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 36 CONCLUSIONS Amantech did not employ a community-based recruiting strategy that targeted specific towns. Consequently, only about half of applicants, and trainees, came from the five most marginalized towns in Karachi (based on the marginalization index) or the three towns USAID identified as priorities. Quantitative and qualitative evidence strongly suggests that the time and cost associated with travel between Amantech’s training facility in Korangi and beneficiaries’ homes are among the most important determinants of individuals’ choices about whether and where to engage in training. The proximity of the training venue to a neighborhood is particularly important for women who face additional cultural barriers to traveling far from their homes. Amantech staff said they are aware of this limitation and that they plan to collaborate with community-based organizations in marginalized/at-risk communities to increase the number of applicants and participants in future training. However, this will not directly address the issues created by Amantech’s having only one training venue. Amantech did not apply any criteria, other than a test, to select trainees from among the pool of applicants. Consequently, the socioeconomic profile of alumni was similar to that of their age cohort in greater Karachi. Training generated more and better jobs measured by a number of indicators. It increased labor force participation and employment rates, increased the unemployment rate slightly (a result entirely attributable to increased labor force participation), facilitated a shift from self-employment toward more stable wage employment for women, and increased median incomes. Disaggregating these overall results generates additional insights. Specifically:  Only 52 percent of the employed alumni were working in occupations for which they had been trained. This demonstrates the importance of (1) training individuals for occupations in which jobs exist and (2) providing career counseling, placement, and follow-up services to place them in jobs matched to their training.  Technical and technology-focused trainings (e.g., CAD-CIVIL, electrical and electronics, general electric, vehicle body technician, and stitching machinist) generated larger increases in employment than more traditional and manual skills training. This suggests that the market demands these skills and emphasizes the importance of strategically selecting training programs to best meet employment and income targets.  Two of the trainings with the highest alumni employment rates reported the lowest rate of matching occupations. Only one-third of electrical and electronics (6 respondents) and one￾quarter of CAD-CIVIL (13 respondents) employed alumni were in occupations that matched their respective trainings. This finding suggests that, while these specific fields may not have a large number of accessible employment opportunities, the technical skills learned from the trainings are relevant to other—sometimes unrelated—occupations. The evaluation did not find strong evidence to confirm or refute the development hypothesis that better employment opportunities contribute to reducing crime and violence. The speculative (i.e., not based on experience) evidence that supports the hypothesis suggests that the primary mechanism through which employment may contribute to these positive outcomes is by productive employment’s taking the place of “negative” activities. Amantech appears to implement its training approach well; it engages with potential employers to identify labor market demands and reportedly prepares trainees relatively well to meet these demands. While stakeholders see a need to expand vocational training projects to engage more marginalized/at- Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 37 risk youth, Amantech’s nontargeted recruiting strategy, single training venue, and fee structure limit its potential to scale up the approach, target it to specific populations, and wean it from donor support. Amantech may be able to scale up recruiting from targeted towns at the margins by increasing its engagement with communities to identify local labor market needs and target potential trainees, but this will require deeper engagement with communities and a community-focused recruiting strategy. RECOMMENDATIONS  To more effectively target the most marginalized or at-risk youth for training, Amantech could implement community-based recruiting activities that specifically target youth from marginalized/at-risk communities and screen prospective trainees for socioeconomic characteristics associated with marginalized or at-risk youth.  It is probably not feasible for Amantech to establish additional training facilities in towns other than Korangi. If USAID/Pakistan or Amantech wishes to increase recruitment from specific, targeted towns, Amantech must develop alternative implementation models. These might include (1) engaging partners with training facilities in the targeted towns and (2) finding some way—e.g., using government or donor support to compensate individuals for the time and cost of travel to Amantech’s training facility—to attract recruits from distant towns. This will be especially important in efforts to boost the number of women recruits. Amantech could pilot one or more of these approaches under the USAID-funded project.  Two strategic approaches Amantech might consider, with the support of the government, to reach more marginalized towns far from Korangi include (1) establishing or supporting satellite vocational training centers in coordination with existing local vocational training centers and (2) setting up mobile skills development/vocational training units to provide vocational training that is certified by the Trade Testing Board in local communities.  To meet employment and income targets, Amantech should research and strategically focus on the trades or occupations in which it provides training and work with the private sector to identify the skills required in these occupations and trades. It then needs to ensure that its training, in hard and soft skills, is focused on meeting market demands. It might also provide better career counseling services that match trainees to trades for which there is a strong demand for skilled workers. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 38 ANNEXES Annex 1: Evaluation Scope of Work Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 39 The curriculum is regularly being updated according to the needs of the garment industry. The training is provided on-site in the factories - this is an in-ki nd contribution from the industry. Similarly, the other component of the WFD program is with Amantech. USAID's $7M cooperative agreement with AMANTECH is a three year grant for skill building and training for 3600 underprivileged youth of Karachi. The program will see students receive one year equivalent certification training after which they will be placed with employers locally and internationally. AMANTECH will work with youth from poor neighborhoods providing a mi x of vocational and soft skills training in order to facilitate employment opportunities locally and abroad, enhancing their earning potential and generating foreign currency remittances for Pakistan. Spread over 6 acres in Karachi, AMANTECH faci lity comprises 21 fully equipped workshops, six computer labs, and lOO+ faculty members where it offers hands-on practical training to students in market and demand dri ven skill requireme nts. In addition to technical skills, AMANTECH also develops its students' competence in soft skills. Soft skills is a sociological term relating to a person's "EQ" (Emotional Intelligence Quotient), the cluster of personality traits, social graces, communication, language, personal habits, friendliness, and optimism that characterize relationships with other people. Soft skills complement hard (technical) skills that are occupational requirements of a job and many other activities. In order to deliver world-class technical training, AMANTECH has accessed the ex pertise of international vocational education organizations, such as City & Guilds, UK, and Skills International, Sri Lanka, that specialize in designing and executing vocational training programs. The main goal of the grant to AMANTECH is to e nhance the earning potential of Pakistani youth from underserved and vulnerable communities through access to internationally accredited and market driven vocational skills. Within this goal there are two main objectives that the project will measure performance agai nst. The first is the graduation of 3,600 ( 12-month course equivalent) students over 3 years and the second is the target placement of 70% of these graduates with companies through institutional linkages. Some of the manufacturing trades in which the youth will be trained are automobile, mechanical, general electrical, refrigeration and air-conditioning, fabrication, welding and pipework, electronics, CAD/CIVIL, CAD/CAM, Body & Paint and textiles. Construction trades will include carpentry and plumbing. Development Hypothesis Making investment in skills development of youth and providing them with employment opportunities will not only help reduce violent extremism in the country but also help establish a foundation for long term economic growth and prosperity. (Eva lua tion team should focus on testing this development hypothesis) p urpose, Ad" u 1ence an dL earnmg Ob" f ( ec 1ve reqm. r ed) Assignment Intended Learning Information Timeline Purpose Audience Objective Source How does the USAID Karachi To inform future Perception of Generally geographic geographic communities and needed to 2 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 40 targeting of the targeting in experts inform future project impact Karachi course of the over all objective project of reducing YE? How effective is USAID Karachi To inform the Ve rification of Generally the beneficiary future selection project reports, needed to selection process for analysis of inform future similar programs performance data course of the and anecdotal project evidence from beneficiaries How effective is To learn what Generally the project's mechanisms are needed to placement more effective inform future mechanism? for placements in course of the future programs project How has securing To describe to Generally jobs impacted the external needed to trainees audiences what inform future household and the effect the project course of the trainees role in the has had, for project community? whom and how. Key Evaluation/Study/Assessment Questions (required for evaluations, studies and assessments): • How effective is the program in targeting beneficiaries from marginalized/at-risk communities/households • How effective is the program's recruitment strategy? What percentage has been employed? • What impact is the program hav ing in keeping youth away from violent groups/activities? • How is the program impacting the targeted communities (towns) at large? o How has the level of reported crime and violence changed in target communities? Since the start of the project? In comparison w/ other areas? o How has the perception of insecurity changed in target communities? Since the start of the project? In comparison w/ other areas? • What is the stakeholder involvement in the project? o What was the community's involvement in planning and impl ementing this project? o What was the private sector's involvement in planning and implementing this project? o What was the local government's involvement in planning and implementing this project? o What, if any, cost-sharing has the project benefitted from? 3 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 41 Methodology (required): USAID's focus for this study is to understand the answers to the questions above so that it can gauge the effectiveness of the program as well as use the information for future programming. Approaches for each question are suggested below but it is left to the assignment team to determine the best methodology to collect and analyze the information. 1) How effective is the program in targeting beneficiaries from marginalized/at-risk communities/households? This question elaborates on which socio-economic groups the trainees are coming from. Economics is one factor, while ' at risk' may reflect the larger social landscape within which a household or community exists. Determining household income levels compared to the city income levels and the neighborhood income levels is important but there also needs to be an index which gauges which neighborhoods are more prone to violence or from where youth are recruited for violent activities. Neighborhoods could be indexed and index values assigned to each trainee to understand the profile of who it is USAID is supporting for its training and whether we are reaching our targeted population. This information can be built off existing resources available with UNDP and Amantech both of whom have carried out surveys and profiles of the neighborhoods they recruit from. The existing information may be cross￾referenced with violent incidents, presence of armed gangs and other factors that may take youth in a violent direction. 2) How effective is the program's recruitment strategy? Who is getting selected and who is getting left out? This is an evaluation of the quality of recruitment strategy by UNDP and Amantech. Are the strategies cost effective but more importantly are they allowing us to attract the right target population? Building on the information from the previous question is it possible to create a profile of trainees who are most vulnerable to fall victim to recruitment by extremist groups and to see if the recruitment strategies employed by USAID are actually getting these boys off the street and into employment training. Once the profile is established, it needs to determine out of the potential pool from the targeted neighborhoods, how many in the vulnerable profile get recruited and how many are left out. 3) What percentage has been employed and what kind of employment? This information should be available from the two IPs right away. However, USAID wants to understand the sector in which trainees are hired and under what conditions (full-term, part-time, contract, etc.). The idea is to understand which training matched with which kind of employment lead to longer term job/financial security so that programming is effective over a longer time. The information gathering here will look at the trainees who have been hired, the kind of contracts they are receiving, whether they have better contracts/income than non-trainees and any other factor the assignment team thinks can be an indicator of better job security. Since Amantech has been doing this for some time there may be availability of alumni data who can be approached to assess how trainees do over a longer period of time and this can also be correlated against sector, profile of trainee, type of contract etc. 4 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 42 4) What impact is the program having in keeping youth away from violent groups/acti vities? This may require comparing the recruited youth profile with those of simj)ar background and profile who do not opt for training. If they do not opt for training then what acti vities are they involved in. This would require taking sample size from trainees vs. non-trainees and if not a statistical analysis then at least a qualitative differe nce between the two group needs to be shown if it exists. This would build on earlier questi ons where profiles are built from different neighborhoods and the n comparing within those pro files those who trai ned and those who did not. 5) How is the program impacting the targeted communities (towns) at large? Since the neighborhoods are very large compared to the trai nees, it would be difficult to assess impact at a large scale. However, a before a nd after comparison between trainee households and a control group households can be conducted and inferences can be drawn as to the impact had a ll household been involved with training. This would mean collecting household income data as well as factors linked to violence (index referred earlier) within households and then compa ring it before and after training for both target and control groups. The sub questions around reported crimes and perception of insecurity may not be perceptibly impacted by the project but are there to provide an understanding of the larger landscape the projects are operating in. 6) What is the stakeholder involvement in the projects? This would require unde rstanding how the project was designed and which stakeholders were in volved and to what exte nt. The methodology should involve qualitative inte rviews with the implementers and possibly other stakeholders to evaluate what worked well and what fell short in the design and why. Team Composition (minimal description required): USAID will review PERFORM's proposed team composition for each evaluatio n which should include, at a minimum: • Team Leader/Evaluation Specialist - prioriti ze socia l science research person rather tha n subject matter expert • 1-2 Subject Matter Experts • 1-2 Local Research Specialists Deliverables (required) 1. Assignment Work Pla ns (A WP): Proposed approach, methodology, timeline, staff composition, and estimated budget for completion of the work requested in the SOW. A WP drafts for both projects will be reviewed and approved by USAID before any work begins on the assignment. T he A WP drafts will be revised and fina lized during the team planning meeting (TPM) once a n assignment team is in country. The fina li zed A WPs can be modified with PERFORM COR approval throughout implementation of the assignment if conditions or needs change. The fina lized A WPs 5 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 43 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 44 Annex 2: Assignment Work Plan Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 45 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 46 ACRONYMS AWP DFID EGA GVI lP MIS MSI M ITI o/a PERSI PHMA SBTE sow SPO STEVTA TPW uc UNDP USAID VE VTCG-K YWFD Assignment Work Plan Department for International Development Economic Growth and Agriculture Government Vocational Training Institute (Women) Nayabad Khadda Lyari Implementing Partner Management Information System Management Systems International M em on Industrial and Technology Institute on or about Pakistan Expanded Regional Stabilization Initiative Pakistan Hosiery Manufacturers and Exporters Association Sindh Board ofT echnical Education Scope of Work Strengthening Participatory Organization Sindh Technical Education and Vocational Training Authority Team Planning Workshop Union Council United Nations Development Programme United States Agency for International Development Violent Extremism Vocation Training Center (Girls)- Korangi-5 Youth Workforce Development PERFORM ASSIGNMENT WORK PLAN Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 47 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 48 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 49 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 50 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 51 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 52 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 53 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 54 Survey of AMANTECH Alumni The evaluation team will select a random sample of AMANTECH alumni from the census of trainees; stratified by type and length of training and time since they completed training. Table 4 shows the distribution of trainees by intake date, and type and length of training. TABLE 4: AMANTECH: NUMBER OF TRANINEES BY TRAINING DURATION AND SESSION Intake date Length of training (months) Total 3 6 12 18 January 20 14 -- -- -- 255 255 June 2014 -- -- 242 290 532 January 20 IS -- 177 355 -- 532 March 2015 -- -- 19 -- 19 June 2015 90 145 806 -- 1,04 1 Total 90 322 1,422 545 2,379 Sampling frame 90 322 628 545 1,573 The sampling frame for AMANTECH alumni includes all trainees in Table 4 except the 806 ind ividuals who enrolled in the 12 month program that started in June 2015. These trainees are expected to graduate in June 2016 and hence will not have graduated and had time to find jobs before data collection begins. The alumni sampling frame for AMANTECH thus consists of 1,573 individuals. TABLE 5: AMANTECH: NUMBER OF TRAINEES BY TRAINING DURATION AND TYPE OF TRAINING Course length (months) Trade/course Total 3 6 12 18 Auto -- -- 169 132 30 1 CAD -- 184 23 I 208 Electrical -- 44 273 232 549 Mechanical -- 94 151 180 425 Textiles 90 -- -- -- 90 Total 90 322 616 545 1,573 AMANTECH offers training in 13 skill areas, some of which it offers at different levels of intensity (duration). Table 5 collapses trainings into four bins for sampling since there are not enough observations by specific training type. The sample will include all of the three-month textile trainees (71 of whom are women and the only women trainees at AMANTECH) because they are directly comparable to UNDP trainees. The team will select equal numbers of trainees from the remaining four bins. stratifying by length of time since completing the training. PERFORM ASSIGNMENT WORK PLAN 8 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 55 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 56 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 57 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 58 I The table shell in Table 9 illustrates how the evaluation team may organize and present the data to answer question I a. TABLE 9: ILLUSTRATIVE TABLE SHELL FOR QUESTION lA AMANTECH UNDP Union Neighborhoo Neighborhood Neighborhood percentage percentage Town Council d ranking (within ranking (within beneficiaries beneficiaries town) city) from from neighborhood neighborhood Korangi Bilal Colony Korangi Nasir Colony Korangi Chakra Colony Add rows for additional towns/UCs To answer question I b, the evaluation will collect detailed demographic, socio-economic, and psychological information3 from the survey of alumni to establish a profile of the average trainee. lt will use these data to determine how many alumni (those recruited and selected into the program) satisfy the measurable definition of "marginalized/at-risk" the team will have established in consultation with USAID and the IPs. The analysis will thus determine whether recruiting and selection strategies are effectively engaging marginalized/at-risk youth. To answer question I c, the evaluation will calculate the cost effectiveness of each recruiting method by dividing the total cost of the recruiting method by the estimated4 number of successful applicants who reported responding to the recruitment method in the survey. The cost-effectiveness analysis will require financial data from implementing partners disaggregated by recruiting method. Group interviews with alumni will provide qualitative information on the reasons why some recruiting strategies were more effective than others. Table I 0 illustrates how the evaluation team may organize and present the data to answer question I c. 3 The data collected for the profile will reflect the way the team decides to operationalize the "marginalized/at-risk" concept. 4 The analysis will estimate the number of trainees who responded to each recruitment approach from data collected from the survey samples. PERFORM ASSIGNMENT WORK PLAN 12 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 59 TABLE 10: ILLUSTRATIVE TABLE SHELL FOR QUESTION IC I AMANTECH UNDP Number of Total Number of Total Method of trainees amount Cost trainees amount Cost recruitment reporting spent by effectiveness reporting [ ... ] spent by effectiveness [ ... ]as primary lP on (cost per as primary lP on (cost per recruiting recruiting applicant) recruiting recruiting applicant) method method method method Social mobilization SMS Print media Other methods Add rows for additional recruiting methods PERFORM ASSIGNMENT WORK PLAN 13 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 60 2. How is the program affecting individual trainees? a. To what extent are trainees obtaining new or better jobs relative to similar individuals who do not receive training? b. What effect is the program having on keeping youth away from violent groups/activities? Data Analysis The analysis will use two approaches to answer question 2a. First, it will compare the pre-training status of alumni to post-training outcomes. Second, it will compare post-training outcomes for alumni to the pre-training situation of current trainees immediately before these current trainees started training. The comparison matches groups of youth with comparable motivation, however these groups may be different along other unobservable baseline characteristics especially neighborhood or time-dependent factors; the only observable differences between the two groups should be related to any differences in selection criteria or that one group received training while the other did not. Table 11 illustrates how the team may organize and report data for the second approach to answering question 2a. TABLE 11: ILLUSTRATIVE TABLE SHELL FOR QUESTION 2A Current p-value of Current p-value of AMANT Difference difference UNDP Difference ECH AMANTECH of of alumni UNDP of difference trainees trainees of alumni (2) (I) and (2) (I) and (3) (4) (3) and (4) (3) and (4) (I) (2) Change in proportion employed Change in individual reported income Change in length of contract Proportion who's training matches job placement .. Add rows for add1t1onal 1nd1cators To answer question 2b, the evaluation will collect data in the surveys to understand how alumni and current trainees who have just started their training use their time. Comparing the two groups will allow the team to draw conclusions about whether alumni are engaged in more productive activities than they would have been otherwise (taking the pre-training situation of current trainees as the counterfactual). The analysis will disaggregate granular time use data into productive and leisure activities. The assumption is that increased productive or leisure activities will crowd o ut violent and extremist activities. The evaluation will also triangulate survey data with the opinions of lP social mobilization teams. PERFORM ASSIGNMENT WORK PLAN 14 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 61 Table 12 illustrates how the team may organize and report data to answer question 2b. TABLE 12: ILLUSTRATIVE TABLE SHELL FOR QUESTION 2B AMANTE Current Difference p-value of UNDP Current Difference p-value of CH AMANTECH difference UNDP alumni trainees of of alumni trainees of difference of (I) (2) (I) and (2) (I) and (2) (3) (4) (3) and (4) (3) and (4) Time spent in productive activities Time spent in leisure activities .. Add rows for additional md1cators 3. How, if at all, is the program affecting the targeted communities (towns/neighborhoods) at large? a. How, if at all, has the level of reported crime and violence changed in target communities? b. How, if at all, has the perception of insecurity changed in target communities? Data Analysis To answer question 3a, the evaluation will use neighborhood-level historic crime data to assess trends in reported violence throughout the intervention period. Neighborhoods with a higher proportion of youth engaged in the USAID-supported training (or similar workforce development activities) may be more likely to experience a decrease in violence and the analysis will explore this relationship.5 To answer question 3b, the analysis will also leverage qualitative information from key informant interviews with social mobilization teams to assess changes in perceptions of insecurity at the ne ighborhood level before the project and at the time of the evaluation. The surveys may also seek information about perceptions of insecurity before the project and at the time of the evaluation from surveyed youth. The analysis will create a composite indicator of perceptions of insecurity from the two data sources. The analysis will relate changes in this indicator to the proportion of neighborhood youth trained. Table 13 illustrates how the team may organize and present data to answer questions 3a and 3b. s This assumes a link between training and engaging in violent behavior, that the proportion of youth in a neighborhood who have undergone training is large enough to affect neighborhood dynamics. The analysis may also need to account for other (i.e., non USAID-funded) workforce development activities in the area. PERFORM ASSIGNMENT WORK PLAN IS I Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 62 TABLE 13: ILLUSTRATIVE TABLE SHELL FOR QUESTIONS 3A AND 3B AMANTECH AMANTECH pre- to UNDP pre- to post- number or UNDP number or proportion of post-project change in proportion of project change in crime Town target crime index (question target population index (question la) or population la) or perceptions of trained perceptions of security trained security (question lb) (question lb) Gulberg Town New Karachi Town Add rows for additional towns The analysis will use appropriate statistical tests to determine if there is a relationship between the number o r proportion of the target population trained and changes in the crime index. 4. What is the stakeholder involvement in the projects? a. What was the community's involvement in planning and implementing this project? b. What was the private sector's involvement in planning and implementing this project? c. What was the involvement of any other relevant stakeholder in planning and implementing this project? d. What, if any, cost-sharing has the project benefitted from? Data Analysis Data from key informant interviews with project stakeholders and lP staff will provide the qualitative data to answer questions 4a through 4c. To answer question 4d, the team will combine qualitative data from key informant interviews with lP staff with any quantitative data implementing partners are able to provide. Additionally the survey and group interviews will also yield information on trainees' views on the appropriateness of the cost share they are expected to pay. Question 5: Is the project scalable and can local resources can be leveraged to transition it out of donor dependence? Data Analysis Data from key informant interviews with local private sector donors will allow the evaluation to assess whether local resources can replace donor funding. Interviews with experts will inform the discussion on how projects have been able to transition out of donor dependence. Also, the survey and group interviews of alumni and current trainees will elicit the willingness of individuals to pay a higher cost share for training. The evaluation will also assess whether alumni support their parent institutions after completing training. Strengths and Limitations The proposed approach has the following strengths and limitations: • The key strength of the approach is its extensive use of both quantitative and qualitative data from multiple sources. lt is truly a mixed methods approach. Furthermore, answers to two of PERFORM ASSIGNMENT WORK PLAN 16 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 63 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 64 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 65 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 66 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 67 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 68 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 69 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 70 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 71 TABLE 16: ANTICIPATED ASSIGNMENT SCHEDULE AND LEVEL OF EFFORT Assignment Staffing LOE Location of Anticipated De live rable (s) Deliverable phase (days) activity schedule due date Team leader 16* Drafts of introduction and background chapters (team leader) Aug 8, 2016 Preparation Statistician .,.. Home base Aug 8-Aug 13 Draft answers to evaluation questions (all team (PERFORM Field researcher # I .,.. deliverable) Field researcher #2 7** members) Team leader 6 Statistician 6 TPW Field researcher #I 6 Karachi Aug 15-Aug 20 Draft instruments Aug 20, 2016 Data analyst #I 8 .. Data analyst #2 8 .. Team leader 6 Statistician 6 Training and Researcher #I 6 lslamabad I Aug 22-Aug 27 pretesting Researcher #2 6 Karachi Data analyst #I 6 Data analyst #2 6 Team leader 3 Statistician 3 Finalize tools Field researcher # I 3 lslamabad Aug 29-Aug 31 Draft data collection and analysis plans and final Aug 3 1, 2016 Field researcher #2 3 instruments Data analyst #I 3 Data analyst #2 3 Team leader 2 Data Statistician 2 lslamabad Sep 1-Sep 2 Data rehearsal Sep 2, 2016 rehearsal Field researcher # I 2 Field researcher #2 2 Fieldwork Team leader 20 Karachi Sep 5-Sep 27 Data collection completion report Sep 30, 2016 Logistics coordinator 20** • Includes days for international travel • Includes days for domestic travel PERFORM ASSIGNMENT WORK PLAN 25 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 72 Assignment Staffing phase Statistician Field researcher # I Field researcher # 2 Data analyst # I Data analyst # 2 Team leader Statistician Analysis Field researcher #I Field researcher #2 Data analyst #I Data analyst # 2 Team leader Reporting Technical editor * Includes 8 days of mternat1onal travel. ** Includes two day of local travel. PERFORM ASSIGNMENT WORK PLAN LOE Location of (days) activity 20 20 20 20 20 14 14 14 lslamabad 14 14 14 IS 5 Home base Anticipated Deliverable( s) Deliverable schedule due date Debriefing note outline to USAID/Pakistan O ct 13,2016 Sep 28-0ct I 3 Debriefing with USAID/Pakistan Oct 14, 20 16 Presentation to USAID/Pakistan & stakeholders TBD Draft report to USAID/Pakistan Nov 11, 2016 Draft one-page summary Nov 18, 2016 Comments on draft report returned by Nov 25,2016 Oct 17-Dec 19 USAID/Pakistan Comments on one-page summary returned by USAID/Pakistan Dec 2, 2016 Final report to USAID/Pakistan Dec 19,2016 Final one-page summary to USAID/Pakistan Dec 19,2016 26 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 73 LOE Summary by Position Status Position LOE (days) I Expat STTA Team leader 82 Local STTA Statistician 58 Local STTA Field researcher # I 58 Local STTA Field researcher #2 58 Local STTA Data analyst # I SI Local STTA Data analyst # I SI Local STTA Logistics coordinator 20 Home office Technical editor 5 Total LOE 385 PERFORM ASSIGNMENT WORK PLAN 27 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 74 COST ESTIMATE A break-down of costs by the four line items is below: Direct Labor Travel Other Direct Costs Subcontractor Grand Total PERFORM COR APPROVAL [COR will indicate approval by signing below or indicating "approval" by return email]. Contracting Officer's Representative (COR) Date , or designate PERFORM ASSIGNMENT WORK PLAN 28 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 75 Annex 3: Data Collection Instruments Questionnaire for Alumni Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 76 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 77 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 78 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 79 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 80 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 81 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 82 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 83 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 84 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 85 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 86 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 87 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 88 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 89 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 90 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 91 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 92 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 93 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 94 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 95 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 96 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 97 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 98 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 99 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 100 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 101 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 102 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 103 Questionnaire for Current Trainees Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 104 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 105 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 106 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 107 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 108 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 109 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 110 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 111 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 112 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 113 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 114 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 115 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 116 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 117 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 118 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 119 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 120 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 121 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 122 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 123 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 124 Group Interview Alumni Basic information sheet (Note: This sheet will be distributed to each participant before the interview and collected after the interview. Provide assistance to the participant if any of the information is missing) Name Course attended Age Current Residential Address District Town Name of Union Council Institution name Year graduated Contact number Email Participant number (1-4) *This table will be copied four times. Once for each participant Group Interview Alumni Instructions for interviewer: Properly introduce the participants-script provided below. Please make sure that you solicit permission for recording the interview from each participant. If a respondent declines to record the interview, then take detailed notes and use the backside of pages or additional pages if required to write your notes (use notebook if required). All the questions require answers unless stated otherwise in skip instructions. Please assign an alphanumeric code such as P1, P2, etc. to each participant. Each participant is expected to respond to each question. If a participant confirms the response from another participant, then input the code “Do” and “the participant number of the person whose response is confirmed by another participant”. For example, if participant number 3 (P3) confirms the response from participant number 1 (P1), then code participant number 3´s response by coding it as follows: Do P1. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 125 Introduction: Assalam-o-Alaikum (Greetings). Hope you are having a good day and thank you for the meeting. My name is ____________. I am a member of the team that is conducting field work for the midterm evaluation of youth workforce development projects funded by USAID and implemented by UNDP/Amantech in different parts of Karachi. The evaluation aims to determine whether the project under which you received training is on track to meet its training and employment targets, and whether we could bring about any improvements at this stage. We would like to hear your views about the project and your experience with it. The information that you share with us will only be used for this evaluation and will remain confidential. I. Does Participant 1 agree to participate in this interview and give permission to record? Yes / No II. Does Participant 2 agree to participate in this interview and give permission to record? Yes / No III. Does Participant 3 agree to participate in this interview and give permission to record? Yes / No IV. Does Participant 4 agree to participate in this interview and give permission to record? Yes / No Note: If all participants say yes, continue the interview. If no, try to motivate all of the participants by answering their questions and explaining the importance of recording the interview. Background information Q. 1. How was your experience with the training? P1 P2 P3 P4 Q. 2. Why did you choose your specific training course? P1 P2 P3 P4 Q. 3. How, if at all, has the training course that you received affected your life? What changes have occurred as a result of completing that training course that you were not doing previously? P1 P2 P3 P4 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 126 Q. 4. How did you learn about the training course that you completed? (Circle all that apply for each participant) Recruitment Strategy Participant 1 Participant 2 Participant 3 Participant 4 Radio show 1 1 1 1 Billboard 2 2 2 2 Internal announcements 3 3 3 3 SMS 4 4 4 4 Cable airing production 5 5 5 5 Social media 6 6 6 6 Newspaper ads 7 7 7 7 Promotion at sports events 8 8 8 8 Street theaters 9 9 9 9 Door to door activity 10 10 10 10 Open house 11 11 11 11 Refer a student campaign 12 12 12 12 Family, Friends, or Social Mobilizers 13 13 13 13 TV 14 14 14 14 Other (Specify): 15 15 15 15 Generally, what is the most influential method that would encourage youth like yourself to apply for such trainings? (Circle up to three responses for each participant) Recruitment Strategy Participant 1 Participant 2 Participant 3 Participant 4 Radio show 1 1 1 1 Billboard 2 2 2 2 Internal announcements 3 3 3 3 SMS 4 4 4 4 Cable airing production 5 5 5 5 Social media 6 6 6 6 Newspaper ads 7 7 7 7 Promotion at sports events 8 8 8 8 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 127 Recruitment Strategy Participant 1 Participant 2 Participant 3 Participant 4 Street theaters 9 9 9 9 Door to door activity 10 10 10 10 Open house 11 11 11 11 Refer a student campaign 12 12 12 12 Family, Friends, or Social Mobilizers 13 13 13 13 TV 14 14 14 14 Other (Specify): 15 15 15 15 Q. 4.1. Why do you think that this recruitment method is more effective than others? P1 P2 P3 P4 Q. 5. How were you selected for this training? What are your views about the selection process? P1 P2 P3 P4 Q. 6. In your opinion, what are the characteristics of a good job? P1 P2 P3 P4 Q. 7. For youth like you, what are the factors that determine how much you earn? P1 P2 P3 P4 Q. 8. If you had not completed your training course, then what do you think you would be doing now? Why? P1 P2 P3 P4 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 128 Q. 9. In your opinion, what are the skills that employers in your industry are looking for when hiring? P1 P2 P3 P4 Q. 9.1. What are the greatest challenges for young people like you to get the jobs that you want? P1 P2 P3 P4 *** UNEMPLOYED Ask the following questions only if the group is currently unemployed, otherwise skip to Q14. Q. 10. Why are you currently unemployed? Were you ever employed since you completed the training? P1 P2 P3 P4 Q. 10.1. If you were previously employed in a job related to your training, why did you leave that job? P1 P2 P3 P4 Q. 10.2. How is being unemployed affecting you and your household? P1 P2 P3 P4 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 129 EMPLOYED Ask the following questions only if the group is currently employed, otherwise skip to Q14. Q. 11. Which of the following best describes your current employment status? Daily wage worker, salaried employee, self-employed, or an unpaid family worker? # Paid employment (daily wage) Paid employment (salaried) Self￾employed Unpaid family work Participant 1 4 3 2 1 Participant 2 4 3 2 1 Participant 3 4 3 2 1 Participant 4 4 3 2 1 Q. 11.1. Please describe your current job? P1 P2 P3 P4 Q. 11.2. How did you end up in this occupation? This question is meant to probe how participants end up in different occupations, rather than how they obtained their current job P1 P2 P3 P4 Q. 12. How does your current job compare to any job that you were doing before the training? What changes came about due to the training? P1 P2 P3 P4 Q. 13. What kind of support, if any, are you receiving or have ever received from the [implementing partner responsible for placement]? P1 P2 P3 P4 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 130 Q. 14. Based on your experience, what can we improve in the recruiting, training, and placement process? P1 P2 P3 P4 Q. 15. In terms of employment, what would better encourage, attract, and retain youth in jobs? P1 P2 P3 P4 Q. 16. To what extent do you agree with the following statement: a project that focuses on vocational training and employment is the correct approach to keep marginalized and at-risk Karachi youth away from negative activities (especially crime and violent extremism)? Do you strongly agree, somewhat agree, somewhat disagree, or strongly agree, or neither agree nor disagree with the statement? # Strongly Somewhat agree Neither Somewhat disagree Strongly disagree Don’t know Participant 1 5 4 3 2 1 99 Participant 2 5 4 3 2 1 99 Participant 3 5 4 3 2 1 99 Participant 4 5 4 3 2 1 99 Q. 1.1. Why do you agree or disagree? Please explain in detail. P1 P2 P3 P4 Q. 17. Anything else that you would like to mention? P1 P2 P3 P4 Thank you for your time and patience. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 131 Group Interview Current Trainees Basic information sheet (Note: This sheet will be distributed to each participant before the interview and collected after the interview. Provide assistance to the participant if any of the information is missing) Name Course attended Age Current Residential Address District Town Name of Union Council Institution name Year graduated Contact number Email Participant number (1-4) *This table will be copied four times. Once for each participant Group Interview Current Trainees Instructions for interviewer: Properly introduce the participants-script provided below. Please make sure that you solicit permission for recording the interview from each participant. If a respondent declines to record the interview, then take detailed notes and use the backside of pages or additional pages if required to write your notes (use notebook if required). All the questions require answers unless stated otherwise in skip instructions. Please assign an alphanumeric code such as P1, P2, etc. to each participant. Each participant is expected to respond to each question. If a participant confirms the response from another participant, then input the code “Do” and “the participant number of the person whose response is confirmed by another participant”. For example, if participant number 3 (P3) confirms the response from participant number 1 (P1), then code participant number 3´s response by coding it as follows: Do P1. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 132 Introduction: Assalam-o-Alaikum (Greetings). Hope you are having a good day and thank you for the meeting. My name is ____________. I am a member of the team that is conducting field work for the midterm evaluation of a couple of youth workforce development projects funded by USAID and implemented by UNDP and Amantech in different parts of Karachi. This evaluation aims to determine whether we could bring about any improvements in training or job placements for youth like you. We would like to hear your views about the project. The information that you share with us will only be used for this evaluation and will remain confidential. I. Does Participant 1 agree to participate in this interview and give permission to record? Yes / No II. Does Participant 2 agree to participate in this interview and give permission to record? Yes / No III. Does Participant 3 agree to participate in this interview and give permission to record? Yes / No IV. Does Participant 4 agree to participate in this interview and give permission to record? Yes / No Note: If all participants say yes, continue the interview. If no, try to motivate all of the participants by answering their questions and explaining the importance of recording the interview. Background information Q. 1. How is your experience with the training so far? P1 P2 P3 P4 Q. 2. Why did you choose your specific training course? P1 P2 P3 P4 Q. 3. How, if at all, do you believe this training course will affect your life in the future? P1 P2 P3 P4 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 133 Q. 4. How did you learn about the training course? (Circle all that apply for each participant) Recruitment Strategy Participant 1 Participant 2 Participant 3 Participant 4 Radio show 1 1 1 1 Billboard 2 2 2 2 Internal announcements 3 3 3 3 SMS 4 4 4 4 Cable airing production 5 5 5 5 Social media 6 6 6 6 Newspaper ads 7 7 7 7 Promotion at sports events 8 8 8 8 Street theaters 9 9 9 9 Door to door activity 10 10 10 10 Open house 11 11 11 11 Refer a student campaign 12 12 12 12 Family, Friends, or Social Mobilizers 13 13 13 13 TV 14 14 14 14 Other (Specify): 15 15 15 15 Q. 5. Generally, what is the most influential method that would encourage youth like yourself to apply for such trainings? (Circle up to three responses for each participant) Recruitment Strategy Participant 1 Participant 2 Participant 3 Participant 4 Radio show 1 1 1 1 Billboard 2 2 2 2 Internal announcements 3 3 3 3 SMS 4 4 4 4 Cable airing production 5 5 5 5 Social media 6 6 6 6 Newspaper ads 7 7 7 7 Promotion at sports events 8 8 8 8 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 134 Recruitment Strategy Participant 1 Participant 2 Participant 3 Participant 4 Street theaters 9 9 9 9 Door to door activity 10 10 10 10 Open house 11 11 11 11 Refer a student campaign 12 12 12 12 Family, Friends, or Social Mobilizers 13 13 13 13 TV 14 14 14 14 Other (Specify): 15 15 15 15 Q. 5.1. Why do you think that this method is more effective than others? P1 P2 P3 P4 Q. 6. How were you selected for this training? What are your views about the selection process? P1 P2 P3 P4 Q. 7. In your opinion, what are the characteristics of a good job? P1 P2 P3 P4 Q. 8. For youth like you, what are the factors that determine how much you earn? P1 P2 P3 P4 Q. 9. In your opinion, what are the skills that employers in your industry are looking for when hiring? P1 P2 P3 P4 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 135 Q. 9.1. What are the greatest challenges for young people like you to get the jobs that you want? P1 P2 P3 P4 *** UNEMPLOYED Ask the following questions only if the group was unemployed before starting the training, otherwise skip to Q12. Q. 10. Why were you unemployed before the training? Were you ever employed? P1 P2 P3 P4 Q. 10.1. If you were previously ever employed in a job, why did you leave that job? P1 P2 P3 P4 Q. 10.2. How did being unemployed affect you and your household? P1 P2 P3 P4 EMPLOYED Ask the following questions only if the group was employed before starting the training, otherwise skip to Q12. Q. 11. Which of the following best describes your employment status before training? Daily wage worker, salaried employee, self-employed, or an unpaid family worker? # Paid employment (daily wage) Paid employment (salaried) Self￾employed Unpaid family work Participant 1 4 3 2 1 Participant 2 4 3 2 1 Participant 3 4 3 2 1 Participant 4 4 3 2 1 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 136 Q. 11.1. Could you please describe your current job? P1 P2 P3 P4 Q. 11.2. How did you end up in that occupation? This question is meant to probe how participants ended up in their occupations, rather than how they obtained their job. P1 P2 P3 P4 Q. 12. What kind of support, if any, will you receive from your training institute in getting a job after you complete your training? P1 P2 P3 P4 Q. 13. Based on your experience, what could be improved in the recruiting, training, and placement process? P1 P2 P3 P4 Q. 14. In terms of employment, what would better encourage, attract, and retain youth in jobs? P1 P2 P3 P4 Q. 15. To what extent do you agree with the following statement: a project that focuses on vocational training and employment is the correct approach to keep marginalized and at-risk Karachi youth away from negative activities (especially crime and violent extremism)? Do you strongly agree, somewhat agree, somewhat disagree, or strongly agree, or neither agree nor disagree with the statement? # Strongly Somewhat agree Neither Somewhat disagree Strongly disagree Don’t know Participant 1 5 4 3 2 1 99 Participant 2 5 4 3 2 1 99 Participant 3 5 4 3 2 1 99 Participant 4 5 4 3 2 1 99 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 137 Q. 15.1. Why do you agree or disagree? Please explain in detail. P1 P2 P3 P4 Q. 16. Anything else that you would like to mention? P1 P2 P3 P4 Thank you for your time and patience. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 138 Key Informant’s Interview Experts Instructions for Interviewer: Properly introduce yourself to the respondent – script provided below. The interviewer should have sufficient knowledge about the project before conducting this interview. Please make sure that you solicit permission for recording the interview. If the respondent declines, take detailed notes and use the back side of pages or additional pages if required to write your notes. All questions require answers unless stated otherwise in skip instructions. Please provide the respondent with a brief description of the projects prior to starting. Introduction: Assalam-o-Alaikum (Greetings). Hope you’re having a good day. Thank you for the meeting. My name is ____________. I am a member of the team that is conducting field work for the midterm evaluation of youth workforce development projects funded by USAID and implemented by UNDP and Amantech in different parts of Karachi. The evaluation aims to determine whether the project is on track to meet its training and employment targets, and whether we could bring about any improvements at this stage. We would like to hear your views about the project and general political economy conditions in the city. The information that you share with us will only be used for this evaluation and will remain confidential. You will not be quoted on any information you provide, without your prior written permission. A. Do I have your consent to proceed with this interview? Yes/No ______ If no, stop interview, but complete the respondent information table below. B. Do I have your permission to record this interview for our notes? Yes/No _______ Explain to the respondent that it will be very difficult for you to conduct the interview and take detailed notes simultaneously, if you don’t record the interview. C. Respondent information 1. Full name: 2. Designation: 3. Organization: 4. Department: 5. Address: 6. Contact:  At least one valid contact number (mobile or landline) is mandatory a. Phone: b. E-mail: Briefly introduce the two projects, their purpose, main activities, training areas, and job placement efforts. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 139 Q 1. How would you characterize marginalized or at-risk youth in Karachi? In your opinion, what are the principle reasons why youth become marginalized or at-risk? Q 2. Based upon the reasons that you have identified, which towns and neighborhoods would you consider to have the highest rates of marginalized or at-risk youth? Why? Q 2.1. Could you share reasons why youth could be more marginalized in Lyari, Korangi, and Sultanabad compared to other areas in Karachi? Q 2.2. Has the security, crime, economic, and political situation in these towns changed lately? Has the situation deteriorated or improved? Q 3. In your opinion, what are the best strategies to identify the most at-risk and marginalized youth in Karachi? Q 4. In your opinion, what type of role do marginalized or at risk youth usually play in their local communities in Karachi? Q 5. The Youth Workforce Development Projects support marginalized/at-risk youth by providing them with vocational training and supporting them in finding a job. To what extent do you agree with the following statement: a project that focuses on vocational training and employment is a correct approach to help marginalized and at-risk youth in Karachi to become more positive actors in their local communities? Do you strongly agree, somewhat agree, somewhat disagree, strongly disagree, or neither agree nor disagree with the statement? Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 140 Strongly agree Somewhat agree Neither agree nor disagree Somewhat disagree Strongly disagree Don’t know 5 4 3 2 1 99 Q 5.1. Please share reasons for your agreement or disagreement? Are you aware of any evidence supporting your views? Q 5.2. To what extent do you agree with the following statement: a project that focuses on vocational training and employment contributes to keep marginalized and at-risk Karachi youth away from negative activities (particularly crime and violent extremism)? Do you strongly agree, somewhat agree, somewhat disagree, or strongly agree, or neither agree nor disagree with the statement? Strongly agree Somewhat agree Neither agree nor disagree Somewhat disagree Strongly disagree Don’t know 5 4 3 2 1 99 Q 5.3. Please share reasons for your agreement or disagreement? Are you aware of any evidence supporting your views? Q 6. Considering the social, political, and economic situation in Karachi, in your opinion, how best could we engage the youth to ensure that they don’t get involved in negative activities, especially crime and violent extremism? Q 7. What other type of support, if any, do you believe that marginalized/at-risk youth should receive in order to facilitate their potential to become positive actors in their local communities? Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 141 Q 8. How could we ensure that marginalized or at-risk youth of Karachi remains gainfully employed? Q 9. What role, if any, do you believe that local community members, government, and the private sector should play in the support of marginalized/at-risk youth in Karachi? Q 10. What recommendations do you have on how best to ¨scale up¨ these projects in Karachi or other parts of the country so that other youth benefit from such initiatives? Which locations in Karachi would most benefit from such training and employment projects? Q 11. Under what conditions could the trainings continue without USAID funding? Who else could fund them? Q 12. In your opinion, how sustainable will the impact of employment training for marginalized/at-risk youth be in Karachi? What components can help in ensuring sustainability? Q 13. What sectors are most in need of skilled workers? Q 14. What technical skills are these sectors most in need of? Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 142 Q 15. What are the best strategies to identify youth to train in these skills that they can find employment? Q 16. Anything else that you would like to mention? Anything important that we have not yet discussed? Thank you for your time and patience. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 143 Key Informant Interview Government Officials Covers STEVTA, and any other government department that is already aware of and working with the projects, such as the Sindh P&DD. Instructions for Interviewer: Properly introduce yourself to the respondent – script provided below. The interviewer should have sufficient knowledge about the project before conducting this interview. Please make sure that you solicit permission for recording the interview. If the respondent declines, take detailed notes and use the back side of pages or additional pages if required to write your notes. All questions require answers unless stated otherwise in skip instructions. Please provide the respondent with a brief description of the projects prior to starting. Introduction: Assalam-o-Alaikum (Greetings) Hope you’re having a good day. Thank you for the meeting. My name is ____________. I am a member of the team that is conducting field work for the midterm evaluation of youth workforce development projects funded by USAID and implemented by UNDP and Amantech in different parts of Karachi. The evaluation aims to determine whether the projects are on track to meet training and employment targets, and whether we could bring about any improvements at this midterm stage. We would like to hear your views about the project(s) and its activities. The information that you share with us will only be used for this evaluation and will remain confidential. You will not be quoted on any information that you provide, without your prior written permission. A. Do I have your consent to proceed with this interview? Yes/No ______ If no, stop interview, but complete the respondent information table below. B. Do I have your permission to record this interview for our notes? Yes/No _______ Explain to the respondent that it will be very difficult for you to conduct the interview and take detailed notes simultaneously, if you don’t record the interview. C. Respondent information 1. Full name: 2. Designation: 3. Organization: 4. Department: 5. Address: 6. Contact:  At least one valid contact number (mobile or landline) is mandatory a. Phone: b. E-mail: Briefly introduce the two projects, their purpose, main activities, training areas, and job placement efforts. Q. 1. How do you see the current economic and political conditions in Karachi? How are these conditions Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 144 affection the youth? Q. 2. Do you believe that certain localities in Karachi are more affected in terms of negative impact on youth? Which areas are more affected and how? Q. 3. What steps is the government taking in Karachi to productively engage and train youth, create employment opportunities, and save youth from getting into crime and violence? Are any major policies or strategies being implemented or planned for the near future? Q. 4. What sectors are most in need of skilled workers in Karachi? Q. 5. What technical skills are these sectors most in need of? Q. 6. What are the best strategies to identify youth to train in these skills that they can find employment? Q. 7. What are the key challenges and opportunities on the demand side of the labor force? Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 145 Q. 8. What are the key challenges and opportunities on the supply side of the labor force? Q. 9. What is the nature of your department / organization’s involvement in these projects that UNDP / Amantech is implementing? Q. 10. What is the best way to engage the government in such projects? Q. 11. What is the best way to engage the private sector in such projects? Q. 12. What is the best way to engage the community in such projects? Q. 13. How likely would it be for the training to continue without USAID funding? Under what conditions could the trainings continue without this funding? Who else could fund them? Could the government share costs in anyway? Q. 14. In your opinion, what is the best approach for scaling such projects? Could the government scale such projects on its own, if yes how? If no, why not? What conditions would need to be in place for such projects to scale? Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 146 Q. 15. What sort of impact do you think such projects have on the trainees and their households? Do such projects contribute to changes at the neighborhood level? How? Q. 16. What are the most important lessons that you have learned from such projects? Lessons that we could apply to these projects. Please share your thoughts on each of the following aspects: Q. 16.1. Identification of trainees, recruitment, and selection Q. 16.2. Training Q. 16.3. Placement in jobs, and follow-up Q. 17. To what extent do you agree with the following statement: a project that focuses on vocational training and employment is the correct approach to keep marginalized and at-risk Karachi youth away from negative activities (especially crime and violence)? Do you strongly agree, somewhat agree, somewhat disagree, or strongly agree, or neither agree nor disagree with the statement? Strongly agree Somewhat agree Neither agree nor disagree Somewhat disagree Strongly disagree Don’t know 5 4 3 2 1 99 Q. 17.1. Why do you agree or disagree? Q. 18. Anything else that you would like to mention? Anything important that we have not discussed? Thank you for your time and patience Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 147 Key Informant Interview Industry Includes private sector owners or managers, representatives of trade associations. Some of the private sector owners or managers should also be from amongst the training partners. Instructions for Interviewer: Properly introduce yourself to the respondent – script provided below. The interviewer should have sufficient knowledge about the project before conducting this interview. Please make sure that you solicit permission for recording the interview. If the respondent declines, take detailed notes and use the back side of pages or additional pages if required to write your notes. All questions require answers unless stated otherwise in skip instructions. Please provide the respondent with a brief description of the projects prior to starting. Introduction: Assalam-o-Alaikum (Greetings). Hope you’re having a good day. Thank you for the meeting. My name is ____________. I am a member of the team that is conducting field work for the midterm evaluation of youth workforce development projects funded by USAID and implemented by UNDP and Amantech in different parts of Karachi. The evaluation aims to determine whether the project is on track to meet its training and employment targets, and whether we could bring about any improvements at this stage. We would like to hear your views about the project and general requirements and conditions of employment in your industry. The information that you share with us will only be used for this evaluation and will remain confidential. You will not be quoted on any information you provide, without your prior written permission. A. Do I have your consent to proceed with this interview? Yes/No ______ If no, stop interview, but complete the respondent information table below. B. Do I have your permission to record this interview for our notes? Yes/No _______ Explain to the respondent that it will be very difficult for you to conduct the interview and take detailed notes simultaneously, if you don’t record the interview. Briefly introduce the two projects, their purpose, main activities, training areas, and job placement efforts. C. Respondent information 1. Full name: 2. Designation: 3. Organization: 4. Department: 5. Address: 6. Contact:  At least one valid contact number (mobile or landline) is mandatory a. Phone: b. E-mail: Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 148 Q. 1. Tell us about your business. What are your main products or services? Q. 2. What sorts of technical skills does your business require the most in producing or delivering your products or services? Q. 3. What kind of skilled workforce does your sector require the most? Q. 4. What types of contracts, if any, do you use when hiring these new employees? Why do you use them? Q. 5. How do firms like yours in this sector, recruit the type of skilled workers that are needed the most? (Circle all that apply) i. Job expo ii. Newspaper advertisement iii. Web advertisement iv. HR recruitment firms v. Head hunting vi. Through relevant universities or training institutes vii. All of the above viii. Other (specify)________________________ Q. 5.1. Which methods are most effective in finding and recruiting skilled workers, and why? Q. 6. What kinds of institutions or organizations do you usually contact to recruit new employees with the skill Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 149 sets that you are looking for? Why? Q. 7. Would you rely more on internal trainings for your workforce in these skills, or on the trainings and courses offered by public and private sector training institutes? Why? If the firm is working with the Youth Workforce Development Projects that are being implemented by AMANTECH/UNDP or any of its training institutes continue interview. Otherwise, skip to Q18. Q. 8. Could you please describe what type of collaboration, if any, your firm has had with the AMANTECH or UNDP program in the following areas? If you have not had any collaboration please say so. Q. 8.1. Recruitment Q. 8.2. Training Q. 8.3. Job Matching Q. 8.4. Job Placement Q. 8.5. Others (Specify ________________) Q. 9. How is your firm’s collaboration with AmanTech/UNDP working so far? What are the main results? Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 150 Q. 10. How could the collaboration between your firm and AmanTech/UNDP be improved? What is missing? What is needed to improve the entire supply chain from recruiting to working? Q. 11. Are there any other training institutes besides Amantech and UNDP that you regularly work with? Yes / No: ___________ Q. 12. Why do you work with them? How do they compare? Q. 13. What recommendations do you have on how best to scale up the youth workforce development projects in other parts of Karachi, or elsewhere in Pakistan? What will be the key factors or requirements? Q. 14. What sort of problems, if any, have you had with the retention of YWFD graduates you have recruited? Q. 15. How would you evaluate the YWFD project’s graduates you have hired in the following areas? Are you highly satisfied, somewhat satisfied, unsatisfied, highly unsatisfied, or do you have no opinion? HS SS NO SU HU Technical capacity 1 2 3 4 5 Practical skills 1 2 3 4 5 Problem solving skills Repair/ maintenance/ 1 2 3 4 5 Behavior 1 2 3 4 5 Communication skills/ report writing 1 2 3 4 5 Safety consciousness 1 2 3 4 5 Quality assurance 1 2 3 4 5 HS=Highly satisfied, S=Somewhat satisfied, NO=No opinion, SU=Somewhat unsatisfied, HU=Highly unsatisfied Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 151 Q. 15.1. Do you know of any examples of really good or really bad vocational training graduates from these projects that have worked for you? Q. 16. How does the on-the-job performance of graduates from these partner training centers compare to other new recruits whom you have hired? In what ways are they similar or different to other new recruits that you have recently hired? ____________________________________________________________________________________ ____________________________________________________________________________________ Q. 17. In your opinion, how will the private sector, local communities and youth benefit from the type of projects that AMANTECH/UNDP is implementing which focuses on preparing marginalized/at-risk youth for gainful employment? Q. 18. What sectors are most in need of skilled workers? Q. 19. What technical skills are these sectors most in need of? Q. 20. What are the best strategies to identify youth to train in these skills so that they may get good jobs? Q. 21. To what extent do you agree with the following statement: a project that focuses on vocational training and employment is the correct approach to keep marginalized and at-risk Karachi youth away from negative activities (especially crime and violent extremism)? Do you strongly agree, somewhat agree, somewhat disagree, or strongly agree, or neither agree nor disagree with the statement? Strongly agree Somewhat agree Neither agree nor disagree Somewhat disagree Strongly disagree Don’t know 5 4 3 2 1 99 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 152 Q. 21.1. What are your reasons for agreement or disagreement? Q. 22. Anything else that you would like to mention? Anything important that we may have missed? Thank you for your time and patience. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 153 Key Informant Interview Implementing Partners Implementing partners include UNDP, AMANTECH, and other training partners. Instructions for Interviewer: Properly introduce yourself to the respondent – script provided below. Please make sure that you solicit permission for recording the interview. If the respondent declines, take detailed notes and use the back side of pages or additional pages if required to write your notes. All questions require answers unless stated otherwise in skip instructions. Introduction: Assalam-o-Alaikum (Greetings). Hope you’re having a good day. Thank you for the meeting. My name is ____________. I am a member of the team that is conducting field work for the midterm evaluation of youth workforce development projects funded by USAID and implemented by your organization in different parts of Karachi. The evaluation aims to determine whether the project you are implementing is on track to meet its training and employment targets, and whether we could bring about any improvements at this midterm stage. We would like to hear your views about the project(s) and its activities. The information that you share with us will only be used for this evaluation and will remain confidential. A. Do I have your consent to proceed with this interview? Yes/No ______ If no, stop interview, but complete the respondent information table below. B. Do I have your permission to record this interview for our notes? Yes/No _______ Explain to the respondent that it will be very difficult for you to conduct the interview and take detailed notes simultaneously, if you don’t record the interview. C. Respondent information 1. Full name: 2. Designation: 3. Organization: 4. Department: 5. Address: 6. Contact:  At least one valid contact number (mobile or landline) is mandatory a. Phone: b. E-mail: Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 154 Q. 1. Please provide us with a brief overview of the youth workforce development project that your organization is implementing. (Project background, your role, objectives, aims, targets) Q. 2. What is the profile and location of your target group for this program? How do you define your target group? Q. 3. What challenges, if any, have you faced in achieving the targets so far? Q. 3.1. What strategies, if any, have you developed and applied to achieve the targets? Q. 4. In your opinion, how successful have you been in recruiting the target group for these projects? Would you say that you have been successful, partially successful, or unsuccessful? Successful Partially successful Unsuccessful Don’t know 3 2 1 99 Q. 4.1. Why do you think the recruiting strategy was successful or unsuccessful? Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 155 Q. 5. Please describe the overall strategy that your organization has used to inform potential candidates about the training, and encourage them to apply (recruitment strategy). (This question is about the overall recruitment strategy. Specific recruitment activities or methods are covered in the following questions.) Q. 5.1. Why did you choose this strategy? Q. 5.2. In what ways, if any, has your recruitment strategy evolved over time? If so, why has it changed? Q. 5.3. What recruitment activities or methods have you implemented under this strategy? Please state in order of priority from most used to least. Q. 5.4. Have you matched recruitment activities to different communities? If yes, how? If not, why not? Please give specific examples by community. Q. 5.5. Based upon your lessons learned, which recruitment activities have been the most cost-effective at recruiting marginalized or at-risk youth? Why do you think they have been effective? Q. 5.6. What kinds of activities have not been cost-effective at recruiting marginalized or at￾risk youth? Why do you think they are not effective? Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 156 Q. 6. After recruiting, selecting and matching youth to a vocational training program, about what percentage of the participants have dropped out of the program? (Don’t require a precise number) ___________% Q. 6.1. In your opinion, what have been the principle reasons why participants drop out of the program? Q. 6.2. What mitigation efforts, if any, have you introduced to reduce trainee drop outs? To what extent have these mitigation efforts been effective? Q. 7. How do you place your graduate trainees into jobs? What is the process? Q. 8. In what ways, if any, do you follow up on your graduates, whether employed or unemployed? For how long do you follow up with alumni? Q. 9. With respect to this project, what is the nature of collaboration, if any, that exists between your institution and the private sector? Q. 10. With respect to this project, what is the nature of collaboration, if any, that exists between your institution and the communities with which you are working? What are the pros and cons of collaborating with communities on such projects? Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 157 Q. 10.1. In what ways, if any, have you engaged community members such as parents, social workers, community members, during the program review, planning and implementation? Q. 10.2. How can the involvement of community members be improved in the future? Q. 11. With respect to this project, what is the nature of collaboration, if any, with relevant government departments or officials? What are the pros and cons of collaborating with the government on such projects? Q. 12. Are there any other stakeholder groups that you have engaged in the implementation of this project? If yes, which stakeholders and what has been the nature of the collaboration? What are the pros and cons of collaborating with these other stakeholders? Q. 13. What kind of cost-sharing arrangements, if any, have you made with the private sector thus far in the project? Please share some examples. Q. 14. Imagine that USAID funding for the project stopped. How likely would it be for the training to continue without USAID funding? Would it be very likely, somewhat likely, neither likely nor unlikely, somewhat unlikely, or highly unlikely? Very likely Somewhat likely Neither likely nor unlikely Somewhat unlikely Highly unlikely Don’t know 5 4 3 2 1 99 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 158 Q. 14.1. Please share some reasons to elaborate on your answer above. Q. 15. Under what conditions could the trainings continue without this funding? Who else could fund them? Q. 16. What recommendations do you have on how best to scale-up the project in Karachi or other parts of the country? Which recruiting and training strategies should be utilized in a larger-scale program? What is your strongest component? Q. 17. What do you consider to be the two or three most important lessons that you have learned from this project so far? Q. 18. Based on these lessons learned, what recommendations would you propose to further improve the outcomes of this project? Q. 19. What sectors are most in need of skilled workers? Q. 20. What technical skills are these sectors most in need of? Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 159 Q. 21. What are the best strategies to identify youth to train in these skills that they can find employment? Q. 22. Are there any characteristics that separate a candidate who will likely complete training and get a job vs those who will not utilize the training properly? Q. 23. To what extent do you agree with the following statement: a project that focuses on vocational training and employment is the correct approach to keep marginalized and at-risk Karachi youth away from negative activities? Do you strongly agree, somewhat agree, somewhat disagree, or strongly agree, or neither agree nor disagree with the statement? Strongly agree Somewhat agree Neither agree nor disagree Somewhat disagree Strongly disagree Don’t know 5 4 3 2 1 99 Q. 23.1. Why do you think that? Q. 24. Anything else that you would like to mention? Anything important that we have not discussed? Thank you again for your time and patience Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 160 Key Informant Interview Social Mobilizers Instructions for Interviewer: Properly introduce yourself to the respondent – script provided below. Please make sure that you solicit permission for recording the interview. If the respondent declines, take detailed notes and use the back side of pages or additional pages if required to write your notes. All questions require answers unless stated otherwise in skip instructions. Introduction: Assalam-o-Alaikum (Greetings). Hope you’re having a good day. Thank you for the meeting. My name is ____________. I am a member of the team that is conducting field work for the midterm evaluation of youth workforce development projects funded by USAID and implemented by UNDP and Amantech in different parts of Karachi. The evaluation aims to determine whether the projects are on track, and whether we could bring about any improvements. We would like to hear your views about the project that you are involved in. The information that you share with us will only be used for this evaluation and will remain confidential. You will not be quoted on any information. A. Do I have your consent to proceed with this interview? Yes/No ______ If no, stop interview, but complete the respondent information table below. B. Do I have your permission to record this interview for our notes? Yes/No _______ Explain to the respondent that it will be very difficult for you to conduct the interview and take detailed notes simultaneously, if you don’t record the interview. C. Respondent information 1. Full name: 2. Designation: 3. Organization: 4. Department: 5. Address: 6. Contact:  At least one valid contact number (mobile or landline) is mandatory a. Phone: b. E-mail: Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 161 Q 1. Please share a description of the local community where you have conducted social mobilization activities for this project, emphasizing on the following aspects: Type of description Description 1.1 Socio-economic 1.2 Security and political 1.3 Ethnic and cultural 1.4 Other (Specify): ____________________ Q 2. Could you please describe the planning, if any, that is usually involved in selecting geographic areas and for your mobilization activities for this project? Q 3. Please provide a description of the activities that you implement at the community level for this project? Additionally, which population are each of these activities targeting? Description of activities Targeted population Q 4. To what extent have each of the following groups played a role in your mobilization efforts i.e. identifying and encouraging youth to apply for the trainings? Have they had a strong role, a weak role, or no role at all in mobilization efforts? Stakeholder Strong role Weak role No role Don’t know Community 3 2 1 99 Current trainees 3 2 1 99 Graduates 3 2 1 99 Q 4.1. Has their participation been helpful or not? Ask if at least one group played a strong or weak role in Q4 Q 4.2. How could these groups further support your mobilization activities? Ask if at least one group played a strong or weak role in Q4 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 162 Q 5. How do you identify the communities and the marginalized or at-risk youth that you need to mobilize for receiving training under this project? Q 6. What criteria do you apply to select the neighborhoods for selecting youth? Q 7. What criteria do you apply for selecting youth in these neighborhoods? Q 8. What are the principle challenges you have encountered in your mobilization activities at the community and at the individual levels? 8.1 At the community level 8.2 At the individual level Q 9. Since the initiation of this project, what kind of changes, if any, have you noticed in the neighborhoods where you are working? Why have these changes occurred? Q 10. In what ways, if any, do the project beneficiaries (trainees) contribute to the situation in their households? Q 11. In what ways, if any, do the project beneficiaries (trainees) contribute to the situation in their neighborhoods? Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 163 Q 12. Have you observed any significant changes in the behaviors or life style of the youth who have received trainings, as compared to those that do not? Yes No (Skip to Q13) Don’t know 2 1 99 Q 12.1. Could you please describe the changes you have observed? Q 13. In your opinion, how successful have the recruitment activities been in engaging marginalized or at-risk youth? Would you say they have been successful, somewhat successful, somewhat unsuccessful, or unsuccessful? Successful Somewhat successful Somewhat unsuccessful Unsuccessful Don’t know 4 3 2 1 99 Q 13.1. Why do you think they were successful or unsuccessful? Q 14. What suggestions or recommendations do you have for improving the project’s mobilization activities? Q 15. In your opinion, how effective is this youth workforce development project been at engaging participants in employment activities? Has it been very effective, somewhat effective, somewhat ineffective, very ineffective, or neither effective nor ineffective? Very effective Somewhat effective Neither effective nor ineffective Somewhat ineffective Very ineffective Don’t know 5 4 3 2 1 99 Q 15.1. Why do you think it was effective or ineffective? Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 164 Q 16. In your opinion, how effective has this project been at contributing to the economic wellbeing of participant’s household and neighborhood? Has it been very effective, somewhat effective, somewhat ineffective, very ineffective, or neither effective nor ineffective? Very effective Somewhat effective Neither effective nor ineffective Somewhat ineffective Very ineffective Don’t know 5 4 3 2 1 99 Q 16.1. Why do you think it was effective or ineffective in this regard? Q 17. To what extent are participants promoting the project to their family, friends or community members? Are they promoting it strongly, promoting it weakly, or not promoting it at all? Strongly promoting Weakly promoting Not promoting Don’t know 3 2 1 99 Q 17.1. If they are, how are they promoting the project? If they are not, why not? Q 18. Anything else that you would like to mention? Anything important that we have not yet discussed? Thank you for your time and patience. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 165 Annex 4: Sampling The evaluation assessed the impact of training by comparing the before and after training status of alumni as well as the pretraining status of current trainees to the posttraining status of alumni, where the pretraining status of current trainees represented the counterfactual, i.e., what the current status of alumni would have been had they not participated in training. Evaluators designed the survey sample to be able to detect a 10 percentage point increase in the proportion of trainees employed because of training. To test this hypothesis with 95 percent confidence and 90 percent power, the sample sizes required were 380 and 281 for current trainees and alumni, respectively. However, a high nonresponse rate was expected considering that, in the absence of government permissions for fieldwork, trainees would be contacted through mobile phone numbers provided by Amantech and would be invited to participate in interviews at the training facility. The team therefore decided to conduct 410 surveys of alumni and 490 of current trainees—although the actual number of surveys completed for alumni was much lower than planned. Table 9 shows the number of surveys required, planned, and completed. TABLE 9: SURVEYS REQUIRED, PLANNED, AND COMPLETED Respondent Type Total Number of Trainees Number of Surveys Required Planned Completed Current trainees 2,365 380 490 491 Alumni 742 281 410 278 Total 3,107 661 900 769 Amantech provided data on trainees’ sex, town of residence, training course attended, and contact number. The list served as the sampling frame from which the evaluation team drew the sample of alumni and current trainees. The team selected simple random samples stratified by sex and town of residence. The sample was allocated across strata to ensure the appropriate presentation of each stratum in the sample. The analysis used design weights to remove the possible bias introduced due to the nonproportional allocation of the sample across strata. In addition to the preselected sample of 490 trainees and 410 alumni, the team also randomly selected replacement samples half the size of the original samples for each stratum. Table 10 summarizes characteristics of survey respondents. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 166 TABLE 10: PROFILE OF SURVEY RESPONDENTS Town (n) Age (average men) Age (average women) Education (% matric & below – men) Education (% matric & below – women) education Received religious (% men) Received religious education (% women) Never married (% men) Never married (% women Household size (mean) Number of dependents (male respondents) Number of dependents (female respondents) Mother tongue - Urdu Mother tongue - Sindhi Mother tongue - Pashto Mother tongue – Punjabi & Siraiki Years Years % % % % % % # # # % % % % Baldia (47) 21.9 - 79.0 - 12.2 - 89.9 - 7.3 2.0 - 35.4 2.0 14.1 28.1 Bin Qasim (18) 22.8 25.0 74.5 100 25.0 0 68.1 100 7.7 1.0 0 34.9 12.1 - 20.7 Gadap (12) 22.1 - 100 - 31.0 - 91.3 - 7.0 2.8 - 73.8 - 8.7 - Gulberg (23) 23.2 - 56.3 - 17.4 - 73.1 - 7.1 1.5 - 87.4 - - 7.7 Gulshan (30) 23.1 26.0 83.5 - 25.9 0 86.7 100 6.2 1.4 0 59.4 11.3 9.6 16.3 Jamshed (40) 22.4 - 86.5 - 18.2 - 80.8 - 6.6 2.0 - 63.6 7.9 4.5 3.7 Kiamri (27) 21.4 - 81.5 - 12.4 - 84.4 - 8.2 1.1 - 15.9 6.4 37.6 5.7 Korangi (141) 20.3 25.1 91.9 79.2 16.8 21.4 94.5 73.7 7.5 1.3 1.7 73.6 2.2 6.5 6.6 Landhi (64) 20.4 23.4 86.7 - 6.0 15.0 98.6 90.0 6.7 1.4 1.1 65.0 3.4 9.2 9.3 Liaquatabad (43) 21.2 - 81.6 - 12.2 - 93.7 - 6.1 1.3 - 90.7 - - 5.2 Lyari (27) 21.0 - 92.1 - 14.6 - 100 - 7.4 0.4 - 36.5 11.0 3.7 15.0 Malir (81) 20.5 23.4 82.9 100 20.6 0 97.7 87.5 7.1 1.3 0.1 63.9 3.6 9.4 12.1 New Karachi (38) 21.6 - 85.9 - 15.6 - 82.8 - 6.3 1.9 - 84.3 6.2 - 7.9 North Nazimabad (35) 21.1 - 54.0 - 9.9 - 80.2 - 9.0 1.0 - 94.9 - 2.6 2.6 Orangi (37) 20.4 - 78.4 - 19.6 - 98.3 - 7.5 1.3 - 78.8 - 6.5 8.2 SITE (27) 22.5 - 71.1 - 10.8 - 89.2 - 8.7 2.6 - 44.1 4.4 23.6 10.8 Saddar (48) 22.4 29 78.7 0 18.8 0 80.6 100 7.0 2.0 4.0 47.5 2.0 16.2 24.2 Shah Faisal (29) 21.5 27 88.7 100 0 0 89.2 0 6.4 1.0 4.0 67.4 4.0 - 24.5 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 167 Annex 5: Composite Marginalization Index To identify marginalized/at-risk towns, the evaluation team developed a composite marginalization index (CMI) using town level information on 19 indicators organized into 3 dimensions: actual crimes that occurred over the past year, trainees’ perceptions of insecurity in their neighborhoods, and other socioeconomic indicators. The team gathered crime data for two randomly selected dates in each month from July 2014 to June 2015 from the daily Express Tribune newspaper. The team obtained perceptions of insecurity and other socioeconomic indicators from the surveys of alumni and current trainees. Crime data on 18 towns of interest included data on murders, shootings, bomb attacks, target killings, deaths, robbery, and other crimes. For perceptions of insecurity, the surveys of trainees (current trainees and alumni from UNDP and Amantech) asked a set of questions on the levels of crime and violence, political parties’ and religious groups’ activities in their neighborhoods, and the effects of crime and violence on their households. In addition, the survey asked respondents about their perceptions of poverty in their neighborhoods, the availability of income-generating opportunities, and their satisfaction with basic government services. For other socioeconomic indicators, the evaluators derived measures of educational attainment, availability of economic opportunities in neighborhoods, poverty scores, and average household income from the surveys. Details and definitions of crimes and other indicators used in the construction of composite marginalization index are given in Table 11. The composite marginalization index follows a nested weighting structure that applies equal weights across dimensions and equal weights for each indicator within a dimension. The weights assigned to each indicator are listed in Table 11. Considering variation in the nature and unit of measurement, the team calculated a relative number for each indicator using the formula given below; these relative numbers (instead of actual numbers) were used to calculate the value of the composite marginalization index for each town. 𝑅𝑖𝑗 = 𝑋𝑖𝑗 − 𝑋𝑖(𝑚𝑖𝑛) 𝑋𝑖(𝑚𝑎𝑥) − 𝑋𝑖(𝑚𝑖𝑛) Where 𝑅𝑖𝑗 = Relative value of indictor i for town j 𝑋𝑖𝑗 = Actual value of indictor i for town j 𝑋𝑖(𝑚𝑖𝑛) = Minimum value of indictor i for all towns 𝑋𝑖(𝑚𝑎𝑥) = Maximum value of indictor i for all towns Table 11 defines indicators and groups them into three dimensions (crime, perception of insecurity, and other socioeconomic indicators). The team assigned a weight to each indicator, and the CMI for each town is the weighted sum of the relative numbers. Each dimension accounts for one-third of the CMI. Table 12 provides the calculation and CMI for each town. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 168 TABLE 11: DIMENSIONS, INDICATORS, DEFINITIONS, AND WEIGHTS FOR THE CMI Dimension Indictor Definition Weight Crime Murder Number of deaths due to shooting by criminals or as a result of personal enmity 1/21 Shooting Number of people injured in shootings/crossfire 1/21 Bomb attack Number of deaths due to explosives, e.g., grenades, vehicle bombs, etc. 1/21 Targeted attack Number of deaths or injuries due to targeted attacks on individuals 1/21 Deaths Number of deaths that do not fit any of the categories above. Examples include suicides, unexplained deaths, drug overdose, criminals killed by Rangers, etc. 1/21 Robbery Number of robberies 1/21 Other Number of any other crime reported that does not fit the categories above but is listed in the crime and misdemeanor section of the Express Tribune, e.g., injured during a brawl, weapons/drugs confiscated, etc. 1/21 Perception of insecurity Effects of political parties’ activity Percentage of respondents who think political parties’ activities in their neighborhood negatively affect their households 1/21 Effects of religious groups’ activity Percentage of respondents who think religious groups’ activities in their neighborhood negatively affect their households 1/21 Level of crime and violence Percentage of respondents who reported “a lot of crime and violence” in their neighborhood 1/21 Effects of crime and violence Percentage of respondents who reported crime and violence “largely” affected their neighborhood 1/21 Level of poverty Percentage of respondents who think level of poverty in their neighborhood is “high or very high” 1/21 Lack of income￾generating opportunities Percentage of respondents who think in their neighborhood there are “no opportunities” to generate income 1/21 Satisfaction with government services Percentage of respondents who were “mostly or very dissatisfied” with government services in their neighborhood 1/21 Other socio￾economic indictors Educational attainment Percentage of respondents with “middle or low grades” before starting training 1/15 Economically inactive population Percentage of respondents who were “not in labor force” before starting training 1/15 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 169 Dimension Indictor Definition Weight Unemployment rate Percentage of respondents who were “unemployed” before starting training 1/15 Poverty score Each town’s average household’s poverty score gap from 100 (i.e., 100 – town’s average household’s poverty score) 1/15 Average income Each town’s average household’s monthly income gap from 40,000 (i.e., 40,000 – town’s average household’s monthly income) 1/15 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 170 TABLE 12: CMI CALCULATION FOR ALL 18 TOWNS Towns Crimes Perception of Insecurity Other Socio-Economic Indicators CMI Murder Shooting Bomb Attacks Targeted Attack Deaths Robbery Other Crimes Effects of Political Parties’ Activity Effects of Religious Groups’ Activity Level of Crime and Violence Effects of Crime and Violence Level of Poverty Lack of Income-Generating Opportunities Satisfaction with Government Services Below or Equal to Middle Economically Not Active Unemployed Poverty Score Median Household Income # # # # # # # % % % % % % % % % % Scale Rs. Liyari 3 11 1 0 15 0 3 8.0 1.4 12.8 32.8 40.6 47.4 66.7 10.6 35.4 34.3 75.88 20000 0.5760 Landhi 6 2 1 0 1 2 9 15.8 5.2 8.4 16.1 28.6 44.5 49.2 12.6 33.7 18.5 73.28 21800 0.5186 Orangi 3 3 4 5 4 1 1 10.1 4.7 7.5 12.9 26.9 34.0 60.3 3.4 40.9 18.1 79.24 26000 0.4740 Korangi 2 12 0 3 1 1 0 16.8 6.3 5.6 8.5 26.3 43.6 48.5 8.3 43.5 27.3 81.01 25000 0.4550 Jamshed 1 2 0 1 4 1 6 17.7 8.4 3.5 8.8 19.0 39.1 51.9 13.5 46.5 23.1 81.99 25000 0.4294 Baldia 2 3 0 0 4 0 0 24.0 13.5 3.9 42.9 23.3 39.7 79.9 7.6 39.0 12.7 77.27 31500 0.4098 Liaquatabad 4 2 0 0 4 1 0 42.3 0.0 11.0 9.1 29.5 41.7 51.2 2.7 35.8 27.7 85.15 29000 0.3957 Bin Qasim 0 4 0 0 0 0 0 19.6 6.1 9.8 0.0 42.8 36.4 56.9 8.3 23.9 26.9 63.40 27300 0.3947 North Nazimabad 3 3 0 1 0 2 3 10.6 3.2 15.4 20.6 17.3 39.9 32.4 5.6 52.0 18.9 82.32 36000 0.3825 Saddar 3 4 0 0 4 2 0 18.9 9.7 5.5 9.1 20.8 49.7 32.6 4.5 40.5 22.9 83.35 32000 0.3778 New Karachi 2 6 0 0 1 2 1 28.6 5.5 10.9 6.9 20.7 44.6 60.6 4.1 36.9 12.7 85.99 30000 0.3764 Kiamri 0 0 0 0 2 0 0 4.0 0.0 0.0 3.8 43.1 29.3 58.4 23.9 59.6 19.8 68.15 28000 0.3741 Shah Faisal 1 1 0 0 1 0 0 24.6 11.7 10.6 19.9 12.3 25.4 54.8 1.9 45.1 30.5 83.70 32000 0.3491 SITE 1 0 1 1 3 1 0 8.3 0.0 4.3 8.0 36.3 34.9 57.1 0.0 36.3 25.4 75.75 30000 0.3348 Gulshan 1 1 1 2 1 3 2 10.6 2.8 0.0 3.3 19.9 42.0 38.6 3.1 47.8 15.7 85.74 30000 0.3324 Malir 1 2 0 1 1 1 2 10.1 4.6 5.1 9.9 26.5 35.8 52.9 0.5 42.5 11.7 81.55 29500 0.3141 Gulberg 3 5 0 2 0 0 1 22.5 3.8 14.1 5.8 20.1 13.2 63.7 3.0 34.8 11.8 86.21 40000 0.3017 Gadap 3 6 2 0 4 0 1 0.0 0.0 0.0 17.3 25.7 46.7 26.7 0.0 25.7 0.0 82.95 35000 0.2368 “#” means number. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 171 FIGURE 20: MARGINALIZATION INDEX, LOCATION OF TRAINING VENUE, AND PERCENTAGE OF APPLICANTS BY TOWN Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 172 Annex 6: Tests of Equality of Distributions The analysis used the nonparametric independent samples Kolmogorov-Smirnov test to determine whether the distribution of applicants by town differed significantly from the distribution of trainees. The test concluded that, for men, women, and the combined sample, the distributions of applicants and trainees were statistically equivalent. Levels of significance for the three tests are 0.964, 0.766, and 0.766 for the women, men, and combined samples, respectively. FIGURE 21: TEST RESULTS FOR EQUALITY OF DISTRIBUTIONS Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 173 Annex 7: Regression Results Effect of Distance on Recruiting TABLE 13: REGRESSION ANALYSIS OF EFFECTS OF DISTANCE ON RECRUITING Variable Estimated Coefficient Standard Error t-statistic P>t All Applicants (N=51, Adjusted R-squared=0.06) Distance –0.002060 0.000984 –2.1 0.041 Constant 0.056123 0.019800 2.83 0.007 Men (N=28, Adjusted R-squared=0.02) Distance –0.001100 0.000760 –1.44 0.157 Constant 0.043771 0.016009 2.73 0.009 Women (N=43, Adjusted R-squared=0.11) Distance –0.004990 0.002366 –2.11 0.045 Constant 0.119512 0.045094 2.65 0.014 The dependent variable is the road distance between the geographic center of the town in which an applicant lives and the center of the town in which he or she received training. Records are the percentage of trainees from each training location/home location pair. Determinants of Job Satisfaction TABLE 14: LOGIT REGRESSION RESULTS FOR JOB SATISFACTION Variable Coefficient Standard Error z P>z Constant term –2.01328 1.15374 –1.74 0.08 Post-training income 0.000034 0.0003 1.28 0.20 Post-training written or verbal contract –0.02382 0.3834 –0.06 0.95 Unmarried 0.96503 0.6033 1.60 0.11 Between 18 and 29 years old –0.23824 0.7036 –0.34 0.74 Job matches training 0.19219 0.3688 0.52 0.60 Completed secondary education 1.40132 0.8184 1.71 0.09 Completed intermediate education 1.466518 0.8096 1.81 0.07 Male –0.15056 0.4392 –0.34 0.73 The dependent variable is an indicator of job satisfaction. All variables except income are dummy variables. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 174 Determinants of Post-Training Personal Monthly Income TABLE 15: REGRESSION RESULTS FOR MONTHLY INCOME Variable Coefficient Standard Error t P>t Constant term –1060.292 10947.088 –.097 .923 Job matched to training 1000.421 1446.753 .691 .490 Employed prior to training 4511.652 1457.878 3.095 .002 Male 5373.529 1850.428 2.904 .004 Age in years 546.049 164.245 3.325 .001 Unmarried –4521.601 2359.553 –1.916 .057 Secondary education –3588.501 8942.685 –.401 .689 Intermediate education –3326.536 8943.327 –.372 .710 Bachelor or master’s degree 2753.838 9322.546 .295 .768 Korangi –4986.129 2344.600 –2.127 .035 Malir –2830.152 2609.944 –1.084 .280 Landhi –2500.559 2634.737 –.949 .344 Saddar –3708.617 3619.897 –1.025 .307 Liaquatabad 1344.148 4719.253 .285 .776 Jamshed 7531.838 4284.533 1.758 .081 New Karachi –8331.841 3619.967 –2.302 .023 Orangi –4968.516 3169.369 –1.568 .119 Gulshan –4388.211 3401.747 –1.290 .199 Shah Faisal –2096.814 4400.079 –.477 .634 Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 175 Annex 8: Effect of Duration on Labor Force Participation after Training TABLE 16: EFFECT OF DURATION ON LABOR FORCE PARTICIPATION Variable Estimated Coefficient Marginal Effect P-Value BT_LFP 1.663 5.276 0.000 Dur .034 1.035 0.861 Dur2 –.005 .995 0.626 Male 1.140 3.126 0.009 Unmarried –.731 .481 0.154 Primary 18.769 1.417 x 107 0.999 Secondary –1.586 .205 0.159 Intermediate –1.222 .295 0.278 Korangi Town .016 1.016 0.977 Malir Town 1.271 3.563 0.091 Landhi Town .205 1.228 0.754 Saddar Town .806 2.238 0.504 Liaquatabad Town –.406 .666 0.595 Jamshed Town –.569 .566 0.467 New Karachi 19.186 2.149 x 108 0.999 Orangi Town 1.025 2.787 0.379 Gulshan Town .895 2.446 0.459 Shah Faisal Town .172 1.188 0.864 Constant 1.684 5.387 0.276 Dependent variable is posttraining labor force participation, and independent variables include pretraining labor force participation, number of months since the completion of training (duration and its square), gender, marital status, education, and neighborhood. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 176 Annex 9: Disaggregates by Training Topic TABLE 16: NUMBER OF OBSERVATIONS BY TRAINING TOPIC Training Topic Count Percentage (by column) Men Women Total Men Women Total Stitching machinist 24 71 95 12% 97% 34% Refrigeration and air conditioning 39 0 39 19% 0% 14% CAD-CIVIL 20 0 20 10% 0% 7% Plumbing 19 0 19 9% 0% 7% General electric 16 0 16 8% 0% 6% Computer-Aided Design and Computer-Aided Manufacturing (CAD￾CAM) 15 0 15 7% 0% 5% Mechanical 14 0 14 7% 0% 5% Automobile 13 0 13 6% 0% 5% Carpentry 12 0 12 6% 0% 4% Fabrication, welding, and pipework 9 0 9 4% 0% 3% Vehicle body technician 9 0 9 4% 0% 3% Electrical and electronics 6 0 6 3% 0% 2% Electronics 6 0 6 3% 0% 2% Vehicle paint technician 2 0 2 1% 0% 1% Textile designing 0 2 2 0% 3% 1% Toyota technology education program 1 0 1 0% 0% 0% Total 205 73 278 100% 100% 100% Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 177 TABLE 17: BEFORE AND AFTER TRAINING MEDIAN INCOME BY TRAINING TOPIC Training Topic Number of Observations Median Income Before Training Number of Observations Median Income After Training Men Refrigeration and air conditioning 21 $78.43 22 $63.73 Plumbing 15 $78.43 12 $129.90 Carpentry 11 $156.86 11 $215.69 General electric 8 $98.04 13 $81.70 Mechanical 8 $66.18 11 $56.37 Stitching machinist 7 $117.65 14 $41.26 CAD-CIVIL 6 $39.22 13 $24.51 Automobile 5 $98.04 5 $98.04 CAD-CAM 5 $122.55 6 $120.10 Electrical and electronics 4 $42.48 6 $17.97 Fabrication, welding, and pipework 4 $160.13 5 $147.06 Vehicle body technician 3 $78.43 5 $11.44 Electronics 2 $64.62 3 $95.59 Vehicle paint technician 0 $0.00 2 $5.72 Toyota technology education program 0 $0.00 1 $11.44 Women Stitching machinist 29 $39.22 41 $49.02 Textile designing 1 $39.22 1 $31.86 Note: None of the changes in income are statistically significant. Karachi Youth Workforce Development Project: Midterm Evaluation Report (Amantech) 178 Annex 10: Conflict of Interest Disclosures The conflict of interest disclosures have been removed to protect the confidentiality of team members. They are available from MSI/PERFORM on request. U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523