PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION OCTOBER 22, 2018 This publication was produced for review by the United States Agency for International Development. It was prepared by, Management Systems International, A Tetra Tech Company. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION Contracted under Order No. AID-391-C-15-00004 Performance Management Support Contract DISCLAIMER This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents are the sole responsibility of the Management Systems International and do not necessarily reflect the views of USAID or the United States Government. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION iii ACKNOWLEDGEMENTS The evaluation team could not have conducted this evaluation without the assistance and cooperation of many others. The team particularly wishes to acknowledge the effort the implementing partners’ staff expended in familiarizing the team with the project, answering their many questions, and helping arrange the fieldwork for this logistically challenging evaluation. The willing cooperation of the project partners substantially improved the quality of the evaluation report. The team must also thank USAID staff for their frequent and productive engagement with the evaluation team. Finally, the evaluation would not have been possible without the contribution of the many individuals who generously gave of their time to participate in interviews and share their thoughts and experiences. The evaluation team, Niaz Murtaza Ali Sohail Rehana Perveen Shaikh Shehzad Ahmed Sara Anver Arslan Ahmad Muhammad Asghar Junaid Ashraf Umer Yar Khan PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION iv CONTENTS Acronyms...........................................................................................................................vi Project Summary .............................................................................................................vii Executive Summary.......................................................................................................... 1 Evaluation Purpose and Questions.................................................................................. 4 Theory of Change................................................................................................................................ 4 Evaluation Questions.......................................................................................................................... 4 Project Background .......................................................................................................... 5 Project Components........................................................................................................................... 8 Evaluation Methods and Limitations............................................................................... 9 Data Collection Methods................................................................................................................... 9 Data Analysis.......................................................................................................................................10 Limitations and Mitigations..............................................................................................................10 Findings and Conclusions................................................................................................ 11 Question 1: Change in Youth Attitudes.......................................................................................11 Perceptions of Employment Prospects.........................................................................................11 Prospects for Better Jobs..................................................................................................................12 Prospects for Increased Income.....................................................................................................13 Constraints to Improving Economic Prospects...........................................................................13 Conclusions..........................................................................................................................................18 Question 2: Effectiveness of Program Approaches...................................................................19 Community Engagement .................................................................................................................19 Institutional Capacity Building ........................................................................................................20 Improving Access to Workforce Education, Training, and Placement................................ 23 Conclusions..........................................................................................................................................27 Question 3: Sustainability.................................................................................................................27 Conclusions..........................................................................................................................................29 Question 4: Recommendations......................................................................................................29 Additional Recommendations ....................................................................................... 30 Annexes............................................................................................................................ 33 Annex 1: Statement of Work .........................................................................................................33 Annex 2: Assignment Work Plan...................................................................................................47 Annex 3: Conflict of Interest Declarations.................................................................................69 Annex 4: Interviews Conducted by the Evaluation Team........................................................70 Annex 5: Profiles of Core Team Members.................................................................................74 Annex 6: Data Collection Instruments.........................................................................................75 Annex 7: Documents Reviewed...................................................................................................106 Annex 8: Sampling ...........................................................................................................................107 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION v List of Tables Table 1: Project Summary................................................................................................................................vii Table 2: Summary of Data Collection Methods by Respondent Type ................................................10 Table 3: Distribution of Types of Partner Training Institutes................................................................21 Table 4: Monthly Salary by Profession of Placed Youth ..........................................................................26 Table 5: Microfinance Recipients by Activity and Sex..............................................................................27 Table 6: Interview Participants – Beneficiaries..........................................................................................70 Table 7: Key Informant Interviews – COs and Reflect Groups............................................................72 Table 8: Key Informant Interviews – Training Institutes.........................................................................72 Table 9: Key Informant Interviews – Employers.......................................................................................73 Table 10: Key Informant Interviews – Provincial Counterparts............................................................73 Table 11: Key Informant Interviews – IP and Sub-Contractors’ Staff ..................................................73 Table 12: Sample Beneficiary Respondents by District .........................................................................107 Table 13: Sample Beneficiary Respondents by Status............................................................................107 Table 14: Key Informant Interview Respondents at District and Provincial Levels........................107 List of Figures Figure 1: Map of the Project Area ................................................................................................................viii Figure 2: Marginalization Indicators for Project Districts......................................................................... 6 Figure 3: Project Overview .............................................................................................................................. 7 Figure 4: Respondents Reporting Improvement in Perceptions of Employment Prospects..........................................................................................................................................12 Figure 5: Pre- and Post-Training Employment Status...............................................................................12 Figure 6: Perceptions of Income Prospects................................................................................................14 Figure 7: Beneficiaries’ Self-Confidence.......................................................................................................15 Figure 8: Status in the Household and Community by Stage of Participation....................................15 Figure 9: Increase in Youth Involvement in Community Issues.............................................................17 Figure 10: Youth Trained by Type of Training Institute..........................................................................22 Figure 11: Fields in Which Graduates are Employed...............................................................................25 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION vi ACRONYMS BISP Benazir Income Support Program CNAQ Career Need Assessment Questionnaire CO Community Organization HDI Human Development Index IRM Institute of Rural Management PSDF Punjab Skills Development Fund PVTC Punjab Vocational Training Council PYWD Punjab Youth Workforce Development TEVTA Technical Education and Vocational Training Authority USAID United States Agency for International Development PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION vii PROJECT SUMMARY TABLE 1: PROJECT SUMMARY Title/Field Project/Activity Information Contract/agreement numbers AID-391-C-16-00001 Contracting/Agreement Officer’s Representative (COR/AOR) Start date September 2016 Completion date September 2019 Location Multan, Muzaffargarh, Bahawalpur, and Lodhran districts in Punjab province Implementing partner(s) Louis Berger Group USAID/Pakistan Mission Strategic Framework objectives addressed DO3: Increased stability in focus areas IR 3.2: Increased resilience in targeted communities vulnerable to violent extremism PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION viii FIGURE 1: MAP OF THE PROJECT AREA PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 1 EXECUTIVE SUMMARY Evaluation Purpose and Questions This midterm evaluation assesses the Punjab Youth Workforce Development (PYWD) project’s progress toward achieving its objectives and recommends course corrections related to design or implementation that can improve performance. Evaluation findings will also inform United States Agency for International Development (USAID) programming decisions for future interventions related to youth employment and preventing violent extremism. The evaluation addresses four questions: 1. How, if at all, have the project activities contributed to positive attitudes among youth beneficiaries? 2. How, if at all, are PYWD implementation approaches building toward achievement of relevant strategic objectives as specified in the project’s results framework? 3. What are the prospects for the sustainability of the project activities by stakeholders beyond the project life? 4. Are there any recommendations to enhance project results within the project’s current period of performance and budget? Summary of Findings and Conclusions The PYWD project contributes to improving economic opportunities and instilling socially constructive attitudes in at-risk youth, i.e., youth who are most vulnerable to the influences of violent extremism. The evaluation concludes that the project’s development hypothesis holds true: Engaging youth in economically productive activities contributes to changes in their attitudes toward themselves and their communities. The project is also influencing some of the individual-level push factors that may contribute to violent extremism—e.g., material incentives; boredom and desire for adventure; social exclusion and marginality; desire for empowerment, dignity, and self-assertion; and systematic socioeconomic discrimination—thus potentially contributing to reducing the vulnerability of at-risk youth to violent extremism. Specific conclusions include: • Youth attitudes: The project is improving at-risk youth’s access to employment and income earning opportunities by providing vocational training, career counseling, microfinance, and market linkages. Key pathways to these outcomes include the unemployed finding employment, individuals moving from unskilled to skilled labor, and the self-employed expanding their business and income. As their economic prospects improve, beneficiaries’ self-confidence and their understanding of, and to some extent their participation in, community activities increases. These changes are more pronounced in women because of their much lower levels of participation in economic and social activities prior to the project. Although few respondents mentioned participating in project-supported civic education events, those who did reported that the events positively influenced their attitudes toward the community, society, and the state. • Community engagement: Employing an inclusive development approach that draws on the connections and experience of trusted local community organizations (COs) to identify and work with at-risk youth is a key success factor of the project that avoids duplicating effort and saves time and resources. Working through COs to identify beneficiary youth improves the quality of intake for training and also helps retain youth in training courses. COs identify at-risk PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 2 youth by engaging with their parents, which may also contribute to building community resilience by enhancing the communities’ ownership of the project’s development activities. However, the failure to engage COs to provide a formal platform for engaging youth in community activities on a regular basis is a missed opportunity. The absence of such a platform or structure at the community level may limit the project’s effectiveness in reducing at-risk youths’ ongoing vulnerability to violent extremist influences. • Institutional capacity building: The project’s approach to institutional capacity building is systematic in that it identifies needs, assesses capacity, provides necessary equipment, trains trainers, and upgrades curricula. This improves employment prospects for graduates by enhancing training quality. Provision of equipment also increases the throughput of training institutes. The project could further enhance capacity-building outcomes by advocating with the provincial government to allocate resources for capacity building and by establishing formal linkages between training institutes and employers to leverage private sector resources for capacity building. • Access to workforce education, training, and placement: The project employs multiple channels and approaches to workforce education and training (e.g., a diverse range of vocations and training providers), which allows it to be flexible and responsive to the skill needs of at-risk youth and local employers. Reaching out to women in remote areas through local training providers and providing them with employment opportunities within their communities enhances the inclusiveness of the project approaches. Important placement mechanisms include career counseling, providing on-the-job training opportunities, and providing microfinance loans to the self-employed to help them start a business. However, some of these services are mutually exclusive for beneficiaries. For instance, beneficiaries of skills training do not receive entrepreneurship training or microfinance services, and they may find it difficult to start a business when they cannot find paid employment. • Prospects for sustainability: Youths’ continued interest in skills training will depend on the relevance of their skills to potential employers. The project’s approaches to enhancing employment prospects are effective. However, the prospects for sustaining some approaches— e.g., building the capacity of training institutes, providing career counseling services in the absence of dedicated staff, and linking training institutes with employers and chambers of commerce—are uncertain. Recommendations The project established processes that effectively engage youth in economic activities, which may lead to preventing violent extremism in beneficiary communities. The project’s inclusive approach, which targeted some of the most marginalized groups, is a step toward making youth stakeholders in development interventions, which may also reduce their vulnerability to violent extremism. Lessons learned from this project can be used to design future workforce development and violent extremism prevention programming. To help the project improve its effectiveness and prospects for sustainability within the current scope, period of performance, and budget, the evaluation team recommends that the project: • Provide need-based services: After successfully establishing processes over the first half of its life, the project now needs to consider providing services based on beneficiaries’ specific needs. For example, those who complete skills training should have access to entrepreneurship training and microfinance services to enhance their employability prospects. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 3 • Establish relationships between training institutes and private firms to leverage private resources to maintain training institutes’ capacity: To address public sector resource constraints and help maintain the capacities of the institutes, the project should begin, while there is still time, establishing relationships between the institutes and the private sector firms that rely on them for labor. District-level forums with stakeholders may be the appropriate venue to begin establishing these relationships and public-private partnership agreements. In addition to the above recommendations for improving results within the project’s current scope, period of performance, and budget, this report recommends the following steps for further enhancing effectiveness, sustainability, and contributions to outcomes related to preventing violent extremism. These recommendations may, or may not, entail additional time and cost. • PYWD should develop a youth engagement strategy: The project needs to develop a youth engagement strategy and build the capacity of COs to engage beneficiary youth in community activities and to socialize youth vocational training and employment within communities. This will enhance the effectiveness of the project’s approaches in contributing to reducing youth vulnerability and will serve as an important platform for communicating with youth about training and employment and motivating them to engage in economic activities in the future. • PYWD should build COs’ capacity and link training institutes to schools to continue identifying at-risk youth for training: The project may consider developing a comprehensive strategy for ensuring the sustainability of the processes for identifying youth for training and engaging them in community issues and activities. This may also require investing resources in the capacities of COs and schools to continue these processes after the project ends. The project may investigate the possibility of collaborating with schools and providing them with information about training institutes and on-the-job training opportunities as an additional approach to identifying at-risk youth for training. • PYWD should establish district-level forums for key stakeholders: The project needs to establish district-level forums for key stakeholders—training institutes, employers, chambers of commerce, and COs. The forums will serve as platforms for assessing labor market needs vis￾à-vis training providers’ capacity and the availability of skilled labor within the districts. They may also help institutionalize the practices currently implemented by the project for enhancing employability. • PYWD should advocate with the provincial government to support and replicate the model: To enhance and scale up outcomes, the project should advocate with the provincial authorities of public sector training providers to provide the resources necessary to continue project activities, e.g., continuously upgrade equipment and curricula, build the capacities of trainers, and hire dedicated staff for career counseling centers. The project should also communicate its successes and lessons learned to the provincial government and advocate for replicating the model in other districts. • USAID may consider extending the activity to enhance prospects for sustainable outcomes: Finally, to compensate for delays in beginning implementation, USAID should consider extending the activity. Developing and implementing an effective youth engagement strategy and building partnerships to sustain institutional capacity and outcomes will require time and sustained engagement. The project will also likely close before it can contribute to PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 4 placing the final batch of trainees. These conditions may argue for extending the project to enhance prospects for achieving sustainable results in all three project components. EVALUATION PURPOSE AND QUESTIONS This midterm evaluation assesses the project’s progress toward achieving its objectives, namely, to improve economic opportunities for targeted youth and to instill in them socially constructive attitudes and behavior. The evaluation identifies lessons learned about whether, how, and under what conditions vocational training enhances economic opportunities and contributes to encouraging socially constructive attitudes. It examines the effectiveness of the project’s approaches to achieving these objectives and recommends course corrections that can improve performance. Evaluation findings will also inform USAID programming decisions for future interventions related to youth employment and preventing violent extremism. Theory of Change The project’s theory of change hypothesizes that providing at-risk youth opportunities for vocational training and productive employment will enhance their economic prospects, thus changing their status in the household and community. Combined with community engagement activities, this will change beneficiary youths’ attitudes toward themselves and their communities, which will reduce their vulnerability to violent extremist influences and will ultimately contribute to increased stability in the project area. The project’s approaches to enhancing economic opportunities for youth and to developing socially constructive attitudes and behaviors include: • Engaging existing COs to identify and mobilize at-risk youth for training; • Building the capacity (by providing equipment, training trainers, and upgrading curricula) of training institutes to impart vocational training that is relevant to local employers; • Placing beneficiary youth in vocational training; • Enhancing trainees’ soft skills; • Supporting self-employment by providing entrepreneurship training and access to microfinance; • Building linkages between training institutes and employers to ensure that training meets employers’ needs and to facilitate placement; and • Engaging youth in community activities to foster an interest in community issues and encourage them to become productive members of the community. Evaluation Questions Based on the theory of change, this evaluation addresses the following questions: 1. How, if at all, have the project activities contributed to positive attitudes among youth beneficiaries? The evaluation focused on the following attitude changes: PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 5 i. Changes in beneficiary youths’ perceptions of their employment and income-earning prospects and ii. Changes in the attitudes of youth toward collective or community issues. 2. How, if at all, are PYWD implementation approaches building toward achievement of relevant strategic objectives as specified in the project’s results framework? This question considered whether activities align with strategic objectives and have been well coordinated and implemented. The evaluation examined the following implementation approaches: i. Community engagement ▪ Approaches to identifying and working with COs ▪ Approaches to identifying at-risk youth for training ii. Institutional capacity building ▪ Approaches to identifying needs and supporting upgrades to infrastructure and curricula in partner training institutes ▪ Developing partnerships for institutional capacity building (including leveraging private sector resources) iii. Improving access to workforce education, training, and placement ▪ Partnerships with government entities to provide skills training to at-risk youth ▪ Partnerships with the private sector to place youth in jobs ▪ Providing access to microfinance (including leveraging private sector resources) 3. What are the prospects for the sustainability of the project activities by stakeholders beyond the project life? Since it is premature to expect sustainable results, this evaluation explored whether the project was taking appropriate steps to enhance prospects for sustainability. The focus was on whether the ingredients for ensuring sustainability were in place at the levels of communities, training institutes, and prospective employers. 4. Are there any recommendations to enhance project results within the project’s current period of performance and budget? The evaluation presents recommendations for USAID and the implementing partner for each of the preceding evaluation questions based on relevant findings and conclusions. PROJECT BACKGROUND With 53 percent of the country’s estimated 208 million people, Punjab is Pakistan’s most populous province. 1 Punjab’s score on the United Nations Development Program’s human development index (HDI) places it first among Pakistan’s four provinces on a composite index of five dimensions of development.2 However, the southern region of the province, which comprises 11 of Punjab’s 36 1 Pakistan Bureau of Statistics, Province Wise Provisional Results of Census – 2017, http://www.pbs.gov.pk/sites/default/files/PAKISTAN%20TEHSIL%20WISE%20FOR%20WEB%20CENSUS_2017.pdf. 2 United Nations Development Program, Pakistan Human Development Index Report 2017, http://www.pk.undp.org/content/dam/pakistan/docs/HDR/HDI%20Report_2017.pdf. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 6 districts,3 is an economically depressed area. In 2017, the average HDI for south Punjab was 0.549 compared to an average of 0.746 for the other 25 districts. The PYWD project is implemented in four of the most marginalized districts in Punjab—Multan, Muzaffargarh, Bahawalpur, and Lodhran, all in south Punjab. These districts rank among the worst in the province in terms of the HDI and the Multidimensional Poverty Index, which measures deprivation based on three components, including living standards (Figure 2). The Multidimensional Poverty Index reports levels of poverty of 65 percent in Muzaffargarh, 50 percent in Bahawalpur, 47 percent in Lodhran, and 36 percent in Multan. Although the project districts rank in the middle or top of all Punjab districts on youth employment metrics (labor force participation, unemployment rates, and idleness, i.e., not being in the labor force or in school), youth unemployment rates are still high. Analysis of the 2014- 15 Labor Force Survey found youth unemployment rates of 46 percent in Lodhran, 44 percent in Muzaffargarh, 40 percent in Bahalwapur, and 32 percent in Multan. FIGURE 2: MARGINALIZATION INDICATORS FOR PROJECT DISTRICTS Ranking is from best (1) to worst (36). Sources: Labor force participation, unemployment rate, idle youth (analysis of the 2014– 15 Labor Force Survey); HDI (Pakistan National Human Development Report 2017, UNDP, http://www.pk.undp.org/content/dam/pakistan/docs/HDR/PK-NHDR.pdf); MPI (Multidimensional Poverty in Pakistan, UNDP, http://www.pk.undp.org/content/dam/pakistan/docs/MPI/MPI%204pager.pdf). The evaluation Statement of Work attributes these low development indicators, in part, to a poorly educated potential workforce; low productivity among those who are employed; and an overall unsatisfactory environment for business. To compound the problem, the technical and vocational training institutes in south Punjab are poorly equipped to train local youth to meet the needs of potential employers. In general, they reach a small percentage of marginalized youth; use outdated training methods, curricula, and equipment; and offer few career counselling or placement services. Although the evidence linking high levels of unemployment, low human development, and high incidence of deprivation to support for violent extremism is largely inconclusive, commonly used frameworks for classifying drivers of violent extremism suggest that these and other enabling environment factors may be important in some localities and contexts. Some of the individual push factors identified by Management Systems International’s drivers of extremism research in Pakistan include: material 3 The project’s monitoring and evaluation plan defines southern Punjab as consisting of Bahawalnagar, Bahawalpur, Bhakkar, Dera Ghazi Khan, Layyah, Lodhran, Mianwali, Multan, Muzaffargarh, Rahimyar Khan, and Rajanpur. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 7 incentives; boredom and desire for adventure; social exclusion and marginality; desire for empowerment, dignity, and self-assertion; and systematic socioeconomic discrimination.4 These factors may put youth who are out of the labor force because of a paucity of skills and limited economic opportunities at higher risk of falling prey to violent extremism. Providing skills training, financial resources, and market exposure may improve employment prospects and thus reduce beneficiaries’ vulnerability to violent extremism. The three-year (September 2016 – September 2019) USAID/Pakistan-funded PYWD project, implemented by the Louis Berger Group—and subcontractors Institute of Rural Management (IRM) and FINCON—addresses these issues on several fronts. The project development hypothesis posits that engaging youth in constructive activities and preparing them for productive employment will improve their social and economic status and, by mitigating some of the factors linked to support for violent extremism, contribute to increasing stability in project communities. The project works with communities to identify at-risk youth and engage them in community activities, build the capacities of training institutes to provide training that is relevant to the needs of local employers, and prepare identified youth for the job market by helping them access market-relevant training. Figure 3 provides a graphical overview of the project. FIGURE 3: PROJECT OVERVIEW Source: Project data, May 2018. 4 Management Systems International, “Study to Explore New Opportunities for Partnership and Programming in South Punjab: Research Report” (unpublished report, 2017). PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 8 Project Components The project comprises three primary components: community engagement, institutional capacity building, and access to workforce education and training. Community engagement: The project engages with communities through existing local COs and community elders and influential persons. COs identify at-risk youth for training and aid in retaining them in training programs. The major activities under this component include identifying and engaging with COs in the four districts; establishing and engaging with REFLECT groups5 to develop community consensus on local issues and identify and implement small-scale infrastructure schemes;6 raising awareness of the training opportunities at the community and district levels; engaging with schools to create awareness of training opportunities for youth; and undertaking communication and media activities to promote project objectives. The project approach seeks to engage all members of the community, including some of the most marginalized (e.g., women, non-Muslims, and madrassa students) so they all have the opportunity to participate in and benefit from development interventions. Institutional capacity building: This component involves upgrading the capacity of 22 public training institutes to improve the quality of training and offer career counseling and placement services to graduates. It includes upgrading equipment and curricula, training trainers, and fostering relationships between training institutes and private sector firms to leverage private investment in maintaining the capacity of institutes, e.g., by periodically upgrading equipment to industry standards. Access to workforce education and training: Under this component, the project supports expanding opportunities for at-risk youth to acquire skills and increase their incomes. The major activities under this component include selecting youth for training in technical skills, entrepreneurship, and soft skills; selecting and placing youth in jobs by providing career counseling, apprenticeship programs, on-the-job training, and placement services; and providing microfinance and toolkits (e.g., tools of the trade such as sewing machines) to self-employed youth. The design and implementation of the project activities adheres to USAID’s youth policy guiding principles, such as 1) invest in assets that build youth resilience; 2) involve and support mentors, families, and communities; 3) pursue gender equality; and 4) embrace innovation and technology by and for youth. Its objectives also align with the youth policy goal of engaging youth “to improve the capacities and enable the aspirations of young people so that they contribute to and benefit from more stable, democratic, and prosperous communities and nations.” The project also contributes to the mandate of Punjab Youth Policy 20127 which is “to facilitate, groom and guide the youth to live in peace and harmony, promoting attitudes, skills and knowledge and preparing them to shoulder responsibilities in the new millennium.” It also aligns with the policy’s strategic guidelines for economically empowering youth, i.e., 1) diversifying employment sectors and enhancing youth’s awareness of the available job market; 2) mapping of demands for skills and demands of multiple sectors in the job market; 3) establishing a program of certified and standardized technical training for literate and illiterate youth; 4) encouraging youth by promoting local crafts through 5 The project establishes REFLECT groups in collaboration with partner COs. Group members include active CO members and community members, including women and youth. 6 At the time of the evaluation, a few REFLECT groups have identified small-scale infrastructure projects, but none have begun implementing the projects. PYWD supports community infrastructure projects to give communities experience in identifying community needs and finding solutions. Because the REFLECT groups include youth, the projects also engage youth in these community decisions and activities. 7 Government of Punjab, Punjab Youth Policy 2012, https://www.unescogym.org/wp-content/uploads/2017/06/Punjab_Youth_Policy_2012.pdf. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 9 exhibitions and trade fairs in a collaborative role under the purview of concerned departments; and 5) encouraging entrepreneurship for youth, especially vulnerable youth. EVALUATION METHODS AND LIMITATIONS Data Collection Methods The evaluation team employed a mixed methods approach that relies on both quantitative and qualitative data from multiple sources to triangulate findings and generate robust evidence to answer the evaluation questions. Data sources included secondary data, comprising information gleaned from a review of project documents and quantitative results extracted from PakInfo, and primary data from interviews with beneficiaries and stakeholders. • Structured review of project documents and datasets: Documents reviewed included quarterly and annual performance reports; monitoring and evaluation plan; workplans; database of beneficiaries, including tracking data; and other project documents. • Interviews with youth beneficiaries: The evaluation team categorized beneficiaries into three groups: youth who were currently enrolled, youth who had graduated but were not yet employed, and youth who had been trained and placed in jobs or were self-employed. The team purposively selected a sample of 55 youth from these 3 groups, ensuring representation of beneficiaries from each district, type of training institute, and training course, and that the sample included at least 30 percent women. The interviews examined whether and how project activities changed beneficiaries’ attitudes toward communal issues and employment prospects. They also explored the mechanisms through which the project interventions influenced attitudes. • Key informant interviews with COs and REFLECT group members: The evaluation team conducted interviews with representatives of eight COs selected to represent each of the four intervention districts, gender diversity (one men’s CO, two women’s COs, and five COs with both men and women), and both urban and rural union councils in each district. The team also selected four REFLECT groups, one from each of the four intervention districts representing rural areas. The team used data provided by the implementing partner to select these groups. The interviews elicited respondents’ perceptions about the effectiveness of project interventions aimed at engaging COs, identifying at-risk youth, and changing attitudes. The team interviewed both male and female members of COs and REFLECT groups. • Key informant interviews with project stakeholders: The evaluation team conducted key informant interviews with representatives of subcontractors, training institutes, provincial-level counterparts (Technical Education and Vocational Training Authority [TEVTA], Punjab Skills Development Fund [PSDF], and Punjab Vocational Training Council [PVTC]), employers, chambers of commerce, and implementing partner staff. These interviews captured insights into the processes, effectiveness, and likely sustainability of project activities. The key informant interview tools were semi-structured, allowing flexibility to follow issues relevant to the evaluation that were not explicitly covered by the predesigned tools. The evaluation team developed separate interview guides for nine respondent types: beneficiary youth (separate instruments for enrolled, graduated but not placed, and placed youth); COs; REFLECT groups; implementing partner staff; subcontractor staff; training institutes; chambers of commerce; employers; and provincial-level counterparts (TEVTA, PSDF, and PVTC). PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 10 Sensitivity around the general elections made obtaining the No Objection Certificate required for on￾site data collection difficult. To mitigate the risk of delaying data collection, the evaluation team invited representatives of COs, REFLECT groups, implementing partners, and subcontractors to Islamabad for face-to-face interviews. The team conducted Skype or telephone interviews with youth beneficiaries and other key stakeholders, most of whom participated via Skype from the project offices in Multan and Bahawalpur. Youth and stakeholders who could not travel to the project offices participated in the interviews via telephone. Table 2 summarizes the number of interviews and respondents by type. TABLE 2: SUMMARY OF DATA COLLECTION METHODS BY RESPONDENT TYPE Method Respondent Type Number of Respondents Remarks Interviews Beneficiary youth 55 Purposive samples of 55 youth benefiting from training, career counseling, toolkits, or microfinance Key informant interviews COs 8 Purposive sample of eight COs engaged by the project Key informant interviews REFLECT groups 4 Purposive sample of four REFLECT groups formed by the project Key informant interviews Implementing partner staff 10 Purposive sample of 10 implementing partner staff members and 3 subcontractor staff members Key informant interviews Project stakeholders 25 Purposive sample of 25 key stakeholders, including provincial counterparts (3), training institutes (13), chambers of commerce (1), and employers (8) Total 102 Data Analysis The evaluation relied on both quantitative and qualitative data. Quantitative data from project documents provided an overview of implementation and youth engagement. Qualitative data from the interviews provided rich evidence of whether project activities are creating the anticipated changes in beneficiaries’ social attitudes and employment prospects, and whether the program logic is appropriate to achieving the project’s strategic objectives. The evaluation team used content analysis to examine and quantify patterns in the qualitative responses from the interviews. Content analysis identifies themes relevant to answering the evaluation questions, records the frequency with which the themes occur, and examines the content of the associated text to better understand the meaning of and context in which statements were employed. The evaluation team used the qualitative analysis software MAXQDA to analyze the qualitative data. Limitations and Mitigations The key limitations of the evaluation and the methods used to mitigate their impact included: • Inability to conduct in-person interviews with beneficiaries: The absence of face-to-face interviews with many respondent groups limited the team’s ability to establish rapport with respondents and to probe as deeply as it might have in an in-person interview. The team mitigated this challenge by employing experienced senior team members to conduct interviews, breaking down complex questions into sub-questions, and keeping interviews focused on key indicators. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 11 • Language/translation: The team conducted interviews in Urdu but developed the initial drafts of the instruments and conducted the analysis in English. Translation of the instruments into Urdu and of the interview notes back into English may have caused some loss of fidelity in the data. To mitigate this risk, the team members conducting interviews agreed on translations and practiced extensively in using the instruments. Senior team members who conducted interviews collaborated with the note takers when writing the notes to ensure data quality during recording, transcribing, and reporting. FINDINGS AND CONCLUSIONS Question 1: Change in Youth Attitudes How, if at all, have the project activities contributed to positive attitudes among youth beneficiaries? The evaluation assessed the extent to which the project affected youths’ attitudes, particularly their perceptions about their economic prospects and their social attitudes. The analysis triangulated evidence on changes in attitudes by interviewing both beneficiary youth and CO members about these issues. Perceptions of Employment Prospects Beneficiary youth report that project-supported activities such as training and access to services such as career counseling and microfinance enhanced their prospects for obtaining new or better employment. Overall, 87 percent of beneficiary youth the team interviewed reported improved perceptions of their employment prospects (Figure 4). Female beneficiaries were more likely than men to report improved perceptions of employment prospects. The data also seem to show that merely enrolling in training dramatically improved beneficiaries’ perceptions of their employment prospects. However, failing to find employment after graduation depressed these perceptions, while the reality of finding work improved them. “I think that I have enough abilities that if I go to find a job [prospect], and they take an interview, inshallah I’ll pass because I have completed six months of training. They have also given me a certificate, which is the proof that I’ve completed the training. So, wherever I go and submit my CV, I hope that I’ll get a job somewhere.” – Placed female beneficiary, Muzaffargarh Project-reported data reveal that 42 percent of the 2,536 youth—with equal percentages of men and women—who had completed training by May 31, 2018, had started working. By the time of the evaluation in August 2018, 74 percent of the youth interviewed who had completed training were employed or self-employed. However, because the sample of youth interviewed was not statistically representative of all trained youth, the team cannot conclude that 74 percent of graduates are currently employed. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 12 FIGURE 4: RESPONDENTS REPORTING IMPROVEMENT IN PERCEPTIONS OF EMPLOYMENT PROSPECTS Source: Evaluation interviews, August 2018. This figure reports the percentage of respondents who reported improvement in their employment prospects. Prospects for Better Jobs Besides improving employment prospects, the project also improved perceived prospects for better jobs and self-employment among the youth interviewed. In the small and nonrepresentative sample, two￾thirds of interviewed youth were unemployed prior to the training and one-fourth were working in skilled jobs. Most who were unemployed or working in unskilled jobs prior to training appeared to have moved to skilled jobs or self-employment by the time of the evaluation (Figure 5). FIGURE 5: PRE- AND POST-TRAINING EMPLOYMENT STATUS Source: Evaluation interviews, August 2018. This figure represents only the 46 interviewed youth who had graduated from training. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 13 The most common reasons beneficiary youth reported for obtaining new or better jobs included the new vocational, soft, and entrepreneurial skills they learned; the certificate provided by the training institute and recognized by employers as a proof of completing training; job placement activities, including on-the-job training; receiving information about employment opportunities; and microfinance loans and toolkits provided to self-employed graduates. “My bakery was near to the training institute. It was easy for me to give one hour for the training. Before attending the course, I was not able to deal with customers. Now I can talk in public as well. When I go to the village the people appreciate the way I communicate with them. I told people that I am attending a one-month soft skills training course to improve my communication skills.” – Self-employed male soft skills training beneficiary, Bahawalpur Female beneficiaries also explained that the training they received prepared them to work in new trades and skilled and formal work, such as making shoes, airline ticketing, videography and photography, and jewelry making. They also noted that exposing women to trades beyond the traditional trades of stitching and embroidery is expanding opportunities for women to engage in economic activities and enhance their incomes. Project data that tracks 627 placed beneficiaries shows that 28 percent of women are employed in marketing, administration, and event management, which are not traditional fields for women in the project areas. “People appreciate if someone learns new things. Bahawalpur is not a big city and people don’t know that jewelry making was possible to do from home. They also give value to my work.” – Graduated female beneficiary, Multan Prospects for Increased Income A majority of beneficiary youth and CO members, and all beneficiary youth who were enrolled in training or placed in a job, reported that the project activities improved youths’ prospects for earning higher incomes (Figure 6). Understandably, youth who were enrolled and those who were placed in jobs were more optimistic than those who had graduated but not yet found jobs. Beneficiary youth explained that the key pathways to higher income prospects include moving from unemployment to employment or self-employment; shifting from unskilled to skilled employment; and increasing business for those who were self-employed before the project intervention. For women, the key pathways to increased income were self-employment in trades such as beautician and garment worker and formal jobs such as photography and airline ticketing. Constraints to Improving Economic Prospects Beneficiary youth the team interviewed noted a number of constraints to enhancing economic prospects. Women mentioned the need to gain a parent’s or husband’s permission to work outside their households, the lack of safe transport in their area, their unwillingness to work in a mixed-sex environment, and their husbands’ or fathers’ not giving them permission to work in retail shops where they would have to interact with men. Beneficiaries also reported limited job opportunities in some trades—e.g., the hospitality, travel, and manufacturing sectors—as well as the low wages in the local areas as key challenges undermining their economic prospects. “I am staying at home; my husband doesn’t allow me [to take a] job, not even teaching. If I am not working, it doesn’t mean that I have not used information or wasted this course. I will use it definitely in the future. I may start my own work from home.” – Graduated but not placed female technical training beneficiary, Multan PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 14 FIGURE 6: PERCEPTIONS OF INCOME PROSPECTS Source: Evaluation interviews, August 2018. This figure reports the percentage of interview respondents who reported improvement in their income prospects. Participating in project activities and improved economic prospects positively affected beneficiary youths’ attitudes, self-confidence, and understanding of and participation in community activities. The project enhanced beneficiary youths’ self-confidence as reflected in their status in their household and community and in their perceptions of their role in the community (Figure 7). Women were more likely than men to report increases in all three measures of self-confidence. Both men and women attributed the change in their status and role in the community to their enhanced skills, economic opportunities, and ability to earn more income. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 15 FIGURE 7: BENEFICIARIES’ SELF-CONFIDENCE Source: Evaluation interviews, August 2018. This figure reports the percentage of respondents who reported improvement in their status and role at the household and community levels. By stage of participation in the project, the data suggest that the status of youth in their household and community increases as they enroll in training and find gainful employment. But their self-confidence wanes if they fail to find a job after graduating (Figure 8). FIGURE 8: STATUS IN THE HOUSEHOLD AND COMMUNITY BY STAGE OF PARTICIPATION Source: Evaluation interviews, August 2018. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 16 The key mechanisms through which interviewees reported that the project improved their status in their household and community were: • Changing their status from an idle to a productive person: Many (38 of 55) beneficiaries the team interviewed were not involved in educational, employment, or other productive activities prior to the project, and family and community members viewed them as idle. Seven recounted that their participation in training and, subsequently, in work removed the stigma of being idle in the eyes of their household and community and enhanced their status. Many women (28 of 30 interviewed) also mentioned that training and employment changed views about gender roles as their families started accepting that women can be active participants in societal activities. • Ability to support personal and family expenses: Twenty-seven of the 35 employed youth the evaluation team interviewed explained that they were now able to meet their own expenses and reduce the financial burden on their families. Some even reported being able to help support their families. This financial ability helped enhance the status of many youth in their families. • Enhanced knowledge and skills: Seventeen of the trained youth mentioned that the knowledge and skills they gained in technical and other areas encouraged their families to involve them in family decision-making, thus enhancing their status within the household. • Ability to act as role models and mentors: Ten of the 55 beneficiaries the team interviewed explained that, as the project activities increased interest in vocational training within communities, other youth and parents actively sought the advice of youth already enrolled in training or subsequently working in deciding whether they should pursue a career or enroll in vocational training. • Ability to contribute to community work: Many of the technical skills beneficiary youth gained, such as electrical wiring and repair, are in demand within communities. Eleven beneficiary respondents said that their ability to contribute to community work and help resolve problems helped enhance their status in the community. Interviewed youth also foresaw greater future roles for themselves in the community, most frequently including roles as local leaders, trainers, entrepreneurs, and social activists. Women were notably more likely than men to see themselves playing an expanded role in the community (Figure 9). Contrary to perceptions of status in the household and community, interviewees’ perceptions of their future role in the community declined as their status advanced from enrolled through placed. The evaluation team hypothesized that the inability to find a job may explain the decline in youths’ perceptions of their interest and participation in community issues between the time when they are enrolled and when they have graduated (but are not yet placed). Being too busy with work may explain the further reduction after they are placed in a job. This may mean that COs need to make an extra effort to keep placed youth engaged with, and contributing to, their communities, especially after they acquire valuable skills. Interviews with COs suggest that the project is socializing youth vocational training and employment within the community, thus making youth participation in vocational training a common discussion topic at the community level. “I completed a six-month course. Now I see that my interest increased in work and I want to do something useful. I guide my friends who are sitting idle, I suggest [that they] learn different skills which will lead [them] to find a job in future.” – Graduated male beneficiary, Bahawalpur PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 17 Although the project increased beneficiary youths’ perceptions of their status and roles in the community, they still have a relatively limited understanding of, and participation in, community issues (Figure 9). FIGURE 9: INCREASE IN YOUTH INVOLVEMENT IN COMMUNITY ISSUES Source: Evaluation interviews, August 2018. This figure reports the percentage of respondents who reported an increase in their interest and participation in community issues. The key mechanisms interviewees reported as explanations for their greater understanding of, and participation in, community activities were: • Advocacy for women’s rights: Women are increasingly becoming aware of issues such as early marriages, the importance of education, vocational training, employment, and environmental issues. Youth, especially girls, have started motivating other youth to pursue vocational training and employment—eight women interview respondents mentioned this. “I want no one girl [to] remain behind. In our society parents do not allow their girls to work outside the home. Parents marry their girls at [an] early age when they get free from education. I encourage and motivate the other girls to work hard. Girls from our mohallah discuss with me and asked if there is any other opportunity, please let us know. Now their parents make me [an] example and say if she can do [it], why not our child [girl]?” – Enrolled female beneficiary, Bahawalpur • Political engagement: Five beneficiary respondents reported playing a key role in motivating people to vote during the 2018 elections. Other youth also started presenting community issues to local government officials due to their improved communication skills after the training. “In earlier elections, I was not aware about the importance of elections and for whom to vote, or not. During the 2018 election, I informed the family that we should vote PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 18 [for] the better candidate which may work well and work for our mohallah. This change came after the PYWD training course.” – Placed male beneficiary, Bahawalpur “Previously I did not care about any issue. But now I understand the issues and discuss these issues with responsible persons and elected officials of local government. This is because of [my] increase in confidence level after training and [getting a] job.” – Placed male beneficiary, Multan • Contribution to community work: Many of the technical skills youth acquired through training are in demand within communities. Eleven of the trained youth mentioned helping resolve key technical issues in communities. “Whenever [the] community or surrounding people need [the] services of an electrician, they contact me.” – Placed male beneficiary, Multan The project engages youth in community work through REFLECT groups. This enhances opportunities for youth to contribute to community work along with other community members. The project’s approach of engaging marginalized groups to identify and address community development needs also improves their perception of self-efficacy. However, the project scope lacks activities for beneficiary youth to engage with their communities on a regular basis through COs, and this has limited the greater integration of youth into their communities. “My status in the community is changed. People appraise my work and my efforts toward my support to my house. If somebody needs any help, they refer each other to me. I help females in terms of financial issues. If any customer who is very poor and can’t afford my services comes to my parlor I try to give them my services free of cost. The main reason is that I am independent and I can bear my own expenses. I try to convince girls in my area to start some work and learn some skills. Their parents ask them to learn something from my example, the way I support my family.” – Placed female beneficiary, Multan Conclusions The data shows that the project enhanced the employment and income-earning prospects of targeted at-risk youth, their self-confidence, and their participation in community activities. Women consistently reported greater improvements than men for both economic prospects and socially constructive attitudes. This is likely due to the much lower levels of participation of women in economic and community activities prior to the project. Thus, women seem much more aware of the improvement in their socioeconomic status than men. However, cultural barriers that limit women’s participation in economic activities outside their homes are blocking further improvements in their perceptions of their economic prospects.8 The limited number of more remunerative economic opportunities available in the targeted districts also constrains improvement in economic prospects. Improving youths’ social attitudes and perceptions of their economic prospects may positively influence some of the individual push factors that contribute to vulnerability to violent extremist influences. Regular activities to engage youth at the community level through COs may enhance these outcomes. 8 Six beneficiary youth, five training institutes, one employer, and one CO mentioned that cultural barriers (e.g., community pressure or requiring permission of a father, brother, or husband) limited women’s participation in economic activities. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 19 Question 2: Effectiveness of Program Approaches How, if at all, are PYWD implementation approaches building toward achievement of relevant strategic objectives as specified in the project’s results framework? Project activities are grouped under three components: community engagement; institutional capacity building of training institutes; and improving access to workforce education, training, and placement. Community Engagement The community engagement component works with local stakeholders to identify at-risk youth for training and then engage them in community issues and activities to improve their social attitudes. The project works on the community engagement component with a range of stakeholders including COs, local leaders, the Benazir Income Support Program (BISP), and the Social Welfare Department. The criteria for identifying and selecting at-risk youth are: • Resident of one of the four focus districts; • Aged between 16 and 29 years; • Beneficiary of BISP and/or low score on the Pakistan Poverty Alleviation Fund poverty scorecard; and • Has the basic educational qualifications for specific trades. The most important and structured relationship in this regard is with COs. Instead of establishing new COs, the project chose to work with over 200 existing COs established by other agencies. This helped the project save time and money by building on existing capacities and networks. The project does not provide any financial incentives to COs for identifying youth, which enhances the prospects that COs will be able to sustain this activity after the project ends. The geographic scope of each CO is flexibly determined, with some COs covering multiple villages in the absence of other viable COs, and multiple COs working in some villages where the project identified more than one capable CO. These COs identify youth by engaging with their parents and families in the community. Engaging parents and community members may also contribute to building resilience by enhancing the community’s ownership of development activities. A majority of interviewed youth and training institutes reported that they faced no major problems in the identification and selection process. The most common challenges COs reported (mentioned at least once by five of the eight COs interviewed) in identifying youth included the sociocultural barriers to engaging women in training and work opportunities outside their communities and the lack of awareness among youth and parents about the value of vocational training. COs usually overcame these challenges by educating people, through public meetings and home visits, about the value of vocational training and the importance of women’s participation in the workforce. Despite these challenges, the project is currently able to identify and select a sufficient number of eligible youth for the various training programs. “If we look at our social context, then we must opt for different strategies for identification of boys and girls because we can motivate boys or males rapidly as compared to females. In that context it is a bit difficult to work with girls or women and plan different strategies. In the case of girls, sometimes organizing big meetings is a challenge, therefore we visit door-to-door and motivate parents.” – CO member, Bahawalpur PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 20 “When they came to us to organize awareness sessions in the communities, it was difficult for us, and we faced challenges. Gathering people and motivating youth who have left schooling, motivating parents; it was a big challenge that parents did not understand the value of training as they were not educated. In order to increase motivation of youth, we organized a cricket tournament followed by an awareness session. In this way we sensitized communities.” – CO member, Multan Even though it is not part of their formal remit under the project, five of the eight COs interviewed also reported encouraging youth to remain in training by working with them and their parents to encourage attendance, arranging (but not paying for) transportation for women to training venues, and monitoring attendance. According to 8 of the 13 training institutes interviewed, this extensive community engagement and mobilization process helped identify qualified and motivated trainees and contributed to reducing dropout rates. Several COs interviewed also reported socializing communities to the benefits of vocational training that may prepare youth for lucrative economic opportunities beyond their traditional involvement in agriculture and livestock activities. “Yes, we work on retention because communities know us; Louis Berger accesses communities through us so it is our prime responsibility. We have asked to receive any complaint that a trainee is not attending the training. We met with the parents to motivate them. After selection, we ensure that trainees attend the training. It was kind of a non-formal agreement and we ensure that the trainee and parents comply with the agreement. We discussed with parents and trainees that when we select a trainee, it means we spend our time; if a trainee does not complete the course he or she has wasted a seat that another deserving candidate could avail. If a trainee leaves the course, he or she will spoil the resources and have wasted a chance not for him- or herself only but also for another deserving candidate. Based on these facts, we have had informal agreements with parents.” – CO member, Bahawalpur To enhance youths’ interest in and engagement with their communities, the project conducts periodic district-level events that promote beneficiary youths’ participation in the community. Although the COs do not have a formal role in these activities, four of the eight COs interviewed mentioned that they were involved in the activities informally. The project anticipates that these activities will instill positive social attitudes in youth and integrate them into communities, ultimately contributing to reducing their vulnerability to violent extremist influences. The project also uses the innovative participatory development approach of REFLECT groups to engage community members. These groups identify and propose solutions to community issues using participatory approaches that directly engage youth as part of the larger community. The project supports (finances) implementing the identified solutions when these solutions involve small infrastructure schemes. Institutional Capacity Building To ensure training quality, the project conducted comprehensive capacity assessments to identify 140 high-capacity institutes across the 4 districts. Based on a second assessment, the project selected 22 of them for institutional capacity-building support that included: • Retooling or upgrading equipment and machinery for six selected institutes; • Upgrading training curricula for two highly demanded trades or vocations annually; • Training 66 trainers (3 from each of the 22 institutes) in pedagogy, teaching methods, relevant vocations, and course management; PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 21 • Establishing six career counseling centers; and • Producing videos of two courses and sharing the videos on the web portals of selected institutes. The 140 partner institutes represent a range of types of institutes, and all are located in the project districts. Working with local institutes helps match training to the identified needs of local employers, while the convenient locations make it easier for marginalized youth, especially women, to attend training. Training partners include district-level TEVTA institutes, PVTC institutes, and Sanatzar industrial homes that focus on vocational training for women.9 The project also partners with reputable private training institutes. This includes some private institutes that establish training centers for women in rural areas in partnership with the PSDF, a not-for-profit company established in 2010 by the government of Punjab. The project also places youth directly with firms for on-the-job training. Finally, the project provides entrepreneurship training through the IRM, one of its implementing partners. Table 3 summarizes the distribution of the 140 institutes with which the project works across institute type in each district. Overall, the project works mostly with public sector institutes but with very few Sanatzar industrial homes. TABLE 3: DISTRIBUTION OF TYPES OF PARTNER TRAINING INSTITUTES Institute Type District Overall Multan Bahawalpur Muzaffargarh Lodhran TEVTA 30% 36% 34% 29% 33% PVTC 22% 23% 29% 53% 28% Sanatzar 2% 3% 3% 6% 3% Private institutes 46% 38% 34% 12% 36% Source: Project data, May 2018. As of May 2018, the project was working with a total of 140 training institutes. Overall, the share of youth trained by each type of training institution roughly mirrors the distribution of types of training institutions summarized in Table 3, i.e., public institutes (TEVTA and PVTC) comprise 61 percent of all partner institutes and trained 62 percent of students (Figure 10). 9 TEVTA is a provincial-level public institution, with over 400 district-based institutes providing vocational training. PVTC is an autonomous corporate body established by the Punjab government that aims to alleviate poverty by imparting demand-driven skills training to beneficiaries at their doorstep, and by involving the private sector to enhance employability. Sanatzar is a vocational training body of the Punjab Social Welfare Department with a specific focus on women’s vocational training. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 22 FIGURE 10: YOUTH TRAINED BY TYPE OF TRAINING INSTITUTE Source: Project data, May 2018. “We started work on career counseling in April 2018 at the time of our training of three days. They trained us to assess the student to know if there is a need for career counseling. For this purpose, we used a pro forma named CNAQ [career need assessment questionnaire]. After filling of this pro forma, we will come to know about student needs and will go the next step of counseling. It is called the GROW model. There are some other questionnaires in the GROW model for counseling purposes. There are 10 grades in the GROW model. We assess the counseling needs with grades that show how students need these services. These grades will also guide us to select a sector for students.” – Career counseling center staff member, Bahawalpur The project provides capacity-building support to public institutes, which usually have limited access to such support. All institutes receiving capacity-building support and their provincial counterparts were satisfied with the support. “PYWD has provided capacity-building support to our embroidery, motorcycle mechanic, electrical, and air conditioning trades. They provided equipment for these trades. For motorcycle mechanic they provided motorcycle, refrigerators, air conditioners for these courses. For every trade they provided 15–16 items, for embroidery they provided machines and tables. Females are satisfied and happy with these improvements and it has also improved the quality of training.” – PVTC respondent, Muzaffargarh The interviewed youth were satisfied with the training experience. The only negative comments (from four respondents) suggested that the training include a greater focus on practice as opposed to theory and use new, instead of old, equipment. The capacity-building activities benefit current trainees who are not supported by the project and will likely benefit future trainees as well. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 23 “Yes, the training has helped a lot. In the past I was not in the habit of keeping record of my expenses and income. Sometimes I used to spend all the money and sometimes I kept all the money in savings and later I spent it on my business. But after the training I have started to follow a procedure of financial management. Now I keep a record of my income and I don’t spend all the money at once. At the end of the month I separate money for saving, for business, and for personal use as well. I have also started keeping a record of my income in written form as well.” – Placed female beneficiary, Multan “The training quality was very good. The duration of the training was fine. We learned stitching, cutting, and tailoring in the given period in detail. Trainers were very friendly and they taught us well. They always answered our questions. We were given a book for home. The book contains information about stitching, tailoring, and how to use a sewing machine. And the book also had information about parts of the sewing machine. The book was fine. I am only in the middle and there were things that I wasn’t able to understand. I asked those things from my trainer and she explained to me about everything.” – Placed female beneficiary, Muzaffargarh Improving Access to Workforce Education, Training, and Placement Besides providing training through the wide range of training institutes mentioned earlier, the project also offers students a choice among three main types of training to address their specific needs. These include training in technical skills, soft skills, and entrepreneurship. The project provides technical training, with courses ranging in duration from three to twelve months, through TEVTA, PVTC, and private institutes, as well as through Sanatzar industrial homes and on-the￾job programs with individual firms. The project currently offers training in the fields of agriculture and livestock; beauty, clothing, and jewelry; food, hospitality, and travel; information technology management; management, marketing, and event management; electrical, mechanical, and technical trades; videography and photography; and other miscellaneous fields. The project also provides a 30-day soft skills training through TEVTA and a 5-day entrepreneurial training through IRM. The project completed a skills gap analysis in July 2018 that identified several additional sectors with unmet industry demand for skilled labor. These sectors, and examples of needed skills, include agriculture and livestock (mango production, seed testing, mushroom culture, and pest management), manufacturing (machine operation, fabrication, and quality assurance), services (health and hospitality), and construction (masonry, electrical, plumbing, and machine operation). The project is using this information to better target its course offerings to the needs of employers. The project also seeks to enhance the capacities of training institutes to provide career counseling services. In its first year, the project established 6 career counseling centers in 6 of the 22 training institutes that received institutional capacity-building support. It equipped the career counseling centers with relevant information technology (hardware and software) and arranged a two-day training workshop for two staff members from each of the six career counseling centers. However, interviews with training institute staff found that the institutes have not hired dedicated staff for the career counseling centers and instead rely on teaching staff to provide career counseling on a part-time basis. Currently established career counseling centers are based in institutes located in main cities, which may limit their accessibility to those living in rural areas, especially women. Through May 31, 2018, the project reported having provided career counseling services to 525 persons, 75 percent of them men and 25 percent women, and none of them graduates of project-supported training according to the implementing partner. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 24 Project beneficiaries reported that they received employment and self-employment services through the following mechanisms: • The training institutes10 help place trainees with employers and also inform trainees of local employment opportunities. • Employers often hire youth who train on-the-job in their facilities. • Private sector institutes are required to guarantee placement of all trainees. • Microfinance services and toolkits are provided to select beneficiaries to help them establish or operate a self-employment business. The project reports that, as of May 31, 2018, 42 percent of graduated youth are employed or self￾employed, with this percentage being equal for men and women. Project-reported tracking data show that most women are placed in tailoring, stitching, and design; marketing, administration, and event management; and jewelry making. Most men, on the other hand, find work in agriculture; food and hospitality; marketing, administration, and event management; and tailoring, stitching, and design (Figure 11). 10 While from the beneficiaries’ perspective, the training institutes were connecting them with employers, it was often project staff that provided these services. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 25 FIGURE 11: FIELDS IN WHICH GRADUATES ARE EMPLOYED Source: Project-maintained trainee tracking data. Beneficiary tracking data on 627 placed graduates maintained by the project suggest that, despite their training, salaries for many beneficiaries are not high (Table 4). Nearly two-fifths (39 percent) of placed graduates are receiving salaries below Rs. 3,000 per month, while more than half (53 percent) are receiving salaries below Rs. 6,000 per month. For reference, at current exchange rates, Rs. 6,000 per month is approximately USD$1.60 per day, which is below the international poverty threshold. Even so, most beneficiaries still live with their families and, according to the 2015–16 Household Integrated Economic Survey, 11 Rs. 6,000 represents 24 percent of the average monthly income of a rural household in Punjab. A salary of Rs. 6,000 per month thus makes a substantial contribution to household income. 11 Pakistan Bureau of Statistics, Household Integrated Economic Survey (HIES) 2015-16, http://www.pbs.gov.pk/sites/default/files/pslm/publications/hies15-16/write%20up%2015-16-HIES-final.pdf. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 26 TABLE 4: MONTHLY SALARY BY PROFESSION OF PLACED YOUTH Professions Salary Ranges Average Monthly Salary (Rs.) Below Rs. 3,000 Rs. 3,001– Rs. 6,000 Rs. 6,001–Rs. 10,000 Rs. 10,001–Rs. 15,000 Rs. 15,000 & Above Food and hospitality services 0% 9% 53% 36% 2% 9,495 Jewelry making and beautician 80% 16% 3% 1% 0% 2,285 Agriculture-related 0% 0% 36% 62% 1% 10,805 Marketing, administration, and event management 5% 10% 14% 48% 24% 11,845 Travel management 8% 8% 67% 8% 8% 8,240 Tailoring, stitching, and design 52% 21% 15% 10% 1% 4,350 Technician, mechanic, electrician 54% 7% 17% 20% 1% 5,160 Videography and photography 0% 0% 67% 22% 11% 10,035 Web development and information technology 16% 16% 32% 19% 17% 8,870 Miscellaneous 33% 21% 21% 14% 10% Overall 39% 14% 22% 20% 4% Source: Project data, May 2018. Reflects data from 627 placed beneficiaries. Note: For domestically employed youth—defined by the project as youth who work in their households without monetary remuneration—the project imputed a salary as the estimated monetary value of their services to the household. It is worth noting that two-thirds of employed male graduates are earning relatively high incomes in the fields of agriculture; marketing, administration, and event management; and food and hospitality. Half of employed women, on the other hand, are earning relatively low incomes in the fields of tailoring, stitching, and design and jewelry making. More than half of the employed women, however, work part￾time from home. Finally, the project also provides access to microfinance services to those who successfully complete the entrepreneurship training program conducted by IRM. As of May 31, 2018, the project reports that 103 beneficiaries have received microcredit, of which 67 percent are men and 33 percent are women. All nine of the beneficiaries the team interviewed who received loans reported that the process for receiving loans was quick and efficient. Most self-employed women who received microfinance loans work in the fields of tailoring, embroidery, and beautician, while men are most likely to take loans to support work in vending and miscellaneous businesses (Table 5). PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 27 TABLE 5: MICROFINANCE RECIPIENTS BY ACTIVITY AND SEX Activity Women (n=34) Men (n=69) Overall (n=103) Transportation 0% 10% 7% Agriculture-related 0% 6% 4% Photocopying, stationery, and stamp papera 0% 4% 3% Vending 9% 29% 22% Tailoring, embroidery, and beautician 79% 3% 28% Auto workshop 0% 4% 3% Electronics and mobile phone sales and service 0% 13% 9% Miscellaneous 12% 30% 24% Total 100% 100% 100% Source: Project-reported data as of May 31, 2018. a Stamp paper is special paper use to prepare official documents. The project’s approach of paying stipends through bank cards is a welcome step toward financial literacy and inclusion. The opportunity for marginalized youth to interact with the banking system may expose them to other financing opportunities for loans in the future. Conclusions In terms of current performance, the youth identification, training, and institutional capacity-building processes are working reasonably well, although there is room for improvement. Working through local COs, elders, and influential persons is an effective approach to identifying at-risk youth. However, the project design does not include regular community-level activities to involve beneficiary youth in community issues and activities. The institutional capacity-building approach for training institutes is comprehensive, systematic, and based on in-depth capacity assessments that help identify needs for upgrading curricula, training trainers, and providing equipment. However, the project has not yet implemented its plan to develop relationships between training institutes and private firms to leverage private funding to sustain institutional capacity-building, e.g., periodically upgrading equipment to industry standards. Though the project incorporates multiple approaches for increasing access to workforce education, training, and placement, it does not offer all types of support to each beneficiary. For example, the project does not offer entrepreneurship training and microfinance loans to graduates of skills training programs. Career counseling centers have no dedicated staff, and the project, rather than the career counseling centers housed in the institutes, is leading placement activities. Question 3: Sustainability What are the prospects for the sustainability of the project activities by stakeholders beyond the project life? At this midpoint of the project’s period of performance, it is too early to evaluate the sustainability of project activities. Thus, the evaluation examines whether the project is introducing the incentives and building the capacities necessary to ensure sustainability of activities and components, as well as the key challenges to sustainability that are already apparent at the community, training institute, and employer levels. At the community level, the key activities that must be sustained are identifying at-risk youth for training, integrating youth into community activities, and the identifying and implementing community infrastructure schemes through REFLECT groups. COs are the key mechanisms for identifying youth. The project does not provide financial incentives to COs for this activity, which improves the prospects PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 28 for sustaining this activity since the COs have demonstrated the ability to manage the activity. Seven of the eight CO representatives interviewed reported that they expect to continue identifying youth for training after the project ends as the project has provided them with the relevant information on training courses and institutes. However, project staff reported that motivating the COs to identify a sufficient number of youth who meet the training eligibility criteria has required the staff’s continuous engagement. Project staff also play a critical role in smoothly transmitting details of the identified youth to training institutes for final selection through a process of document review, interviews, and tests. Once the project ends, COs will have to manage the entire process themselves, and the youth identification process may slow. This issue may be particularly acute in areas where the project relies on the help of local influential persons and other stakeholders to identify youth. However, the COs are not yet proactively working with employers and training institutes to identify their requirements so they can nominate youth for training and placement in jobs. Furthermore, the project currently pays tuition and a stipend while beneficiaries are enrolled in training. While public institutes may be able to find alternative resources to subsidize training for marginalized youth after the project ends, private institutes may not have this capacity.12 Youth may therefore have to cover these costs themselves for some of the courses. Thus, there are significant barriers to the sustainability of the youth identification and application process after the project ends. COs currently have no formal role in engaging youth in community activities. The REFLECT groups involve some youth in their work, but the project itself arranges most of the activities designed to integrate youth into communities. Unless COs and REFLECT groups become more actively involved in activities designed to integrate youth into communities, the sustainability of these activities, and the associated outcomes, is questionable. At the level of training institutes, the key activities that must be sustained are institutes’ willingness to work with COs to identify marginalized youth at the community level for training and the continued capacity building of the training institutes to ensure high-quality training. Many training institutes focused on training marginalized individuals prior to the project, and their collaboration with the project has further sensitized them to the needs of these students. Especially at the district level, training institute staff are now more aware of the utility of working with COs to identify suitable youth to help them meet their training and retention targets. This dynamic creates joint incentives to sustain this project component, although, as noted above, the COs may lack the capacity to sustain their role in this process. The institutional capacity-building activities undertaken by the project (e.g., improving curricula, improving teaching quality, and upgrading equipment) will continue to enhance training quality for several years. The project’s plans to help establish relationships between training institutes and private firms to leverage private resources to maintain the capacity of institutes (e.g., by periodically upgrading equipment to industry standards) are important to sustaining the capacity-building component. However, the evaluation team found no evidence of such linkages. The career counseling centers established within the training institutes are an important step for enhancing employability; however, the lack of dedicated staff for these centers may compromise the effectiveness and sustainability of career counselling activities in the future. Five of eight employers interviewed for this evaluation expressed their willingness to continue hiring graduates of the project’s partner training institutes, as they can now find skilled labor locally for many positions they earlier had to fill from other geographical areas at higher cost. Six of the eight employers 12 Donors or government programs often support or subsidize training for marginalized youth through public sector institutes. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 29 consistently rated the project graduates they hired as having better skills and attitudes than other local labor. They particularly appreciated the soft skills of project-supported employees and attributed their better attitudes to the greater importance of a job to marginalized youth as opposed to those from better-off households. Conclusions Many of the necessary incentives and capacities to ensure the sustainability of project components and processes are lacking. It is not clear how the likely termination of subsidized training (i.e., payment of stipends and fees) will affect youths’ willingness to enroll in training. Although COs have expressed their intention to continue identifying and motivating youth for training beyond the project life, their ability to do so without the continuous engagement of project staff is also questionable. Successful socialization of skills training and employment in communities is likely to spur youth and parents to put pressure on COs to continue their role in this regard. However, youths’ success in finding employment after training will be crucial for such socialization to be successful in the future. Although some training institutes have established career counseling centers to support job placement, they have not yet demonstrated the willingness to staff the centers appropriately. In the absence of dedicated and trained staff, career counseling services may not continue beyond the project life. Sustaining the enhanced capacity of training institutes, and keeping them current, will require the support of provincial counterparts and authorities to provide resources to regularly train trainers and upgrade curricula and equipment to maintain relevance to the changing needs of employers. Currently, the project serves as a bridge between employers, training institutes, and the community to support training and placement. However, there is no institutional mechanism in place to sustain this role beyond the project life. Hence, many of the positive outcomes created by the project to date may not continue after the project ends unless the project takes steps to create the necessary incentives for this purpose within communities, training institutes, and employers. Question 4: Recommendations Are there any recommendations to enhance project results within the project’s current period of performance and budget? The PYWD project is appropriately designed to improve economic prospects and social attitudes among marginalized youth in the project areas. Lessons learned from this project can be used to guide the design of similar programming in new areas and/or to expand existing programs to include youth￾focused interventions. The main strengths of the project design are the following: • Engaging local COs in identifying at-risk youth for training leveraged their contacts in, and the trust of, the communities. • Partnering with a wide range of training institutes and offering training in many fields increased the relevance of training to youth and made participating convenient. • Providing job placement, career counseling, and microfinance services as well as training to support self-employment increased the project’s effectiveness. These project features represent a credible design for improving the economic prospects of marginalized youth. Despite these strengths, the achievements of the project on the three evaluation questions are mixed, and the project needs to take measures to enhance the effectiveness of its approaches and the prospects for sustainability. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 30 The following recommendations address identified gaps in the project’s processes and are meant to help the project build on its current strengths, bridge the gaps, and improve prospects for sustainability within the project’s current scope, period of performance, and budget as defined in project documents. They also provide insights for designing projects that have similar change objectives and operate in comparable environments. PYWD should provide need-based services to beneficiary youth for placement in paid and self-employment. After successfully establishing processes over the first half of its life, the project now needs to consider providing services based on beneficiaries’ specific needs. For example, the project should allow those who complete skills training to also enroll in entrepreneurship training and to access microfinance services in case they wish to pursue self-employment. Six of 13 graduates of skills training interviewed said they were interested in self-employment but did not have the necessary financial resources. Two of them also mentioned that they needed a toolkit, in addition to financial resources, to start their own work. PYWD should proceed immediately with its plans to establish relationships between training institutes and private firms to leverage private resources to maintain the capacity of institutes. To address public sector resource constraints and help maintain the capacities of the institutes, the project should begin, while there is still time, establishing relationships between institutes and private sector firms that rely on the institutes for labor. For example, it may be in the interests of some private sector firms to help institutes periodically upgrade equipment to industry standards or fine-tune curricula to make training more relevant to their needs. District-level forums to enhance coordination among key stakeholders may be the appropriate venue in which to begin establishing these relationships and public-private partnership agreements. ADDITIONAL RECOMMENDATIONS Several additional recommendations could also enhance results but would require expanding the project’s scope by adding new activities and, perhaps, reallocating unused funds from existing activities. These activities may or may not increase the project’s period of performance or overall cost. They may also help PYWD and similar projects contribute to the high-level outcome of preventing violent extremism in at-risk communities. PYWD should develop a youth engagement strategy to capitalize on the positive attitude change among youth, integrate them into community activities, and provide an alternative platform against violent extremism. The community engagement approach of the project is effective in identifying youth for training. However, the project lacks a strategy to engage beneficiary youth in community activities on a regular basis, nor is it engaging its local partners in such activities. The project should develop a youth engagement strategy and build the capacity of COs to implement the strategy and to socialize youth vocational training and employment within communities. This will enhance the effectiveness of the project’s approaches in contributing to reducing youth vulnerability and will serve as an important platform for communicating with youth about training and employment opportunities and motivating them to engage in economic activities in the future. Such activities will also provide opportunities for youth to use the skills they have gained to contribute to communal activities and assets. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 31 The ability of the COs to continue identifying at-risk youth for training is questionable. The project must adopt multiple strategies to overcome this, and other, barriers to sustaining this activity before it ends. To sustain this activity, the project must build the capacity of the COs to implement it, and it must motivate them to continue the activity after the project ends. The project may consider developing a comprehensive strategy for ensuring the sustainability of the processes for identifying youth for training and engaging them in community issues and activities. This may also require investing resources in the capacities of COs and schools to continue these processes after the project ends. Schools have close links with youth and a more permanent existence than COs. The project may investigate the possibility of collaborating with schools and providing them with information about training institutes and on-the-job training opportunities as an additional approach to identifying at-risk youth for training. The project may also invest resources in further socializing the concept of vocational training in communities to convince parents and youth of its value. This may include holding events at schools for youth who have completed their training and have been placed to share their experiences with students. Such socialization will ensure that parents and youth motivate COs and schools to continue their role in youth identification and referral to training institutes. It will also encourage youth to take the initiative to apply directly to training institutes. But the success of such socialization will depend heavily on current graduates’ finding good jobs easily. PYWD should establish district-level forums for key stakeholders for coordination between employers, training institutes, and community members to effectively plan and implement skills development training and enhance the employability of trained youth. Currently, project staff is coordinating activities among stakeholders. This coordination is a key factor in ensuring the effectiveness of identifying, training, and placing youth. To enhance this effectiveness and ensure continuity in coordination, PYWD should establish district-level forums for key stakeholders such as training institutes, employers, chambers of commerce, and COs. The forums will serve as platforms for assessing labor market needs vis-à-vis training providers’ capacity and the availability of skilled labor within the districts. They may thus help training institutes remain attuned to the changing needs of employers so they can adjust their curricula accordingly. They will also help institutionalize the practices currently implemented by the project for enhancing employability. Encouraging the participation of chambers of commerce that represent employers at the district level may enhance the prospects for the sustainability of such forums. PYWD should advocate with the provincial authorities and its counterparts to support the project model in the current implementation areas and replicate successes in other districts to expand the project’s benefits to a larger youth population. The project is providing capacity-building support to a select few training institutes. However, the lack of resources for activities—such as upgrading equipment and curricula to stay abreast of changes in technologies and the needs of employers, training trainers, and hiring dedicated career counselors— limits the ability of the institutes to maintain training quality and ensure employability. To enhance and scale up outcomes, the project should advocate with the provincial authorities to provide the resources necessary to address these constraints. The project should also communicate its successes and lessons learned to the provincial government and advocate for replicating the model in other districts. This will expand the benefits of this effective model to youth in other districts of the province. Advocacy should also address the need for resources to ensure that the institutes continue to prioritize and support the intake of marginalized youth, perhaps by subsidizing their training. It should also include supporting the entrepreneurship and soft skills training. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 32 USAID may consider extending the activity to enhance prospects for sustainable outcomes. Finally, it took the project longer than expected to obtain government permissions to begin field activities. These delays set back implementation of an already short (three-year) project and may compromise the project’s ability to achieve sustainable results. Changing attitudes, improving economic prospects, and enhancing resilience to violent extremist influences may also require sustained engagement. Furthermore, developing and implementing an effective youth engagement strategy and building partnerships to sustain institutional capacity-building activities and outcomes will require time. Finally, the project will likely close before it can contribute to placing the final batch of trainees. These conditions may argue for extending the project to enhance the prospects for achieving sustainable results for all three project components. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 33 ANNEXES Annex 1: Statement of Work PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 34 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 35 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 36 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 37 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 38 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 39 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 40 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 41 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 42 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 43 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 44 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 45 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 46 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 47 Annex 2: Assignment Work Plan PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 48 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 49 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 50 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 51 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 52 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 53 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 54 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 55 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 56 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 57 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 58 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 59 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 60 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 61 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 62 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 63 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 64 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 65 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 66 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 67 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 68 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 69 Annex 3: Conflict of Interest Declarations Team composition and conflict of interest disclosures were removed to preserve the confidentiality of evaluation team members. They are available from PERFORM upon request. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 70 Annex 4: Interviews Conducted by the Evaluation Team This annex provides details of the interviews the evaluation team conducted. To protect the anonymity of the respondents, the annex does not reveal individuals’ names or their positions if these would identify the individual. TABLE 6: INTERVIEW PARTICIPANTS – BENEFICIARIES S. No Respondent Gender District Date of Interview 1 Enrolled Female Multan 13-Aug-18 2 Placed Female Muzaffargarh 13-Aug-18 3 Placed Male Multan 13-Aug-18 4 Enrolled Female Multan 13-Aug-18 5 Placed Female Muzaffargarh 15-Aug-18 6 Enrolled Male Muzaffargarh 15-Aug-18 7 Enrolled Female Muzaffargarh 15-Aug-18 8 Placed Male Muzaffargarh 15-Aug-18 9 Placed Female Muzaffargarh 17-Aug-18 10 Placed Male Multan 17-Aug-18 11 Placed Male Bahawalpur 17-Aug-18 12 Graduated Male Bahawalpur 18-Aug-18 13 Enrolled Female Bahawalpur 18-Aug-18 14 Placed Male Bahawalpur 18-Aug-18 15 Placed Male Lodhran 18-Aug-18 16 Placed Male Bahawalpur 19-Aug-18 17 Placed Female Muzaffargarh 19-Aug-18 18 Graduated Male Bahawalpur 19-Aug-18 19 Graduated Male Lodhran 20-Aug-18 20 Placed Female Lodhran 20-Aug-18 21 Graduated Female Lodhran 20-Aug-18 22 Enrolled Female Lodhran 20-Aug-18 23 Graduated Male Bahawalpur 20-Aug-18 24 Graduated Male Bahawalpur 20-Aug-18 25 Enrolled Female Lodhran 20-Aug-18 26 Graduated Male Lodhran 20-Aug-18 27 Placed Male Bahawalpur 21-Aug-18 28 Placed Female Multan 28-Aug-18 29 Placed Female Muzaffargarh 28-Aug-18 30 Placed Female Multan 28-Aug-18 31 Placed Male Multan 28-Aug-18 32 Placed Female Multan 29-Aug-18 33 Placed Male Multan 29-Aug-18 34 Graduated Female Multan 29-Aug-18 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 71 S. No Respondent Gender District Date of Interview 35 Placed Male Lodhran 29-Aug-18 36 Placed Female Multan 29-Aug-18 37 Graduated Female Multan 29-Aug-18 38 Graduated Female Muzaffargarh 29-Aug-18 39 Placed Male Muzaffargarh 29-Aug-18 40 Placed Male Muzaffargarh 29-Aug-18 41 Placed Male Multan 29-Aug-18 42 Placed Female Multan 30-Aug-18 43 Placed Female Multan 30-Aug-18 44 Placed Female Bahawalpur 30-Aug-18 45 Placed Female Bahawalpur 30-Aug-18 46 Placed Male Lodhran 30-Aug-18 47 Placed Male Lodhran 30-Aug-18 48 Graduated Female Bahawalpur 30-Aug-18 49 Placed Male Lodhran 30-Aug-18 50 Placed Female Bahawalpur 30-Aug-18 51 Placed Male Multan 30-Aug-18 52 Placed Female Multan 31-Aug-18 53 Placed Female Muzaffargarh 31-Aug-18 54 Placed Female Muzaffargarh 31-Aug-18 55 Enrolled Male Multan 4-Sep-18 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 72 TABLE 7: KEY INFORMANT INTERVIEWS – COS AND REFLECT GROUPS S. No Respondent Gender CO/REFLECT Group Distict Date of Interview 1 Female Mixed CO Bahawalpur 31-Aug-18 2 Male Reflect group Lodhran 31-Aug-18 3 Female Female CO Lodhran 31-Aug-18 4 Male Reflect group Bahawalpur 31-Aug-18 5 Male Mixed CO Bahawalpur 31-Aug-18 6 Male Mixed CO Lodhran 31-Aug-18 7 Male Reflect group Multan 1-Sep-18 8 Male Male CO Multan 1-Sep-18 9 Male Male CO Muzaffargarh 1-Sep-18 10 Male Reflect group Muzaffargarh 1-Sep-18 11 Male Mixed CO Multan 1-Sep-18 12 Male Mixed CO Muzaffargarh 1-Sep-18 TABLE 8: KEY INFORMANT INTERVIEWS – TRAINING INSTITUTES S. No Respondent Gender Training Institute Type District Date of Interview 1 Male Private Bahawalpur 1-Aug-18 2 Male Public Bahawalpur 1-Aug-18 3 Male Public Muzaffargarh 1-Aug-18 4 Female Private Multan 1-Aug-18 5 Female Public Lodhran 1-Aug-18 6 Female Facilitation center Multan 1-Aug-18 7 Male Private Lodhran 1-Aug-18 8 Male Public Bahawalpur 1-Aug-18 9 Male Public Muzaffargarh 1-Aug-18 10 Male Facilitation center Lodhran 1-Aug-18 11 Female Public Multan 1-Aug-18 12 Male Facilitation center Bahawalpur 1-Aug-18 13 Female Facilitation center Muzaffargarh 1-Aug-18 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 73 TABLE 9: KEY INFORMANT INTERVIEWS – EMPLOYERS S. No Respondent Gender Business/ Trade District Date of Interview 1 Male Poultry Multan 13-Aug-18 2 Male Auto industry Bahawalpur 16-Aug-18 3 Male Footwear Multan 13-Aug-18 4 Female Garments Muzaffargarh 13-Aug-18 5 Male Electronics Bahawalpur 17-Aug-18 6 Male Information technology/data entry Muzaffargarh 15-Aug-18 7 Male Photography Bahawalpur 16-Aug-18 8 Female Garments Multan 17-Aug-18 9 Male Chamber of Commerce Bahawalpur 31-Aug-18 TABLE 10: KEY INFORMANT INTERVIEWS – PROVINCIAL COUNTERPARTS S. No Respondent Gender Location District Date of Interview 1 Male Provincial office - TEVTA Lahore 5-Sep-18 2 Male Provincial office - PVTC Lahore 4-Sep-18 3 Male Provincial office - PSDF Lahore 31-Aug-18 TABLE 11: KEY INFORMANT INTERVIEWS – IP AND SUB-CONTRACTORS’ STAFF S. No Respondent Gender Location District Date of Interview IP-Staff 1 Male Head office Lahore 27-Aug-18 2 Female Field office Bahawalpur 28-Aug-18 3 Male Head office Lahore 27-Aug-18 4 Female Head office Lahore 29-Aug-18 5 Male Head office Lahore 27-Aug-18 6 Male Field office Multan 27-Aug-18 7 Male Head office Lahore 27-Aug-18 Sub-Contractors 1 Male Head office Islamabad 15-Aug-18 2 Female Head office Islamabad 1-Sep-18 3 Male Head office Lahore 4-Sep-18 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 74 Annex 5: Profiles of Core Team Members This annex provides brief profiles of the core team members, i.e., it excludes those who supported the sector specialists in the field and data coding and analysis. Team Leader/Evaluator The team leader is a senior evaluator with nearly 20 years of experience in the development sector across more than 50 countries, including Pakistan. He holds Ph.D. in development studies from the University of California, Berkeley and is currently affiliated there as a senior fellow conducting research on sustainable development and impact assessments. He has participated in more than 50 evaluations, assessments, and monitoring assignments including at least nine assignments related to vocational skills training and development. Sector Specialist in Workforce Development The sector specialist in workforce development is a Pakistani national with an MSc degree in local economic development and a BSc in economics from University of London. He is an accomplished researcher in the field of skill development and youth employment. He has over nine years of experience in research, monitoring, and evaluation and has participated in more than 20 assessment and evaluation assignments including at least five assignments relevant to skills development and employment in Pakistan. Sector Specialist in Community Development The sector specialist in community development is a Pakistani national with an MSc degree in public policy and management from Monash University, Australia. She has more than 20 years’ experience in the development sector in Pakistan, Bangladesh, and Afghanistan. In addition to her program management and implementation experience, she has at least 18 research assignments to her credit, most of which are directly related to community development, workforce education, equity and inclusion, and gender mainstreaming. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 75 Annex 6: Data Collection Instruments KII Guideline for Enrolled Youth SECTION A: RESPONDENT DATA GI(P)1. Interviewer name: GI(P)11. Intervention type: (circle – can choose GI(P)2. Date: multiple) GI(P)3. Respondent name: a. Technical/vocational training GI(P)4. Gender: b. Soft skills training GI(P)5. Education: c. Entrepreneurship training GI(P)6. Age (years): d. Career counselling GI(P)7. Urban/Rural: e. Provision of microfinance loan GI(P)8. Village: f. Provision of toolkit GI(P)9. Union Council: GI(P)12. Stage of participation GI(P)10. District: a. Enrolled b. Graduated but not working c. Placed/self-employed GI(P)13. Name of TI (if applicable) Introduction Assalam-o-Alaikum. Hope you’re having a good day. Thank you for the meeting. My name is ____________. I am a member of a research team that is assessing youth experiences with skills training and participation in community activities. The information from this interview will contribute to improving training programs and socio-economic opportunities for youth. The information you share will only be used for this study and will remain confidential (we will not share it with anyone else), so please express your thoughts freely. We will not use your name in any reports without your prior written permission. GI(P)14. Do I have your consent to proceed with this interview? Yes/No Only interview respondent if he/she provides consent. GI(P)15. Do I have your permission to record this interview for our notes? Yes/No Explain to the respondent that it will be very difficult for you to conduct the interview and take detailed notes simultaneously, if you don’t record the interview. If the respondent declines, do not record, take detailed notes and use the back side of pages or additional pages for notes. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 76 SECTION B: TRAINING COURSE KIIB(E)1. What training course are you enrolled in? KIIB(E)2. When did you enroll in this course? KIIB(E)3. How did you learn about this course? a. Punjab Youth Workforce Development project staff b. Community organization c. Training institute d. Other (specify) KIIB(E)4. Did you receive any information or guidance in choosing this specific course? KIIB(E)5. [if yes to question 4] Did this guidance or information help you choose a course? What additional information would have been helpful? KIIB(E)6. What were your reasons for choosing this specific course? a. To get a job or a better job b. To earn more income c. Social status d. Personal interest e. Other (specify) KIIB(E)7. Can you describe how you were selected for the training? KIIB(E)8. Did you face any challenges in the selection process? If yes, please explain the challenges and how they can be addressed. KIIB(E)9. What are your views about the quality of the training? [Probe for content, its duration, skill of trainer, teaching methodology, learning aids] KIIB(E)10. How can the quality of training be improved? [Probe for content, its duration, skill of trainer, teaching methodology, learning aids] PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 77 SECTION C: WORK PROSPECTS KIIB(E)11. What were you doing before enrolling in the training program? [Probe for employment status and type of job in the three months before enrollment] KIIB(E)12. Do you expect your chances of finding work/finding better work will be better, worse, or the same after you complete the training? Why and how? KIIB(E)13. Do you expect your income to increase, decrease, or stay the same after you complete the training? Why and how? KIIB(E)14. Do you foresee any challenges in finding a job or a better job? Explain. How can these be addressed? SECTION D: CAREER COUNSELING AND MICROFINANCE KIIB(E)15. Did anyone give you advice on how to choose a career? If yes, who gave you the advice? What was the advice? Was it useful? Why or why not? [Probe about career counseling service] KIIB(E)16. Has anyone provided you information on finding jobs yet? If yes, who provided you information? What information was provided? What more information would be helpful on finding a job? [Probe for paid and self-employment] KIIB(E)17. Are microfinance loans available in your area? If yes, what do you know about these loans and how did you learn this? [Probe for who provides loans and eligibility requirements] KIIB(E)18. Do you know how to apply for a microfinance loan? If yes, what is your view about the process? [Probe in terms of convenience and challenges] SECTION E: ATTITUDES KIIB(E)19. In the past 12 months, has your status in your household changed? [Probe for examples: participation in decision making, responsibilities assigned] If yes, how has it changed and what caused the change? PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 78 KIIB(E)20. In the past 12 months, has your status in your community changed? [Probe for examples: participation in decision making, responsibilities assigned] If yes, how has it changed and what caused the change? KIIB(E)21. In the past 12 months, has your understanding of community issues changed? If yes, how has it changed and what caused the change? [Probe in terms of social, economic, political, and environmental issues] KIIB(E)22. In the past 12 months, has your interest and participation in community activities changed? If so, how has it changed and what caused the change? [Probe for social, economic, political, environmental issues] KIIB(E)23. What role do you see for yourself in the community in the future? [Probe for specific roles of community service and leadership] SECTION F: CONCLUSION KIIB(E)24. Is there anything else you would like to tell us about your interaction with the Punjab Youth Workforce Development project? [Probe for successes/effective approaches and possible improvements] PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 79 KII Guideline for Graduated Youth (Not yet employed) SECTION A: RESPONDENT DATA KIIB(P)A. Interviewer name: KIIB(P)A. Intervention type: (circle – can KIIB(P)A. Date: choose multiple) KIIB(P)B. Respondent name: a. Technical/vocational training KIIB(P)C. Gender: b. Soft skills training KIIB(P)D. Age (years): c. Entrepreneurship training KII(TI)A. Education: d. Career counselling KII(TI)B. Urban/Rural: e. Provision of microfinance loan KII(TI)C. Village: f. Provision of toolkit KII(TI)D. Union Council: KIIB(P)A. Stage of participation KII(TI)E. District: a. Enrolled b. Graduated but not working c. Placed/self-employed KIIB(P)B. Name of TI (if applicable) Introduction: Assalam-o-Alaikum. Hope you’re having a good day. Thank you for the meeting. My name is . I am a member of a research team that is assessing youth experiences with skills training and participation in community activities. The information from this interview will contribute to improving training programs and socio-economic opportunities for youth. The information you share will only be used for this study and will remain confidential (we will not share it with anyone else), so please express your thoughts freely. We will not use your name in any reports without your prior written permission. KIIB(P)C. Do I have your consent to proceed with this interview? Yes/No Only interview respondent if he/she provides consent. KIIB(P)D. Do I have your permission to record this interview for our notes? Yes/No Explain to the respondent that it will be very difficult for you to conduct the interview and take detailed notes simultaneously, if you don’t record the interview. If the respondent declines, do not record, take detailed notes and use the back side of pages or additional pages for notes. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 80 SECTION B: TRAINING COURSE KIIB(G)1. I understand that you completed a training course supported by the Punjab Youth Workforce Development project. When did you enroll in this training program? KIIB(G)2. What training course have you completed? KIIB(G)3. How did you learn about this course? a. Punjab Youth Workforce Development project staff b. Community organization c. Training institute d. Other (specify) _____________________________ KIIB(G)4. Did you receive any information or guidance in choosing this specific course? KIIB(G)5. [If yes to question 4] Did this guidance or information help you choose a course? If not, what additional information would have been helpful? KIIB(G)6. What were your reasons for choosing this specific course? a. To get a job or a better job b. To earn more income c. Social status d. Personal interest e. Other (specify) _____________________________ KIIB(G)7. Can you describe how you were selected for the training? KIIB(G)8. Did you face any challenges in the selection process? If yes, please explain the challenges and how can these be addressed? KIIB(G)9. What are your views about the quality of the training? [Probe for content, its duration, skill of trainer, teaching methodology, learning aids] KIIB(G)10. How can the quality of training be improved? [Probe for content, its duration, skill of trainer, teaching methodology, learning aids] PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 81 SECTION C: WORK PROSPECTS KIIB(G)11. What were you doing before enrolling in the training program? [Probe for employment status and type of job in the three months before enrollment] KIIB(G)12. Do you think your chances of finding work/finding better work, are better, worse, or the same as they were before the training? Why and how? KIIB(G)13. Do you expect your income to increase, decrease, or stay the same due to the training that you have completed? Why and how? KIIB(G)14. Are you facing any challenges in finding work/finding better work? If yes, what are the challenges and how can these be addressed? SECTION D: CAREER COUNSELING AND MICROFINANCE (Ask if applicable) KIIB(G)15. Did anyone give you advice on how to choose career? If yes, who gave you the advice? What was the advice? Was it useful? Why or why not? [Probe about career counseling service] KIIB(G)16. Did anyone provide you any support in finding job? If yes, who and what support was provided? Was it useful? Why and why not? [Probe for paid and self-employment] What further support do you need to find a job? KIIB(G)17. Are microfinance loans available in your area? If yes, what do you know about these loans and how did you learn this? [Probe for who provides loans and eligibility requirements] What further information do you need about such loans? KIIB(G)18. Have you applied for a microfinance loan? If yes, have you received the loan? What were the challenges in attaining the loan? How can these challenges be addressed? SECTION E: ATTITUDES KIIB(G)19. In the past 12 months, has your status in your household changed? [Probe for examples: participation in decision making, responsibilities assigned] If yes, how has it changed and what caused the change? KIIB(G)20. In the past 12 months, has your status in your community changed? [Probe for examples: participation in decision making, responsibilities assigned] If yes, how has it changed and what caused the change? PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 82 KIIB(G)21. In the past 12 months, has your understanding of community issues changed? If yes, how has it changed and what caused the change? [Probe in terms of social, economic, political, and environmental issues] KIIB(G)22. In the past 12 months, has your interest and participation in community activities changed? If so, how has it changed and what caused the change? [Probe for social, economic, political, environmental issues] KIIB(G)23. What role do you see for yourself in the community in the future? [Probe for specific roles of community service and leadership] SECTION F: CONCLUSION KIIB(G)24. Is there anything else you would like to tell us about your interaction with the Punjab Youth Workforce Development project? [Probe for successes/effective approaches and possible improvements] PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 83 KII Guideline for Placed Youth SECTION A: RESPONDENT DATA KII(TI)A. Interviewer name: KII(TI)K.Intervention type: (circle – can choose KIIB(P)B. Date: multiple) KIIB(P)C. Respondent name: a. Technical/vocational training KIIB(P)D. Education: b. Soft skills training KIIB(P)E. Gender: c. Entrepreneurship training KII(TI)F. Age (years): d. Career counselling KII(TI)G. Urban/Rural: e. Provision of microfinance loan KII(TI)H. Village: f. Provision of toolkit KII(TI)I. Union Council: KIIB(P)L. Stage of participation KII(TI)J. District: a. Enrolled b. Graduated but not working c. Placed/self-employed KIIB(P)M.Name of TI (if applicable) KIIB(P)N. Name of employer (if applicable) Introduction: Assalam-o-Alaikum. Hope you’re having a good day. Thank you for the meeting. My name is ____________. I am a member of a research team that is assessing youth experiences with skills training and participation in community activities. The information from this interview will contribute to improving training programs and socio-economic opportunities for youth. The information you share will only be used for this study and will remain confidential (we will not share it with anyone else), so please express your thoughts freely. We will not use your name in any reports without your prior written permission. KIIB(P)O. Do I have your consent to proceed with this interview? Yes/No Only interview respondent if he/she provides consent. KIIB(P)P. Do I have your permission to record this interview for our notes? Yes/No Explain to the respondent that it will be very difficult for you to conduct the interview and take detailed notes simultaneously, if you don’t record the interview. If the respondent declines, do not record, take detailed notes and use the back side of pages or additional pages for notes. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 84 SECTION B: TRAINING COURSE KIIB(G)P. I understand that you completed a training course supported by the Punjab Youth Workforce Development project. When did you enroll in this training program? KIIB(G)Q. What training course have you completed? KIIB(G)R. How did you learn about this course? a. Punjab Youth Workforce Development project staff b. Community organization c. Training institute d. Other (specify) KIIB(G)S. Did you receive any information or guidance in choosing this specific course? KIIB(G)T. [If yes to question 4] Did this guidance or information help you choose a course? If not, what additional information would have been helpful? KIIB(G)U. What were your reasons for choosing this specific course? a. To get a job or a better job b. To earn more income c. Social status d. Personal interest e. Other (specify) KIIB(G)V. Can you describe how you were selected for the training? KIIB(G)W. Did you face any challenges in the selection process? If yes, please explain the challenges and how can these be addressed KIIB(G)X. What are your views about the quality of the training? [Probe for content, its duration, skill of trainer, teaching methodology, learning aids] KIIB(G)Y. How can the quality of training be improved? [Probe for content, its duration, skill of trainer, teaching methodology, learning aids] PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 85 SECTION C: WORK PROSPECTS KIIB(G)Z. What were you doing before enrolling in this training program? [Probe for employment status and type of job in the three months before enrollment] KIIB(G)AA. What are you doing now? [Probe for employment status and type of job] KIIB(G)BB. Do you think your chances of finding work/finding better work, are better, worse, or the same as they were before the training? Why and how? KIIB(G)CC. Do you expect your income to increase, decrease, or stay the same due to the training that you have completed? Why and how? KIIB(G)DD. Did you face any challenges in finding new/better job? If yes, what were these challenges and how did you overcome them? Did anyone help you in addressing these challenges? Who and how? SECTION D: CAREER COUNSELING AND MICROFINANCE KIIB(G)EE. Did anyone give you advice on how to choose career? If yes, who gave you the advice? What was the advice? Was it useful? Why or why not? [Probe about career counseling service] KIIB(G)FF. Did anyone provide you any support in finding job? If yes, who and what support was provided? Was it useful? Why and why not? [Probe for paid and self-employment] What further support do you need to find a job? KIIB(G)GG. Are microfinance loans available in your area? If yes, what do you know about these loans and from where did you get this information? [Probe for who provides loans and eligibility requirements] What further information do you need about such loans? KIIB(G)HH. Have you applied for a microfinance loan? If yes, have you received the loan? What were the challenges in attaining the loan? How can these challenges be addressed? SECTION E: ATTITUDES KIIB(G)II. In the past 12 months, has your status in your household changed? [Probe for examples: participation in decision making, responsibilities assigned] If yes, how has it changed and what caused the change? PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 86 KIIB(G)JJ. In the past 12 months, has your status in your community changed? [Probe for examples: participation in decision making, responsibilities assigned] If yes, how has it changed and what caused the change? KIIB(G)KK. In the past 12 months, has your understanding of community issues changed? If yes, how has it changed and what caused the change? [Probe in terms of social, economic, political, and environmental issues] KIIB(G)LL. In the past 12 months, has your interest and participation in community activities changed? If so, how has it changed and what caused the change? [Probe for social, economic, political, environmental issues] KIIB(G)MM. What role do you see for yourself in the community in the future? [Probe for specific roles of community service and leadership] SECTION F: CONCLUSION KIIB(G)NN. Is there anything else you would like to tell us about your interaction with the Punjab Youth Workforce Development project? [Probe for successes/effective approaches and possible improvements] PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 87 Community Organization KII Guideline SECTION A: RESPONDENT DATA KII(CO)A. Interviewer name: KII(CO)B. Date: KII(CO)C. Respondent name: KII(CO)D. Gender: KII(CO)E. Age (years): KII(CO)F. CO name: KII(CO)G. Respondent position in CO: KII(CO)H. Year of formation: KII(CO)I. Urban/Rural: KII(CO)J. Village: KII(CO)K. Union Council: KII(CO)L. District: Introduction: Assalam-o-Alaikum. Hope you’re having a good day. Thank you for the meeting. My name is . I am a member of a research team that is assessing experiences of youth with skills training and participation in community activities. The information from this interview will contribute to improving training programs and socio-economic opportunities for youth. The information you share will only be used for this study and will remain confidential (we will not share it with anyone else), so please express your thoughts freely. We will not use your name in any reports without your prior written permission. KII(CO)M. Do I have your consent to proceed with this interview? Yes/No ______ Only interview respondent if he/she provides consent. KII(CO)N. Do I have your permission to record this interview for our notes? Yes/No _______ Explain to the respondent that it will be very difficult for you to conduct the interview and take detailed notes simultaneously, if you don’t record the interview. If the respondent declines, do not record, take detailed notes and use the back side of pages or additional pages for notes. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 88 SECTION B: YOUTH IDENTIFICATION GI(P)16. Tell us about your community organization? When was it established? Is this a male, female, or mixed organization? GI(P)17. When did your organization begin working with the Punjab Youth Workforce Development Project? Why did you decide to work with this project? GI(P)18. How do you work with the project? How do you support the project and how, if at all, does the project support your organization? [Probe for level of satisfaction with the support] GI(P)19. What role, if any, does your organization play in identifying youth for training? [Probe for males and females separately] GI(P)20. [Ask if the organization has a role in identifying youth for training] Do you have criteria for selecting youth for training? If yes, what are the criteria? What challenges do you face identifying youth that satisfy the selection criteria? How do you address these challenges? [Probe separately for males and females] GI(P)21. What, if at all, does your organization do to ensure that selected youth complete the training course? If you don’t do anything, why not? [Probe for steps like mobilization of parents and youth to discourage absenteeism and dropout, explore separately for males and females] GI(P)22. Do you think your organization will continue to identify youth for training after the Punjab Workforce Development project ends? If yes, how? What steps, if any, is the project taking to ensure that your organization is able to continue identifying youth for training after the end of the project? [probe separately for males and females] GI(P)23. What role, if any, does your organization play in helping youth find work/better work, or start their own businesses? What challenges do youth in your area face in finding work or starting a business? What, if any, role does your CO play in addressing these challenges? Wil your organization continue playing this role after the end of this project? If yes, how? GI(P)24. What, if any, role does your organization play in helping youth obtain microfinance loans? What challenges do youth face in getting such loans? What, if any, role does your CO play in addressing these challenges? Will your organization continue playing this role after the end of the project? If yes, how? PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 89 SECTION C: YOUTH ATTITUDES GI(P)25. Does your organization work to increase youths’ interest and participation in the community? If yes, how do you do? [Probe for the project’s role] [Probe for project activities like Pakistan day, 16 days of activism etc.] [Probe separately for males and females] GI(P)26. In the past 12 months, have you seen any changes in understanding of youth towards community issues? If yes, how has it changed and what caused the change? [Probe in terms of social, economic, political, and environmental issues] [Probe separately for males and females] GI(P)27. In the last 12 months, have you seen any changes in the youth’s interest or participation in community activities? If yes, how has it changed and what caused the change? [Probe in terms of social, economic, political, and environmental issues] [Probe separately for males and females] GI(P)28. Do you think that beneficiary youth have better, worse or the same chances to finding work/better work after completing training course? Why and how? [Probe separately for males and females] [Probe for role of project interventions: training, career counseling, microfinance, toolkit] GI(P)29. Do you think that beneficiary youth are able to earn higher income after completing the training course? Why and how? [Probe separately for males and females] SECTION E: CONCLUSION GI(P)30. Is there anything else you would like to tell us about your interaction with the Punjab Youth Workforce Development project? [Probe for successes/effective approaches and possible improvements] PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 90 REFLECT Group KII Guideline SECTION A: RESPONDENT DATA KII(RE)A. Interviewer name: KII(RE)B. Date: KII(RE)C. Respondent name: KII(RE)D. Gender: KII(RE)E. Age (years): KII(RE)F. CO name: KII(RE)G. Respondent position in REFLECT group: KII(RE)H. Year of formation: KII(RE)I. Urban/Rural: KII(RE)J. Village: KII(RE)K. Union Council: KII(RE)L. District: Introduction: Assalam-o-Alaikum. Hope you’re having a good day. Thank you for the meeting. My name is . I am a member of a research team that is assessing experiences of youth with skills training and participation in community activities. The information from this interview will contribute to improving training programs and socio-economic opportunities for youth. The information you share will only be used for this study and will remain confidential (we will not share it with anyone else), so please express your thoughts freely. We will not use your name in any reports without your prior written permission. KII(RE)A. Do I have your consent to proceed with this interview? Yes/No Only interview respondent if he/she provides consent. KII(RE)B. Do I have your permission to record this interview for our notes? Yes/No Explain to the respondent that it will be very difficult for you to conduct the interview and take detailed notes simultaneously, if you don’t record the interview. If the respondent declines, do not record, take detailed notes and use the back side of pages or additional pages for notes. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 91 SECTION B: FORMATION AND ROLE GI(P)31. How was this REFLECT group formed? What role, if any, did the community have in group formation? How were group members selected? GI(P)32. What were the key challenges in forming your REFLECT group? How can these challenges be addressed? GI(P)33. What is the gender and age composition of this REFLECT group? If the group has representation of both men and women, what are the key benefits and challenges of having a mixed group? GI(P)34. [Ask only if this group has representation of women and youth] What roles do women and youth have in the REFLECT group? GI(P)35. How, if at all, is this REFLECT group involved with the Punjab Youth Workforce Development project? What kind of support does the group receive from the project? Is the support adequate? How can it be improved? GI(P)36. What role does the REFLECT group have in the community? How is it different from the role of a community organization? What are the key challenges in playing this role? How can these challenges be addressed? GI(P)37. Has your REFLECT group helped identify and/or resolve community issues? If so, please explain. What were the key challenges faced by the group in identifying and/or resolving these community issues? How can these challenges be addressed? [Probe about community infrastructure projects identified, if any] GI(P)38. Will your REFLECT group continue to identify and resolve community issues after the end of the Punjab Youth Workforce Development project? If so, how? What further support would you need? GI(P)39. What role, if any, do REFLECT groups play in engaging youth in community activities? If so, please explain? What were the key challenges faced by the group in engaging youth in community activities? How can these challenges be addressed? [Probe for engagement of male and female youth separately] SECTION C: CONCLUSION GI(P)40. Is there anything else you would like to tell us about your interaction with the Punjab Youth Workforce Development project? [Probe for successes/effective approaches and possible improvements] PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 92 Training Institute/ Facilitation Center Head KII Guideline SECTION A: RESPONDENT DATA KII(TI)A. Interviewer name: KII(TI)I. Training type: (circle – can choose multiple) KII(TI)B. Date: a. Technical/vocational training KII(TI)C. Respondent name: b. Soft skills training KII(TI)D. Gender: c. Entrepreneurship training KII(TI)E. Position in TI: d. Career counseling KII(TI)F. TI name: KII(TI)J. Type of institute KII(TI)G.Location: a. Government KII(TI)H.District: b. Private c. PSDF d. Non-profit Introduction: Assalam-o-Alaikum. Hope you’re having a good day. Thank you for the meeting. My name is . I am a member of a research team that is assessing youth experiences with skills training and its impact of their employment opportunities and participation in community activities. The information from this interview will contribute to improving training programs and socio-economic opportunities for youth. The information you share will only be used for this study and will remain confidential (we will not share it with anyone else), so please express your thoughts freely. We will not use your name in any reports without your prior written permission. KII(TI)K. Do I have your consent to proceed with this interview? Yes/No If no, stop interview. KII(TI)L. Do I have your permission to record this interview for our notes? Yes/No Explain to the respondent that it will be very difficult for you to conduct the interview and take detailed notes simultaneously, if you don’t record the interview. If the respondent declines, take detailed notes and use the back side of pages or additional pages if required to write your notes. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 93 SECTION B: TRAINING KII(TI)1. Please tell us about your institute/center. When was your training institute/center established? KII(TI)2. What courses does your institute/ center deliver? Who are your key target beneficiaries? KII(TI)3. When did your institute/center start working with the Punjab Youth Workforce Development project? KII(TI)4. Can you describe how your institute/center works with the project? KII(TI)5. How does your institute identify and enroll youth in project-supported training? Are you satisfied with the process? Why? What are the key challenges in this process? How can it be improved? [Probe for males and females separately] KII(TI)6. Did the project assess the capacity of your institute/center? If yes, how was it conducted? Was it useful? How and why? KII(TI)7. What courses does your institute/center offer for youth enrolled through the Punjab Youth Workforce Development project? How and why were these courses selected? KII(TI)8. How were the courses developed? What role, if any, did the Punjab Youth Workforce Development project play in this process? Was it adequate and useful? What additional support do you need from the project in developing courses? Explain. KII(TI)9. How does your training institute ensure that its curriculum meets the skills gaps of trainees and the requirements of the industry? [Explore for males and females separately] How could the courses be made more responsive to the needs of youth and industry? PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 94 KII(TI)10. Has your involvement with the project enhanced your institute’s/center’s ability to deliver quality training? If yes, how has the project helped? [Probe in terms of curriculum, equipment, training of teachers and linkages with industry] If no, why not? [Probe for whether project assistance was relevant or useful] KII(TI)11. [If the respondent says that project support increased capacity, ask] Will your institute/center be able to maintain this capacity after the project ends? Explain. KII(TI)12. Do you need any additional support from the project to enhance your ability to deliver better quality training? If yes, what support do you need? KII(TI)13. Does the project help your institute/center provide counseling services, apprenticeship, and placement support to trainees? If yes, how? Is the support adequate and useful? How can they further support you in this regard? KII(TI)14. Are graduates of project-supported training able to find work easily? What challenges do they face in this regard? How can the project support them to address these challenges? SECTION C: CONCLUSION KII(TI)15. Is there anything else you would like to tell us about your interaction with the Punjab Youth Workforce Development project? [Probe for successes/effective approaches and possible improvements] PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 95 Employer KII Guideline SECTION A: RESPONDENT DATA KII (Emp)A. Interviewer name: KII (Emp)B. Date: KII (Emp)C. Respondent name: KII (Emp)D. Gender: KII (Emp)E. Name of Company: KII (Emp)F. Nature of Business: KII (Emp)G. Position in company: KII (Emp)H. Location: KII (Emp)I. District: Introduction: Assalam-o-Alaikum. Hope you’re having a good day. Thank you for the meeting. My name is . I am a member of a research team that is assessing youth experiences with skills training and its impact of their employment opportunities and participation in community activities. The information from this interview will contribute to improving training programs and socio-economic opportunities for youth. The information you share will only be used for this study and will remain confidential (we will not share it with anyone else), so please express your thoughts freely. We will not use your name in any reports without your prior written permission. KII (Emp)J. Do I have your consent to proceed with this interview? Yes/No If no, stop interview KII (Emp)K. Do I have your permission to record this interview for our notes? Yes/No Explain to the respondent that it will be very difficult for you to conduct the interview and take detailed notes simultaneously, if you don’t record the interview. If the respondent declines, take detailed notes and use the back side of pages or additional pages if required to write your notes. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 96 SECTION B: PYWD APPROACHES AND SUSTAINABILITY GI(P)41. Does your organization face any problems finding skilled labor? Explain. If yes, how do you address these problems? [Probe the role of training/ vocational institutes] GI(P)42. Are you aware of the Punjab Workforce Development project? If yes, how, if at all, do you engage with the project? [Probe for project support to the company, and the company’s support to the project] GI(P)43. Has your company provided apprenticeships or jobs to youth trained through project￾supported institutes/centers? (particularly females). If yes, in what trades/vocations? GI(P)44. What role, if any, did the Punjab Youth Workforce Development Project play in placing trained youth in apprenticeships or jobs at your company? Are you satisfied with the project’s work in facilitating placements and apprenticeships? Please explain. [probe for suggestions for improvement] GI(P)45. [Ask only if the respondent mentions having hired or apprenticed project-supported trainees] Are you satisfied with the skill level of trainees of the Punjab Youth Workforce Development project? [Probe reasons for level of satisfaction/dissatisfaction where applicable] [Ask and write response as satisfied, dissatisfied or no response/not applicable for each of the following skills] i. Technical skills ii. Soft Skills 1. Computer skills 2. Communication skills 3. Numeracy skills iii. Compliance with rules iv. Safety consciousness v. Quality consciousness vi. Other (specify): GI(P)46. [Ask only if the respondent mentions having hired or apprenticed project-supported trainees] How do PYWD beneficiary graduates compare to graduates of other programs/ institutes that you hire from? Compare strengths and weaknesses? [Probe for technical and soft skills as above] GI(P)47. What more can this project do to address skill gaps and promote employment of youth (particularly females)? Are there any particular skills that you would recommend they focus on? PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 97 GI(P)48. [Ask only if the respondent mentions having hired or apprenticed project-supported trainees] Will your company continue to hire from project-supported training institutes after the project ends? If so, why? GI(P)49. How, if at all, does your company work with project supported training institutes for their capacity building? Please explain. If yes, will you continue to provide similar support in the future? If so, how and why? SECTION D: CONCLUSION GI(P)50. Is there anything else you would like to tell us about your interaction with the Punjab Youth Workforce Development project? [Probe for successes/effective approaches and possible improvements] PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 98 Chambers of Commerce KII Guideline SECTION A: RESPONDENT DATA KII (CoC)A. Interviewer name: KII (CoC)B. Date: KII (CoC)C. Respondent name: KII (CoC)D. Gender: KII (CoC)E. Position in CoC: KII (CoC)F. CoC name: KII (CoC)G. District: Introduction: Assalam-o-Alaikum. Hope you’re having a good day. Thank you for the meeting. My name is . I am a member of a research team that is assessing youth experiences with skills training and its impact of their employment opportunities and participation in community activities. The information from this interview will contribute to improving training programs and socio-economic opportunities for youth. The information you share will only be used for this study and will remain confidential (we will not share it with anyone else), so please express your thoughts freely. We will not use your name in any reports without your prior written permission. KII (CoC)H. Do I have your consent to proceed with this interview? Yes/No If no, stop interview KII (CoC)I. Do I have your permission to record this interview for our notes? Yes/No Explain to the respondent that it will be very difficult for you to conduct the interview and take detailed notes simultaneously, if you don’t record the interview. If the respondent declines, take detailed notes and use the back side of pages or additional pages if required to write your notes. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 99 SECTION B: PYWD APPROACHES AND SUSTAINABILITY GI(P)51. What problems do industries in your area face finding skilled labor? How, if at all, does your chamber help address these problems? GI(P)52. How does your chamber engage with the Punjab Youth Workforce Development project? What support, if any, does your chamber provide to the project? (Probe support/guidance for curriculum, use of equipment, and training of teachers) GI(P)53. What role, if any, does the Punjab Youth Workforce Development project play in helping industry meet its need for skilled labor? How adequate is that role and how can it be improved? GI(P)54. Do you know about the capacity-building support that the Punjab Youth Workforce Development project provides to training institutes to improve training quality? If yes, how adequate is that support and how can it be improved? GI(P)55. Are there any additional courses/skills that you would recommend the project should offer? If yes, what courses/skills and why? GI(P)56. Does your chamber play any role in linking training institutes/centers to industry to better align training to industry needs? How, if at all, has Punjab Youth Workforce Development project played any role in this regard? If yes, does your chamber expect to continue linking training institutes/centers to industry in the future? SECTION C: CONCLUSION GI(P)57. Is there anything else you would like to tell us about your interaction with the Punjab Youth Workforce Development project? [Probe for successes/effective approaches and possible improvements] PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 100 PROVINCIAL COUNTERPART KII GUIDELINE SECTION A: RESPONDENT DATA KII (PC)A. Interviewer name: KII (PC)B. Date: KII (PC)C. Respondent name: KII (PC)D. Gender: KII (PC)E. Position: KII (PC)F. Organization: Introduction: Assalam-o-Alaikum. Hope you’re having a good day. Thank you for the meeting. My name is . I am a member of a research team that is assessing youth experiences with skills training and its impact of their employment opportunities and participation in community activities. The information from this interview will contribute to improving training programs and socio-economic opportunities for youth. The information you share will only be used for this study and will remain confidential (we will not share it with anyone else), so please express your thoughts freely. We will not use your name in any reports without your prior written permission. KII (PC)G. Do I have your consent to proceed with this interview? Yes/No If no, stop interview KII (PC)H. Do I have your permission to record this interview for our notes? Yes/No Explain to the respondent that it will be very difficult for you to conduct the interview and take detailed notes simultaneously, if you don’t record the interview. If the respondent declines, take detailed notes and use the back side of pages or additional pages if required to write your notes. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 101 SECTION B: EFFECTIVENESS OF PYWD APPROACHES AND SUSTAINABILITY GI(P)58. Can you tell me about your organization? What are the organization’s core activities? [Probe with respect to training] How many private and public training organizations do you support? How do you support them? GI(P)59. Do you work with the Punjab Youth Workforce Development project? If yes, how do you work with the project? What, if any, support does your organization receive from the project and what, if any, support does your organization provide to the project? GI(P)60. Has the Punjab Youth Workforce Development project provided any capacity-building support to your training institutes/centers? If so, how and what support did it provide? What role, if any, did your organization have in this process? Was the support adequate and useful? How can the project further support your institutes? (Probe for: curriculum, equipment, teacher training, training methodology, learning aids, industry linkages) GI(P)61. Is there any difference between the Punjab Youth Workforce Development project’s approach to youth training and other training programs your institutes/centers deliver? If yes, what is the difference? How would you compare the strengths and weaknesses of the project￾supported training program approach to that of other training programs? GI(P)62. Has the Punjab Youth Workforce Development project’s support affected the ability of your affiliated institutes/centers to better address the skill gaps of youth and meet the requirement of the industry? How and why? GI(P)63. [Ask only if the respondent answers yes to question 5] Will your affiliated institutes/centers be able to address the skill gaps of youth and meet the requirement of the industry with the same quality standards after the project ends? What steps, if any, is the project taking to ensure this? SECTION C: CONCLUSION GI(P)64. Is there anything else you would like to tell us about your interaction with the Punjab Youth Workforce Development project? [Probe for successes/effective approaches and possible improvements] PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 102 Implementing Partner KII Guideline SECTION A: RESPONDENT DATA KII (IP)A. Interviewer name: KII (IP)B. Date: KII (IP)C. Respondent name: KII (IP)D. Gender: KII (IP)E. Designation: KII (IP)F. Location: Introduction: Assalam-o-Alaikum. Hope you’re having a good day. Thank you for the meeting. My name is . I am a member of a research team that is assessing youth experiences with skills training and its impact of their employment opportunities and participation in community activities. The information from this interview will contribute to improving training programs and socio-economic opportunities for youth. The information you share will only be used for this study and will remain confidential (we will not share it with anyone else), so please express your thoughts freely. We will not use your name in any reports without your prior written permission. KII (IP)G. Do I have your consent to proceed with this interview? Yes/No If no, stop interview KII (IP)H. Do I have your permission to record this interview for our notes? Yes/No Explain to the respondent that it will be very difficult for you to conduct the interview and take detailed notes simultaneously, if you don’t record the interview. If the respondent declines, take detailed notes and use the back side of pages or additional pages if required to write your notes. PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 103 SECTION B: COMMUNITY ENGAGEMENT GI(P)65. What role do community organizations have in the Punjab Youth Workforce Development project? How, if at all, has working with community organizations contributed to achieving project objectives? [Probe the role of male, female, and mixed organizations separately] GI(P)66. How do you select the community organizations you work with? [Probe: how they ensure adequate representation of males and females]. Give examples. Have you encountered any challenges in selecting community organizations and, if so, how have you dealt with these challenges? GI(P)67. Have you encountered any challenges in working with community organizations and, if so, how have you dealt with these challenges? GI(P)68. What role, if any, do the community organizations play in identifying youth for training? [Probe the role in identifying male and female youth separately] GI(P)69. Do you have selection criteria for identifying youth for training? If yes, what are the criteria? What, if any, challenges did you encounter in implementing the selection criteria? If so, how have you addressed these challenges? GI(P)70. Will the community organizations continue encouraging youth to enroll in training after the end of the project? If yes, how will they help them meet the training expenses? What steps, if any, is the project taking to ensure that the organizations can continue encouraging youth to enroll in training? GI(P)71. What role, if any, do the community organizations play in encouraging youth’s interest and participation in community activities? [Probe the role of male, female, and mixed groups separately] Give examples. Do the organizations face any challenges in encouraging youth engagement and, if so, what challenges and how do they address these challenges? GI(P)72. [Ask only if the respondent identifies CO’s role in youth engagement in question 7] Will the community organizations continue to play this role after the end of the project? If so, how? What steps, if any, is the project taking to ensure this? GI(P)73. How and why are REFLECT groups formed? What role, if any, do REFLECT groups play in identifying and resolving community issues? [Probe for men’s and women’s participation in establishing groups and managing group activities]. How, if at all, do REFLECT groups contribute to achieving project objectives? Give examples. GI(P)74. In your opinion, what were the main challenges community organizations faced in forming and working with REFLECT groups? How have they addressed these challenges? PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 104 GI(P)75. In your opinion, will the REFLECT groups continue to play their role after the end of the project? If so, how? What steps, if any, has the project taken to ensure this? GI(P)76. How, if at all, has the project affected economic opportunities for youth? Give examples. [Probe for males and female separately] GI(P)77. How, if at all, has the project affected youth’s attitudes? What attitudes have changed? Give examples. [Probe for males and female separately] SECTION C: INSTITUTIONAL CAPACITY-BUILDING GI(P)78. How do you select the training institutes with which the project works? What challenges, if any, have you encountered selecting partner institutes and how have you addressed these challenges? What are the advantages and challenges of working with each type of institute? [Probe for public institutes, private institutes, non-profit institutes (PSDF), industrial homes] GI(P)79. How do you select training institutes to receive capacity building support? [Probe for processes and criteria] What challenges, if any, have you encountered with this approach and how have you dealt with these challenges? [Probe for the following: public institutes, private institutes, non￾profit institutes (PSDF), industrial homes] GI(P)80. How do you identify and address capacity building needs of training institutes? [Probe for formal needs assessments, setting up partnerships with industry, curricula, equipment, and capacity of trainers]. What, if any, challenges have you encountered with this approach and how have you addressed these challenges? [Probe for formal needs assessments, setting up partnerships with industry, curricula, equipment, and capacity of trainers] GI(P)81. How do you ensure that your capacity-building interventions with training institutes are in line with the needs of the targeted youth (particularly with respect to females) and industry? How, if at all, have institutional capacity building approaches contributed to achieving project objectives? Give examples. What, if any, challenges have you encountered and how have you addressed challenges? Give examples. GI(P)82. How will these institutes continue to be responsive to the needs of youth and industry after the project ends? What steps, if any, are you taking to ensure this? PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 105 SECTION D: TRAINING AND PLACEMENT GI(P)83. How, if at all, do you work with chambers of commerce in your project districts? How, if at all, has your engagement with chambers of commerce contributed to achieving project objectives? Give examples. What, if any, challenges have you encountered in working with chambers and how have you addressed these challenges? GI(P)84. How, if at all, do you work with potential employers? How, if at all, has engagement with employers contributed to achieving project objectives? Give examples. What, if any, challenges have you encountered working with employers and how have you addressed these challenges? GI(P)85. How has the project established or supported career counseling centers? [Probe for process/criteria for selecting sites, management] Who are your target beneficiaries for career counseling? [Probe for process/criteria for selecting sites, management] GI(P)86. What role do the career counselling centers play in the project? How, if at all, have the centers contributed to achieving project objectives? Give examples. What, if any, challenges have you encountered establishing and running these centers and how have you addressed these challenges? How can you improve the function of these centers? GI(P)87. How, if at all, will these centers continue to perform their function after the project ends? What steps is the project taking to ensure this? GI(P)88. What role, if any, does the project play in improving access to and use of microfinance services among project beneficiaries? GI(P)89. How do youth apply for and obtain these loans? What role, if any, does the project play in facilitating this process? GI(P)90. What challenges do youth face in applying for and obtaining these loans? What role, if any, has project played in addressing these challenges? GI(P)91. Will youth continue to have access to microfinance loans after the project ends? What steps, if any, are you taking to ensure this? SECTION E: CONCLUSION GI(P)92. Is there anything else you would like to tell us about your work in the project? [Probe for successes/effective approaches and possible improvements] PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 106 Annex 7: Documents Reviewed PYWD. (2017, June 01). Monitoring & Evaluation Plan PYWD. (2017, November 20). Annual Progress Report: October 2016 to September 2017 PYWD. (2017, January 16). Quarterly Progress Report: October 1 to December 31, 2016 PYWD. (2017, April 14). Quarterly Progress Report: January 1 to March 31, 2017 PYWD. (2017, July 15). Quarterly Progress Report: April 1 to June 30, 2017 PYWD. (2018, January 16). Quarterly Progress Report: October 1 to December 31, 2017 PYWD. (2018, April 25). Quarterly Progress Report: January 1 to March 31, 2018 PYWD. (2017, February 20). Annual Work Plan (Year One): October 1, 2016 to September 30, 2017 PYWD. (2017, October 23). Annual Work Plan (Year Two): October 1, 2017 to September 30, 2018 PYWD. (2018). Beneficiary Data Sheet PYWD. (2018). List of Active Community Organizations and REFLECT Groups PYWD. (July 2018). TEVET Sector Skills Gap Analysis Benazir Income Support Program, Population of Pakistan, An Analysis of NSER 2010-11, Youth and Employment Participation Management Systems International. Study to Explore New Opportunities for Partnership and Programming in South Punjab: Research Report. 2017. Unpublished report Pakistan Bureau of Statistics, Household Integrated Economic Survey 2015-16 Pakistan Bureau of Statistics, Province Wise Provisional Results of Census – 2017 United Nations Development Program, Pakistan Human Development Index Report – 2017 PUNJAB YOUTH WORKFORCE DEVELOPMENT PROJECT MIDTERM EVALUATION 107 Annex 8: Sampling This annex describes the final sample of beneficiary respondents and key informants interviewed by the evaluation team. Beneficiaries were purposively selected to give appropriate representation to training type, districts, and gender. An equal number of community organizations and REFLECT groups were selected from each district while the rest of the key informants were selected based on their role in the project. TABLE 12: SAMPLE BENEFICIARY RESPONDENTS BY DISTRICT District Male Female Total Bahawalpur 7 5 12 Lodhran 8 4 12 Multan 8 11 19 Muzaffargarh 3 9 12 Total 26 29 55 TABLE 13: SAMPLE BENEFICIARY RESPONDENTS BY STATUS Training Type Enrolled Graduated but not in Work Placed or Self Employed Total Technical training 9 9 20 38 Soft skills 0 2 3 5 Entrepreneurship 0 2 7 9 Career counselling 0 0 3 3 Total 9 13 33 55 TABLE 14: KEY INFORMANT INTERVIEW RESPONDENTS AT DISTRICT AND PROVINCIAL LEVELS Respondents Districts Level Provincial Level Total Bahawalpur Lodhran Multan Muzaffargarh Training institutes 4 3 4 2 0 13 Employers and chambers of commerce 5 0 3 1 0 9 COs and REFLECT groups 3 3 3 3 0 12 Provincial counterparts 0 0 0 0 3 3 IP and sub-contractors 1 0 1 0 8 10 Total 13 6 11 6 11 47 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523