11 PPL LER M&E IDIQ Contract No.: AID-OAA-I-15-00015 Task Order Contract No.: 72060818F00004 May 17, 2018 This document was prepared for review by the United States Agency for International Development. It was prepared by Banyan Global under the PPL LER M&E IDIQ, Task Order 72060818F00004. MIDTERM PERFORMANCE EVALUATION OF USAID CAREER CENTER ACTIVITY FINAL EVALUATION REPORT MAY 2018 Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report i This publication was produced for the United States Agency for International Development, USAID Contract Number AID-OAA-I-15-00015/72060818F00004. It was prepared by Banyan Global through the PPL LER Monitoring and Evaluation (M&E) IDIQ. Implemented by: Banyan Global 1120 20th Street NW, Suite 950 Washington, DC 20036 Phone: +1 202-684-9367 Fax: +1 202-697-5020 Disclaimer: This report is made possible by the support of the American People through the United States Agency for International Development (USAID). The contents of this report are the sole responsibility of Banyan Global and do not necessarily reflect those of USAID or the United States Government. Project Description: This midterm performance evaluation is for the USAID Career Center Activity in Morocco, implemented by Banyan Global under the Policy, Planning and Learning – Learning, Evaluation and Research Monitoring and Evaluation indefinite delivery indefinite quantity contract.1 The task order’s objective was to analyze the USAID Career Center’s performance at the midway point and to make action-oriented, evidence-based recommendations to USAID/Morocco’s Office of Economic Growth and the implementing partners to improve ongoing and future programming. Specifically, this evaluation assessed progress to date against planned results; captured and analyzed stakeholder feedback to inform the USAID Career Center of its perceived value to different stakeholders; made recommendations to improve its value proposition by adjusting services; and informed the USAID Career Center on necessary adaptations to improve the model’s sustainability and better support career services in Morocco. This evaluation, extending from February 27 through May 7, 2018, covered the USAID Career Center’s award date through December 2017. The USAID/Morocco Mission, specifically the Office of Economic Growth, will use the analysis to guide the direction of the USAID Career Center in the remaining activity life. The implementing partners (FHI 360, IREX, EFE, and Golden Resources) will learn about the strengths and weaknesses of activities to date and adjust the upcoming workplan based on the evaluation recommendations. Recommended Citation: Banyan Global, 2018. Mid-term Performance Evaluation of USAID Career Center Activity Final Evaluation Report. USAID. Mid-term Performance Evaluation of USAID Career Center Task Order. 1 The USAID-funded Career Center, AID-608-C-15-00004, implemented by FHI 360 in partnership with IREX, EFE, and Golden Resources is valued at $23.8 million and spans a period of performance from February 20, 2015 through February 11, 2020. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report ii MIDTERM PERFORMANCE EVALUATION OF THE USAID CAREER CENTER ACTIVITY FINAL EVALUATION REPORT MAY 2018 PPL LER M&E IDIQ Contract No.: AID-OAA-I-15-00015 Task Order Contract No.: 72060818F00004 Submitted to: Craig Smith Task Order Contracting Officer USAID, Office of Acquisition and Assistance, MERP Kurt Krausse Task Order Contracting Officer Representative USAID/Morocco/PMO Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report iii TABLE OF CONTENTS Acronyms....................................................................................................................................................................... iv Abstract........................................................................................................................................................................... v EXECUTIVE SUMMARY......................................................................................................... vi BACKGROUND ........................................................................................................................ 1 The Development Problem.......................................................................................................................................1 Overview of the USAID Career Center................................................................................................................1 EVALUATION PURPOSE........................................................................................................ 1 evaluation Questions..................................................................................................................................................2 EVALUATION METHODOLOGY.......................................................................................... 2 Data Collection Methods and Sampling .................................................................................................................2 Data Analysis.................................................................................................................................................................5 Limitations.....................................................................................................................................................................5 FINDINGS.................................................................................................................................. 6 Evaluation Question 1: Workforce Readiness......................................................................................................6 Evaluation Question 2: Validation of the Career Center Model....................................................................14 Evaluation Question 3: Sustainability and scalability..........................................................................................18 CONCLUSIONS AND RECOMMENDATIONS.................................................................. 21 Evaluation Question 1: Workforce Readiness....................................................................................................21 Evaluation Question 2: Validation of the Career Center Model....................................................................23 Evaluation question 3: Sustainability and Scalability...........................................................................................24 ANNEX 1: EVALUATION DESIGN MATRIX ..................................................................... 26 ANNEX 2: FINDINGS MATRIX............................................................................................ 34 ANNEX 3: YOUTH SURVEY RESULTS .............................................................................. 68 ANNEX 4: LIST OF PARTICIPANTS................................................................................... 75 ANNEX 5: DESK STUDY BIBLIOGRAPHY ........................................................................ 76 ANNEX 6: PROJECT PROGRESS AGAINST MELP INDICATORS ................................. 78 ANNEX 7: EVALUTION SCOPE OF WORK ...................................................................... 92 ANNEX 8: SUMMARY OF EVALUATION TEAM ............................................................ 108 ANNEX 9: DATA COLLECTION PROTOCOLS.............................................................. 111 ANNEX 10: CONFLICT OF INTEREST DISCLOSURE FORMS..................................... 149 Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report iv ACRONYMS ANAPEC National Agency for the Promotion of Jobs and Skills (Agence Nationale de la Promotion de l’Emploi et des Compétences) CSO Civil society organization EFE Education for Employment ENSA National School of Applied Sciences EQ Evaluation question FGD Focus group discussion FHI-360 Family Health International 360 FST Faculté des Sciences et Techniques GOM Government of Morocco IREX International Research and Exchanges Board ISTAHT Institut Spécialisé de Technologie Appliquée Hôtelière et Touristique KI Key informant KII Key informant interview M&E Monitoring and Evaluation MELP Monitoring, Evaluation and Learning Plan MOU Memorandum of Understanding OFPPT Office for Professional Training and Job Promotion (Office de la Formation Professionnelle et de la Promotion du Travail) OPI Organizational performance index SME Small and medium-sized STEM Science, technology, engineering, and math TOT Training of trainers USAID United States Agency for International Development VCC Virtual career center VSME-SME Very small, small and medium-sized enterprises Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report v ABSTRACT Banyan Global conducted a mid-term performance evaluation of the United States Agency for International Development (USAID) Career Center Activity2 under the Policy, Planning, and Learning￾Learning, Evaluation, and Research Monitoring and Evaluation indefinite delivery/indefinite quantity contract. USAID/Morocco designed the USAID Career Center, a four-year program with an additional option year, to bolster the Government of Morocco’s efforts to assist youth to transition from education to employment. The evaluation’s objective was to review progress toward achieving intended results and outcomes as well as to provide USAID/Morocco’s Office of Economic Growth and the USAID Career Center’s implementing organizations with evidenced-based recommendations for possible programmatic adjustments to improve career center services, strengthen its relevance to beneficiaries and key stakeholder and promote sustainability in the time remaining for implementation. The USAID Career Center has made substantial progress towards achieving its primary objectives. In a short period of time, it has succeeded to create youth-friendly career centers, and a virtual career center that largely correspond to youth expectations and needs. Based on feedback from youth and the private sector, the USAID Career Center helps participants acquire needed work readiness and soft skills. The current model’s strengths include the provision of small group and individualized services and connecting students and trainees to the private sector. The evaluation also highlights the model’s limitations, including the ability to serve the centers’ host institutions’ large number of students and trainees across multiple campuses and the centers’ Najahi work readiness training program’s relatively strong focus on job search skills at the expense of some of the broader, foundational soft skills training modules developed by the USAID Career Center. Other workshops offered in the career centers address a fuller range of soft skills and are partially addressing the limitation of Najahi core training, as it is currently being delivered. Additionally, the evaluation identified several issues that are outside the scope of the USAID Career Center but that may be worth consideration for the future, most notably the delivery of timely orientation services to enable students to benefit from career counselling at an earlier stage than provided in the career centers. The evaluation also found that USAID Career Center successfully tested the career center model during the initial start-up of the project to ensure relevance to Moroccan stakeholders using a variety of participatory methods. Overall, the USAID Career Center’s practices for informing and obtaining feedback from key stakeholders have been effective, although additional efforts are needed to more fully engage some stakeholders (including Government of Morocco counterparts and university host institutions) in the model’s evolution. More evaluations and assessments are required at this stage to ensure the model’s fuller adaptation to the Moroccan context. At the midpoint of implementation, USAID Career Center has established a good foundation for the sustainability and future scalability of the model. While the USAID Career Center and its institutional partners are discussing potential solutions to replace USAID funding and expertise, these alternatives need to be more fully modeled and tested. The evaluation team found that while the host institutions are likely to sustain and replicate the six existing career centers, challenges exist to maintaining the quality of services after USAID’s assistance ends. 2 Throughout this document the shortened term USAID Career Center is used in place of USAID Career Center Activity. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report vi EXECUTIVE SUMMARY The United States Agency for International Development, Morocco’s (USAID/Morocco) Office of Economic Growth engaged Banyan Global to conduct a mid-term performance evaluation of the USAID Career Center Activity. 3 The objective was to review progress toward achieving intended results/outcomes, identify lessons learned and recommend possible programmatic adjustments at the midway point of implementation. This evaluation covers the launch of the USAID Career Center in February 2015 through December 2017. USAID/Morocco designed the USAID Career Center, a four-year program with an additional option year, to bolster the Government of Morocco’s (GOM) efforts to assist youth to transition from education to employment. The USAID Career Center is working toward the following objectives: • establishing career centers in three universities and three vocational training institutes in Casablanca, Marrakech and Tangiers, as well as a Virtual Career Center (VCC); • providing customized work readiness courses; • fostering a network to engage the private sector; and • developing a career center model that is transferable and sustainable. The program (AID-608-C-15-00004), implemented by Family Health International 360 (FHI 360) in partnership with the International Research and Exchanges Board (IREX), Education for Employment (EFE), and Golden Resources, was contracted at $23.8 million on February 20, 2015. The USAID Career Center is implemented in close coordination with Morocco’s Ministry of National Education, Vocational Training, Higher Education and Scientific Research. METHODOLOGY An independent evaluation team, consisting of five members (three international and two Moroccan nationals, four women and one man), conducted fieldwork in Casablanca, Marrakech, Rabat and Tangiers over three weeks in March and April 2018. This evaluation used mainly qualitative methods consisting of document review, direct observation, key informant interviews (KIIs), focus group discussions (FGDs), and stakeholder workshops to inform the evaluation questions (EQs). In addition, the evaluation team administered youth beneficiary and private sector partner online surveys. As described in the body of the evaluation, due to the low rate of return on the private sector survey, the evaluation team does not refer to this data in its analysis. Limitations. The evaluation encountered a few limitations in conducting the mid-term performance evaluation of the USAID Career Center. Selection bias is an inherent risk when implementers help to facilitate contact with beneficiaries. To mitigate the risk of selection bias, the team randomly selected participants from the USAID Career Center’s beneficiary lists; only those selected were contacted by career center staff and invited to attend FGDs. The timeframe when the evaluation was conducted also posed a limitation as it corresponded with a university exam period, an internship period for some vocational training institutions, and the eve of a one-week spring break for university and vocational institutions. Collecting data on beneficiaries’ perceptions of past activities partially mitigated this limitation. The evaluation team also experienced difficulties arranging meetings with private sector partners and only were able to meet with a small number of companies. Furthermore, despite the large sample size and persistent communication, the evaluation received low response rates to online surveys. 3 Throughout this document the shortened term USAID Career Center is used in place of USAID Career Center Activity. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report vii EVALUATION QUESTIONS This mid-term performance evaluation focused on the following EQs: 1. To what extent are USAID Career Center participants – men and women between the ages of 15 and 29 – acquiring the work readiness and soft skills required by the private sector? What are the enabling and disabling factors contributing to workforce-ready students? 2. How effectively has USAID Career Center tested and validated the career center model to ensure relevance to Moroccan stakeholders (students and their immediate environment, educators, government counterparts, private sector, and civil society organizations [CSOs])? 3. How likely are the career centers to be sustainable? FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS In a short period of time, the USAID Career Center has succeeded to create youth-friendly centers and a virtual career center that largely correspond to youth expectations and needs. Based on feedback from youth and the private sector, the USAID Career Center is helping participants in its activities to acquire needed work readiness and soft skills. The current model’s strengths include the provision of small group and individualized services and connecting students and trainees to the private sector. The evaluation highlights the model’s current limits to serve the center host institutions’ large numbers of students and trainees spread across multiple campuses, and the centers’ Najahi work readiness training program’s relatively strong focus on job search skills at the expense of some of the broader, foundational soft skills training modules. Other workshops offered in the career centers address a fuller range of soft skills and are partially addressing the limitations of Najahi core training, as it is currently being delivered. Additionally, the evaluation identified several issues that are outside the scope of the USAID Career Center but may be worth consideration for the future- most notably the delivery of timely orientation services. Key recommendations include the following: 1. Expand the foundational Najahi training core curriculum. USAID Career Center should integrate existing foundational soft skills modules such as “Managing my Emotions and my Stress,” “Working as a Team,” “Me and Problem Solving” and “Negotiate with Gusto” into the core (versus supplemental) curriculum of Najahi training and advocate for expanding planned institutional offerings to likewise include these foundational modules. (High priority, to be addressed in the short term) 2. Elevate business developers as key personnel alongside career counselors. USAID Career Center should integrate the post of business developer as key center personnel alongside the career counselors to ensure strong linkages with the private sector and CSOs. (High priority to be addressed in the short term) 3. Assess private sector demand for career center services among small and medium-sized enterprises (SME)s. USAID Career Center has tested the model and engaged with structured, large and multinational companies in its pilot activities. Additional engagement with SMEs would be useful to more fully test the model with smaller companies that make up a significant portion of the Moroccan economy and employers. (High priority, to be addressed in the short term) EQ1: To what extent are USAID Career Center participants –men and women between the ages of 15 and 29 – acquiring the work readiness and soft skills required by the private sector? What are the enabling and disabling factors contributing to workforce-ready students? Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report viii 4. Build the capacity of host institutions to strengthen their institutional communication activities to mobilize youth. To mobilize a greater number of service users, USAID Career Center should advocate with host institutions to strengthen institutional communication activities by including information about the center in orientation materials, on the institutions’ websites, in existing outreach activities, and within faculty and trainer orientation and directives. The program should maintain and expand current practices of joint activities with student clubs, youth-serving CSOs, and existing university and training institution entrepreneurship programs to make existing centers multi￾service trainee and student hubs. (Medium level priority, to be addressed in the medium term) USAID Career Center has successfully tested the career center model to ensure relevance to Moroccan stakeholders. The evaluation found that the USAID Career Center used a variety of participatory methods to engage Moroccan stakeholders in the design and start-up phase. Overall, the USAID Career Center’s practices for informing and obtaining feedback from key stakeholders have been effective, although additional efforts are needed to more fully engage some key stakeholders, including central GOM institutions, EFE trainers, ANAPEC and SMEs in the model’s evolution. As the USAID Career Center enters the mid-point of its performance, additional evaluations and assessment are required to ensure the model’s continued adaptation to the Moroccan context and to the size of student bodies, geographic dispersion of campuses, weaknesses in secondary school orientation4, structure of Morocco’s economy, and funding available from host institutions. Key recommendations include the following: 5. Strengthen the strategic planning and oversight roles of its institutional partners for center management. Continue the emerging good practices observed among some universities, reinforcing the institutional hierarchy’s oversight of the centers through regular meetings and ongoing reporting. The role of regional coordinator for center oversight should be redefined and limited to transferring capacity to career center staff. (High priority, to be addressed in the short term) 6. Engage key stakeholders in activities to evaluate the current career center model. The USAID Career Center should continue and expand its collaboration efforts, engaging with key stakeholders to monitor the career center model’s performance. Members of the steering committee should be involved through center visits, FGDs with beneficiaries, review of internal evaluation data, and validation of assessment reports modeling center sustainability. (High priority, to be addressed in the medium term) 7. Consider ways to maintain the current model of delivering soft skills and work readiness training through extra-curricular services after USAID funding ends. To reach undergraduates and preserve the current catalogue of center services in universities and vocational training institutions, USAID Career Center should consider ways to maintain the current model of delivering soft skills and work readiness training through extra-curricular services after USAID funding ends. Training services should be delivered through the career centers using center personnel, through partnerships between the center and external partners, and/or through faculty with specialized training. (High priority, to be addressed in the medium term) 4 Providing orientation services to secondary school students is outside the scope of the current Activity. The lack of strong orientation services provided at the secondary school level limits the effectiveness of career center orientation services because beneficiaries have already made career path decisions that are difficult to change given the relative inflexibility of the Moroccan education system. EQ2: How effectively has USAID career center tested and validated the career center model to ensure relevance to Moroccan stakeholders (students and their immediate environment, educators, government counterparts, private sector, and civil society organizations [CSOs])? Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report ix At the mid-point of implementation, USAID Career Center has established a good foundation for the model’s sustainability and future scalability. The USAID Career Center and its institutional partners are discussing potential solutions to replace USAID funding and expertise, that need to be more fully modeled and tested. While host institutions are likely to sustain and replicate the six existing career centers, some significant challenges exist to maintaining the quality of services after USAID assistance ends. Key recommendations include the following: 8. Develop a business plan that estimates the cost of maintaining the USAID Career Center services post USAID funding and assesses alternative income generation strategies. The business model should be developed in collaboration with institutional partners taking into consideration how similar activities are managed (for example current income generating activities in the area of continuing education). (High priority, to be addressed in the short term) 9. Before continuing the development of the VCC, the post-USAID Career Center management model should be determined. Future development of online services should be in collaboration with the entities that will assume the service’s management. In this way, the VCC will develop according to the capacity of its future owner(s) to manage its evolution. (High priority, to be addressed in the short term) 10. Upon identifying an appropriate VCC management model, consider the following improvements: – Improving the website’s design and functionality and offering content in Arabic; – Integrating online modules with university e-learning strategies and potential strategies to offer credits for soft skills courses.5 – Consider blended learning models that integrate learning on the VCC with Najahi training programs, which is currently a goal for year four; and, – Reinforcing social networking strategies. (Medium priority, to be addressed in the medium term) 11. Build the capacity of each university to sustain and scale the career center model. To ensure strong buy-in from university leadership, USAID Career Center should focus their sustainability strategy on building the capacity of each university to sustain and scale the career center model within the university structure. USAID Career Center should continue collaboration with the State Secretariat for Higher Education to scale the model within other universities. In the case of the OFPPT and Ministry of Tourism, it should continue to work with the central office to develop a plan to replicate the model in additional sites. (High priority, to be addressed in the short term) 5 The universities’ e-learning strategy is still in its early stages. Currently, a few massive open online courses are offered, but without academic credit. Cadi Ayyad University in Marrakech has discussed an e-learning strategy, first, however, it must receive accreditation for its online courses, which will be in 2019 at the earliest. EQ3: How likely are the career centers to be sustainable? Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 1 BACKGROUND THE DEVELOPMENT PROBLEM Young Moroccans comprise approximately 10 million of the 35 million total population and 80 percent of the unemployed in the country. According to Morocco’s High Commission for Planning, the youth unemployment rate reached an all-time high of 25.5 percent in the first quarter of 2017. Unemployment rates in Morocco rise with education levels, leaving many young university and vocational training graduates with a choice between unemployment and emigration: approximately 4.5 million Moroccans reside and work abroad (European Union, 2017). In the past 15 years, Morocco has undertaken a number of economic and social reforms aimed at increasing productivity and improving living standards (World Bank, 2017). Youth employability continues to be a major focus of King Mohammed VI; the Government of Morocco’s (GOM) Integrated National Youth Strategy (2015-2030) aims to increase economic opportunities for youth and promote their employability. Despite these efforts, however, many graduates lack the skills and information to choose career paths that fit their education and interests and match labor market needs. OVERVIEW OF THE USAID CAREER CENTER The United States Agency for International Development (USAID) designed the USAID Career Center, a four-year project with an additional option year, (award number AID-608-C-15-00004, February 20, 2015 – February 11, 20202, $23.8 million) to bolster the GOM’s efforts to assist youth to transition from education to employment. The USAID Career Center, implemented by FHI 360 in partnership with IREX, EFE, and Golden Resources, was launched in 2015 and is implemented in close coordination with GOM counterparts, specifically the Ministry of National Education, Vocational Training, Higher Education and Scientific Research. It is working towards: • Establishing career centers in three universities and three vocational training institutes in Casablanca, Marrakech and Tangiers, as well as a Virtual Career Center (VCC); • Providing customized work readiness courses; • Fostering a network to engage the private sector; and • Developing a career center model that is transferable and sustainable. The USAID Career Center theory of change posits that if youth receive counseling, work readiness training and better information on changes in private sector demand, their choice of career pathways will be better founded, and their expectations will be more in line with that sector’s expectations. EVALUATION PURPOSE The purpose of the USAID Career Center midterm performance evaluation, implemented by Banyan Global, was to analyze the USAID Career Center’s performance at the midway point and make action￾oriented, evidenced-based recommendations to USAID/Morocco’s Office of Economic Growth and the USAID Career Center’s implementing partners to improve ongoing and future programming. The evaluation reviewed the progress of the USAID Career Center towards achieving the planned results/outcomes; captured and analyzed stakeholder feedback to inform the USAID Career Center of its Core Services to be Provided by the Career Centers • Diagnostic tools and resources to help youth clarify their career interests and discover their potential; • Information to help them understand possible career pathways; • Connection with the private sector; • Preparation for work readiness (soft skills and job search skills); and • Opportunities to interact and learn real-time in the work environment. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 2 perceived value to different stakeholders; and provided recommendations and possible programmatic adaptations to improve its value proposition, sustainability, and stability, ultimately leading to improved support of career services in Morocco. This evaluation covered the program’s award date, February 20, 2015, through December 31, 2017. The USAID/Morocco Mission, specifically the Office of Economic Growth, will use the findings of this evaluation to guide the USAID Career Center’s direction in the remaining timeframe. The implementing partners, FHI 360, IREX, EFE, and Golden Resources, will have the opportunity to use findings to adjust programming, if warranted, based on the evaluation recommendations. EVALUATION QUESTIONS This mid-term performance evaluation focused on the following three evaluation questions (EQs): 1. To what extent are USAID Career Center participants– men and women between the ages of 15 and 29– acquiring the work readiness and soft skills required by the private sector? What are the enabling and disabling factors contributing to workforce-ready students? 2. How effectively has USAID Career Center tested and validated the Career Center model to ensure relevance to Moroccan stakeholders (students and their immediate environment, educators, government counterparts, private sector, and civil society organizations [CSOs])? 3. How likely are the career centers to be sustainable? EVALUATION METHODOLOGY The mid-term performance evaluation of the USAID Career Center in Morocco, extending from February 27 through May 7, 2018, applied a mixed method, participatory approach, drawing on qualitative and quantitative data to inform the three EQs. This approach allowed the evaluation team to meet the evaluation’s aforementioned objectives. Additionally, the team ensured that the evaluation applied a gender and youth sensitive, locally nuanced approach grounded in USAID’s Evaluation Policy and best practices in research and positive youth development. DATA COLLECTION METHODS AND SAMPLING The evaluation team, consisting of five members (three international and two Moroccan nationals, four women and one man), conducted fieldwork over three weeks in March and April 2018. The evaluation collected data in the three main cities where USAID Career Center conducts activities, namely Casablanca, Tangiers, and Marrakech, and in each of the six career centers the USAID Career Center established. The evaluation team also interviewed key stakeholders in Rabat, where the program office and some national level GOM partners are located. This evaluation used mainly qualitative methods consisting of document review, direct observation (DO), key informant interviews (KIIs), focus group discussions (FGDs), and a stakeholder workshop to inform the EQs. In addition, the evaluation team administered two online surveys: one of youth career center participants (including VCC) and the other of USAID Career Center’s private sector partners. Desk Study. The evaluation team reviewed a selection of USAID Career Center documents, and the content on the VCC as well as secondary materials relevant to youth employability in Morocco and elsewhere. The document review informed the development of the evaluation’s data collection protocols and contributed to the EQ findings, supplementing and cross-checking information contained in the USAID Career Center reports and gathered through the evaluation’s tools and instruments. A full list of documents reviewed is in annex 5. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 3 Key Informant Interviews. Seventy-two key informants (KIs) (31 female, 41 male) representing USAID, the USAID Career Center implementing partners (managers, specialists and trainers), career center personnel, national and regional GOM counterparts, university and training center administration, and representatives of private sector and civil society partners were interviewed as part of the evaluation (see Figure 1). The sampled KIs were purposively selected, using the following criteria: involvement with the USAID Career Center and/or their role in sustaining the activity’s interventions at the conclusion of their contract. The team selected civil society organizations (CSOs) working on gender and persons with disabilities to gain insight as to what extent the USAID Career Center has collaborated and integrated social inclusion considerations into programming design and implementation. Interviews lasted between one and two hours and were conducted in French or English. Figure 1. Key Informant Interviews Rabat Casablanca Tangiers Marrakec h Total F M F M F M F M F M USAID and Implementing Partners Managers and Specialists 3 3 2 1 1 1 2 - 8 5 Career Centers Staff - - 3 3 3 2 3 3 9 8 EFE Trainers - - 5 3 - - 2 4 7 7 University/Training Center Administration - - 1 2 - 2 - 1 1 5 Government of Morocco Representatives 1 2 1 - - - - 1 3 Private Sector Employers - - 1 1 2 1 - 3 3 5 Intermediary Organizations (Professional Associations and CSOs) Representatives 1 - - 2 1 4 - 2 2 8 Sub-Total 5 5 12 13 7 10 7 1 3 31 41 Total 10 25 17 20 72 Focus Group Discussions. The evaluation team conducted FGDs with three categories of beneficiaries: youth, participants in USAID Career Center training of trainers (TOT) workshops, and participants in an outreach activity in collaboration with a CSO partner. The evaluation team held separate FGDs with each beneficiary category. Youth FGDs were further organized in each of the existing six career centers by: participants of the three to five-day soft skills training, workshops, individualized counseling, and job fairs (segmented by university or vocational training center beneficiaries), youth ambassadors (mixed group of vocational and university ambassadors), and summer camp participants (university students only). In an effort to provide an environment for female participants to speak freely on specific issues affecting women entering the job market and the career centers’ response, two FGDs, one in Marrakech and one in Tangiers, were female-only. The evaluation team organized a FGD with CSO leaders and youth who took part in activities serving persons with disabilities to assess career center outreach activities as well as one of its strategies to ensure access by youth with special needs. The team also organized a FGD with six participants in Najahi TOT programs in order to assess activities to integrate soft skills training in partner institutions’ core curriculum. Unfortunately, Office of Professional Training and Job Promotion (OFPPT) TOT participants were not available to meet during the evaluation period. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 4 To facilitate ease of expression among youth participants, whose French and/or English language skills varied, FGDs with youth were conducted primarily in Moroccan Arabic. FGDs took place in career center meeting rooms during regular center hours, lasting between 1.5 and 2 hours. No career center personnel were present during FGDs, so participants would feel free to share their impressions of the center and its services. Figure 2. Focus Group Discussions Casablanca Tangiers Marrakech Total F M F M F M F M University youth participants in career center activities (training, workshops, job fairs, etc.) 6 4 3 - 2 6 11 10 Vocational youth participants in career center activities (training, workshops, job fairs, etc.) 2 5 6 5 5 3 13 13 Youth participants in summer camp (university only) 3 1 - - 2 1 5 2 Youth Ambassadors (mixed vocational and university) 5 4 2 6 6 9 13 19 Najahi training of trainers 6 4 5 - 2 6 13 10 Instructors/Participants in Outreach Events - - 5 - 1 - 6 - Total 22 18 21 11 18 25 61 54 Online Surveys. With logistical support from the USAID Career Center, the evaluation team administered two online surveys, one to youth beneficiaries and the other to private sector partners.6 The purpose of the surveys was to complement qualitative data gathered in face-to-face interviews with the USAID Career Center stakeholders with quantitative data using a survey instrument. The online surveys are in annex 9. Youth survey: The online survey was structured around three areas: career centers user profile information, utilization of services, and, respondents’ perceptions of the career centers, and the impact of services. USAID Career Center sent this survey via email to 10,000 youth, split evenly between males and females, who had participated in center activities and/or were registered users of the VCC.7 The email survey sampling consisted of 2,500 beneficiaries per city hosting a career center, divided evenly between vocational and university career center participants and an additional 2,500 youth from elsewhere in Morocco (assumed to be VCC users). In addition to the email campaign, the link to the survey was posted on each of the six career center’s Facebook pages. Despite the large sample size, the evaluation team only received 218 responses to this online survey Private sector survey: To collect feedback from USAID Career Center private sector partners, the evaluation team sent an online survey to 66 representatives of companies using contact information USAID Career Center staff provided. Unfortunately, only two enterprises responded to the survey. As a result of this response rate, the evaluation team determined that the survey results were not representative of the sample group and were not included in the evaluation. 6 Due to Moroccan laws that prohibit USAID Career Center from sharing the personal information of youth who subscribed to career center services via the program’s online interface, USAID Career Center staff were delegated the task of sending the survey to youth using contact information contained in its database. 7 The survey was first sent to 5,000 youth on March 27, 2018, and follow-up reminder was emailed on April 7, 2018. On the same day USAID Career Center sent the reminder, emails invitations to participate in the survey were sent to an additional 5,000 youth in an effort to increase the response rate. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 5 Debriefing and Validation Workshop. The evaluation team facilitated two participatory debriefings to present and validate initial findings, discuss draft conclusions and preliminary recommendations and gather feedback. The first debriefing was delivered on April 10, 2018 to USAID and the USAID Career Center implementing partners. Additionally, the evaluation team held a participatory workshop on April 16, 2018 and included other key stakeholders in addition to USAID and implementing partners. DATA ANALYSIS The evaluation team’s findings from the desk study were summarized in short internal reports. Evaluation team members transcribed and cleaned KII and FGD notes on a rolling basis throughout fieldwork. Team members conducted periodic internal debriefs to identify trends and begin formulating preliminary findings. To reduce potential cognitive bias in the research and ensure the evaluation findings’ validity and reliability, the evaluation team coded trends and results to crosscheck and systematically triangulate the findings against multiple data sources, including interviews, FGDs, survey responses and USAID Career Center performance data. Additionally, through coding and cleaning the evaluation data, the evaluation team maintained respondent anonymity. The cleaned, coded and triangulated findings were recorded in a matrix organized by EQ which the team used to develop its preliminary conclusions and recommendations. The findings matrix is in annex 2. The team used feedback from stakeholder debriefings and reviews to refine and further validate the preliminary findings, conclusions and recommendations before presenting its final report. LIMITATIONS Selection bias: Selection bias is an inherent risk when implementers facilitate contact with project beneficiaries. To mitigate this risk in the constitution of youth focus groups, the team randomly selected participants from beneficiary lists the USAID Career Center provided; only those selected were to be contacted by career center staff and invited to attend FGDs. Career center personnel tasked with mobilizing selected participants, indicated that most of the actual FGD participants did not come from the lists the evaluation team selected; they reported they were unable to reach participants on the lists because of changes in contact details and that some selected participants were unable to attend because of exams and/or internships. Upon request, the evaluation team expanded the sample, but this step did not resolve the problem for the same reasons as described above. As a result, many FGD participants were youth the career center managers selected because they were available and easily mobilized. The evaluation team is unable to know the extent to which these participants represented youth with stronger connections to the center than randomly selected ones. Triangulating the findings with data collected via the online survey partially mitigated this limitation. Effect of evaluation timing on centers’ level of activity: The evaluation’s period corresponded with a university exam period, spring break for universities and vocational institutions, and to the time when some OFPPT trainees were engaged in internships outside the training institutions.8 Although the career centers were open and visited by youth, there were not many workshops or other activities occurring there. The team was able to observe only one activity.9 Collecting data on beneficiary perceptions of past activities partially mitigated limitations on the evaluation team to directly observe center activities. Difficulty getting meetings with the private sector: The evaluation team met with only eight private sector partners, a small number relative to the important role of private sector companies in the USAID Career Center’s design and implementation. Several enterprises the team contacted were unavailable during the evaluation period. This limitation is highlighted in the evaluation report. Findings based on 8 Evaluation fieldwork was carried out in the two weeks prior to spring break. 9 In Tangiers, there were a number of workshops advertised on the VCC that were cancelled because no youth signed up to participate. The evaluation team in Marrakech attended part of one EFE training. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 6 private sector KIIs are triangulated with other sources of data, including data the USAID Career Center provided. Online survey response rate: Although the youth online survey was sent to a large sample, the response rate was low. Only 218 youth participated in the online survey and just 135 (62 percent) completed the entire survey. Among the respondents, 149 were university students (91 were fully completed, 67 percent); and 69 were filled out by vocational institution trainees (44 were fully completed, 33 percent). While this sample is small, survey results were triangulated with other data sources to validate the findings. Due to the low rate of return on the private sector survey (only two responses), the evaluation team does not refer to this data in its analysis. FINDINGS EVALUATION QUESTION 1: WORKFORCE READINESS In response to EQ 1, the evaluation sought to determine to what extent the USAID Career Center has supported participating youth in acquiring the work readiness and soft skills the private sector requires. The evaluation team assessed the progress and performance the USAID Career Center made toward establishing career centers, facilitating work readiness courses, and engaging the private sector; the career center participants’ and private sector actors’ perceptions on the acquired work readiness and soft skills; and the enabling and disabling factors contributing to and/or limiting workforce-ready students. ESTABLISHING CAREER CENTERS, FACILITATING WORK READINESS COURSES AND ENGAGING THE PRIVATE SECTOR USAID Career Center Progress and Performance Detailed analysis of project performance to date meeting its indicator targets is in annex 6. Performance data is presented by project year which corresponds to March through April. Data presented on year three does not include results in January-March 2018. Establishing Career Centers. By December 31, 2017, USAID Career Center had successfully established five of six planned pilot career centers as well as the VCC (http://vcc.careeercenter.ma/vcc/). One university administrator remarked that the program established the center in his university in “record time.” It inaugurated the sixth and final planned center on February 14, 2018. Facilitating Work Readiness Courses. Based on performance reports, at the midpoint of implementation, the USAID Career Center had reached or surpassed its targets indicating the Career Centers’ effectiveness as measured by the number of youth receiving services, percentage of youth who return to use two or more services, and percentage of youth satisfied with the quality and relevance of training and services. To what extent are USAID Career Center participants –men and women between the ages of 15 and 29 – acquiring the work readiness and soft skills required by the private sector? What are the enabling and disabling factors contributing to workforce-ready students? Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 7 By the end of December 2017, the USAID Career Center had developed and tested workforce readiness modules, known as Najahi – Prêt pour l’Emploi, and engaged with private sector representatives in the delivery of career center activities. In year two, 6,044 individuals completed the USAID Career Center’s work readiness training programs, exceeding the annual target of 4,500. In the first three quarters of year three, 11,215 individuals completed the training, suggesting that the Career Center is on track to meet the year three annual target of 13,000. Additionally, the USAID Career Center largely has surpassed its targets for the number of workforce development programs using labor-market information in year two and with one quarter remaining in year three, has achieved 70 percent of this year’s annual targets. Engaging the private sector. The USAID Career Center has met or surpassed the performance targets aligned to its objectives for engaging and collaborating with private organizations (includes CSOs, private sector companies, and private intermediary organizations, such as employers’ associations) during the period this evaluation covers. In year two and the first three quarters of year three, the USAID Career Center surpassed its targets for the number of private organizations partnering with universities and vocational training centers. At the conclusion of the first three quarters of Year 3, the USAID Career Center partnered with 89 organizations more than double the number planned for the entire year. While achieving less than a 50 percent rate of return (partners who engage in more than one collaboration) in year two, the USAID Career Center currently is in line to meet its annual target in year three. ACQUIRING WORK READINESS AND SOFT SKILLS Youth Career Center Participants Perceptions Levels of Satisfaction of the Physical Career Centers. Most participants in youth beneficiary FGDs expressed a high degree of satisfaction with USAID Career Center services. Youth beneficiary online survey respondents corroborate the overall positive feedback from FGD and project reports. 10 Based on the online survey, 73 percent of respondents (68 percent of university students and 82 percent of vocational trainees), strongly agreed or agreed with the following statement: "The Career Center offers useful tools and content.” (n=135 respondents) Youth representing both universities and vocational institutions appreciated the centers’ counselors’ individualized approaches, opportunities to connect with the private sector through guest speakers, job fairs and challenge events, the look and feel of the centers’ space, and (although not an across the board finding), the quality of the EFE trainers.11 A large majority of the participants in the Tangiers career center university FGDs (the USAID Career Center’s longest operating career center) noted that they would like a greater variety of topics addressed in center workshops and activities. Levels of Satisfaction with the Virtual Career Center. Youth feedback on the VCC was overall positive with some caveats. Of the online survey respondents, 72 percent (69 percent of university students and 77 percent of vocational trainees), strongly agreed or agreed with the following statement: “The Online Career Center is easy to use and responsive to youth needs.” (n=135 respondents) Youth FGD 10 Project indicator eight reports an 80 percent satisfaction rate with career center training and services in year three to date, based on post training satisfaction surveys. 11 Some youth participating in FGD in Tangiers felt some of the EFE trainers were not effective. “The enterprise visits were super helpful, much better than what I expected.” –OFPPT Youth, Tangiers Of the 53,218 youth who received services from the USAID Career Center during the first three quarters of year three (March through December 2017), 30 percent returned for a second service. - USAID Career Center, performance data Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 8 participants’ feedback was more nuanced. Some echoed the survey results, highlighting the site’s usefulness. Youth as well as center managers and institutional partners expressed satisfaction with the VCC as a strategy to overcome distance from the center. Data on the number of “hits” on the site show that the orientation test (11,000+ hits), model cv (10,000+ hits) and industry profiles (3000 to 4000 hits each) to be among the most popular areas of the site. Two career center directors highlighted that many registered users are “passive” ones, mainly using the site to enroll in an activity and retrieve their participation certificate. In Marrakech and Casablanca, several youth, especially those representing beneficiaries of OFPPT and ISTAHT, declared "I have no idea about the VCC." A number of youth FGD participants likewise highlighted their difficulty using the online services; one said, “I tried one day to explore the site, but I felt lost.” Several youth and other KIs, including GOM counterparts and center directors, thought an Arabic version of the site would increase the accessibility and utilization of the online services. One center director highlighted that most of the youth VCC users in his/her center first participated in services provided at the physical center and then started using online services. Another director indicated that in his/her opinion youth motivation for using websites and online learning was low because youth are mainly using social media platforms. Outcomes Attributed to Participation in Career Center Services. Youth who participated in FGDs and the online survey highlighted positive outcomes as a result of their participation in career center services including improved self-confidence, a more realistic understanding of the job market, and awareness of the importance of soft skills. One youth noted that the “My Professional Development Plan” module helped her/him to find common ground between her/his passions and studies. Another affirmed that s/he had learned the steps to follow to find a job, how to behave with professionals, and how to have a successful job interview. According to USAID Career Center performance reporting, more than 90 percent of youth respondents, as measured by the post-career service activity assessment report, demonstrated improvement in skills after completing career center activities. Online survey respondents also indicated that the center had enabled them to acquire new skills, 58 percent of surveyed youth, including 55 percent of university students and 64 percent of vocational trainees, affirmed the following statement “I have acquired new skills through the Career Center such as controlling my emotions, self-esteem, communicating clearly and thinking critically.” (n=135 respondents). Differences between Male and Female Beneficiaries’ Experience According to USAID Career Center data, male and female students/trainees participate in career center services in roughly equal numbers. During FGDs with youth, the evaluation team did not notice any significant differences in feedback from male and female participants. Data from the online survey, however, shows interesting differences between female and male perspectives on career center services: 85 percent female vs. 74 percent male respondents strongly agreed or agreed that the career center space was welcoming for youth. (n=73 male and 62 female respondents) Likewise, 76 percent female versus 63 percent male respondents strongly agreed or agreed that they had learned about the jobs and careers that fit their personality and skills. (n=73 male and 62 female respondents) Female participants’ perceptions on the centers’ effectiveness linking them with the private sector appear more positive than males: 52 percent of female versus 36 percent of male respondents affirmed that the career center helped them to network with potential employers. (n=73 male and 62 female respondents) “I liked the personality test – that it gives you what you can do as a job. Also, in the towns where there is not a career center the VCC is very useful" – University Youth, Casablanca “I have gained self-confidence and learned how to face the job market.” – University Youth, Casablanca Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 9 Differences in Private Sector Engagement between University Students and Vocational Institution Trainees Based on youth FGD and survey responses, youth from vocational training centers indicate a greater sense that their participation in career center activities is enabling them to connect with the private sector. For example, 59 percent and 66 percent of vocational institution online survey respondents affirmed the statements “The Career Center helped me to get a job,” and “The Career Center helped me to access an internship,” as compared to 26 percent and 40 percent of university respondents, respectively. (n=91 university and 44 vocational training respondents) Additionally, 65 percent of vocational education respondents versus. 40 percent of university respondents state that the Career Center helped them to network with potential employers (n=114 respondents). One OFPPT trainee in Tangiers highlighted his/her positive experience participating in a job fair: “During the AMITH job fair, we benefited from contacts that allowed me and my fellow trainees to find internships. Other trainees were able to apply for jobs.” In contrast, one university career center director indicated that s/he thought center capacity to cultivate private sector partnerships needed to be strengthened, “The weak link in the chain of career center services is private sector engagement: we are not able to help with internships and job placement. It is a structural problem – the business developer is not close enough to the center to be effective. We think the business developer should be part of the team.” USAID Career Center program managers note that based on its scope of work, the USAID Career Center is not a job and internship placement program but rather it enhances youth employability, including filling the gap in soft skills training and providing youth with real-world exposure to the private sector through a wide range of programming, including guest speaker workshops, company visits, and job fairs. Private Sector Perceptions Levels of Satisfaction of the Career Center Activity. USAID Career Center reports that a large percentage of employers are satisfied when they employ youth from the center services. In year two12, the project reported 100 percent satisfaction based on feedback from 11 companies recruiting 93 young people through the career centers. In the first three quarters of year three, 92 percent of companies surveyed were satisfied with the career centers’ services. Based on data the USAID Career Center provided, 45 organizations have collaborated with career centers for recruitment activities. According to the USAID Career Center management, the career centers piloted Najahi trainings with three companies: Xceed (call center), SEBN (automotive manufacturing) and Domino’s Pizza (restaurant). USAID Career Center staff held meetings with company representatives immediately following the pilot to gather their verbal feedback. According to the USAID Career Center program managers involved, feedback from the three was highly positive. Xceed and Domino’s Pizza, as indicated by the USAID Career Center, found that the trainings responded well to the need for soft skills in the service industry, and SEBN found that the communication skills modules in the “My Career Pathway” course were beneficial for their job candidates. Through the online survey, the evaluation team aimed to independently corroborate the private sector partners’ satisfaction but was unable to do so because of to low response rates. Despite the aforementioned positive feedback, the USAID Career Center received, the evaluation’s findings were inconclusive as to the extent to which private sector partners felt that career center participants were acquiring the work readiness and soft skills they required. Based on five KIIs with USAID Career Center private sector partners engaged in the center’s recruitment activities, most informants did not feel sufficiently informed to comment on the soft skills of recruits because they were not the recruits’ supervisors. Two informants also indicated that they did not find any significant difference in their soft skills as compared to other recruits who had not received center services. These informants, however, 12 April 2016through March 2017 Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 10 did indicate high levels of satisfaction with the support they had received from the USAID Career Center for their recruitment drives. ENABLING FACTORS CONTRIBUTING TO WORKFORCE-READY STUDENTS Quality of Career Center Counselors Many informants across multiple stakeholder categories (activity managers, institutional partners, and youth) underlined that the quality of career center personnel is the most important factor in determining the caliber of center services. When asked what the characteristics of an effective counselor were, youth FGD participants identified good communication, ability to listen, desire to serve youth, tolerance, and proficiency using participative training methods. Several youth, representing both university and vocational training institutions, also thought youthfulness was important. One university student said, “Counselors have to master the content they share and have a good knowledge of the private sector.” The majority of youth FGD participants representing both university and vocational training institution career centers, felt positive towards the career center counselors. One university student indicated that the career counselors were the “face” of the career centers. A vocational trainee attributed his/her positive behavior change to coaching, “With the help of our trainers, my friends and I have changed our attitudes and are more serious about our studies.” In contrast to broadly positive feedback, some university students from Tangiers and Marrakech felt their counselors could be better qualified and/or more dynamic in terms of organizing interesting activities and communicating with center users. For example, youth cited events that were cancelled at the last minute, without notice. (FGDs in Tangiers and Marrakech) Career Center Staff Training and Capacity Building Most career center staff expressed satisfaction with training and exchange opportunities offered by the program. Staff who participated in United States study tours highlighted many positive learning outcomes such as being able to see what a mature center looks like. Additionally, collaboration with other career centers in the United States and Morocco have provided staff with the opportunity to learn best practices for overcoming challenges (for example, how to attract students to center services). In looking ahead, center staff highlighted the need for additional training in areas including finance and administrative management, training program design, coaching and counseling techniques, and certification. Center staff appreciated the training and capacity building opportunities to experiment with different approaches, types of activities and partners; but some respondents noted that central office and regional coordinators had been more restrictive on the types of allowed activities in recent months. Several staffers felt that the USAID Career Center regional coordinators’ oversight roles needed to be redefined now that the centers were operational and center directors were reporting more directly to their institutional hierarchy. One director, who had been without a regional coordinator for a period of time indicated it was useful to learn to manage his/her center without a regional coordinator in preparation for the end of the USAID Career Center Activity. DISABLING FACTORS LIMITING THE DEVELOPMENT OF WORKFORCE READY STUDENTS Mobilizing Youth Even though the career centers are meeting their targets for numbers of participants, as discussed above, most career center managers still reported challenges mobilizing youth. Similarly, the majority of youth FGD participants felt that the centers largely were unknown and unused by many of their fellow students and trainees. During the evaluation, most centers were empty, which as center personnel explained, can largely be attributed to the timing of the evaluation as it took place during an exam period, just before spring break. Center personnel also highlighted that they were organizing some events outside of the center during the evaluation period; center directors indicated that they increasingly do so to reach youth Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 11 who would not otherwise be able to benefit from the centers’ services due to their distance from the physical centers. Some universities have large numbers of students on multiple campuses across several cities. According to youth and center personnel FGD participants representing universities and vocational training institutions, the main obstacles to mobilizing greater numbers of center users are: • poor communication by institutions on center services, • trainees’ and students’ lack of time to engage in center activities, • lack of awareness of the importance of work readiness and soft skills training, • physical distance from the center, and • some youth’s reticence to try something new. To improve communication and mobilize more youth, several youth FGD participants from both the universities and training institutions suggested that the centers should improve their social media strategy. Currently, all six career centers have Facebook pages that are updated regularly with information about center activities and links with the VCC content. One vocational training career center manager countered this finding, indicating that attracting youth was less of a problem in recent months; rather, the challenge now was how to organize to satisfy demand. Activity managers explained that they initially had wanted to be prudent in their communication efforts during the center start-up and testing phase so as not to communicate broadly on offerings not yet fully tested. Participants, in a FGD with a CSO serving persons with disabilities, noted satisfaction with project efforts to provide services to their beneficiary population, including by addressing accessibility considerations in some of its services, but they also highlighted that barriers remain. Counselor to Student/Trainee Ratio Feedback from university students and vocational trainees during youth FGDs show that individualized and small group center services are an essential part of the existing centers success. Youth expressed strong appreciation for the centers’ personalized services and many expressed the importance of increasing the number of career counselors to ensure this type of service. Informants across different stakeholder categories (including USAID Career Center program managers, GOM counterparts, and host institutions), cited the size of the student body and number of trainees as a significant challenge to future career center effectiveness in the post-pilot phase. Several KIIs in Casablanca cited the ratio of career counselors to students at the University of Pennsylvania (60:25,000 students) compared to University Hassan II (3: 120,000 students) to illustrate the problem. The rapid growth of public universities’ student bodies is an issue affecting all university services. According to GOM reports, between 2012 and 2016, the number of university students grew from 400,000 to 800,000.13 One central GOM vocational training counterpart likewise indicated that the number of centers relative to potential demand was low; s/he put her/his hopes in the development of the VCC to overcome the challenge, “We have 120,000 trainees in Casablanca￾Mohammedia. Fortunately, there is VCC to meet the demand.” As demand continues to grow for career center services, following the initial start-up period, the challenge of low counselor to student/trainee ratio and the growing student population, could limit center effectiveness after the USAID Career Center ends. 13 Morocco Ministry of National Education, Vocational Training, Higher Education and Scientific Research Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 12 Geographic Dispersion KIIs highlighted the geographic dispersion of campuses as problematic to the “center” model, as the campuses (universities and vocational training institutions) meant to be served by the centers are spread across distant neighborhoods and even other cities. Career center directors and counselors highlighted that the location of centers on one campus tended to favor students from that campus or proximate campuses and hinder others from using the center facilities. Although they cited some examples of student/trainees from distance campuses visiting the physical centers, they indicated that many students simply could not afford the cost of transportation to the center. In addition, two career centers are located on engineering school campuses; center personnel reported that many “open access” degree students, even though studying close by, were reluctant to enter an “elite” campus. KII among career center staff, university and training center administration, and institutional counterparts highlighted various strategies being employed to overcome the challenge including: • Conducting short outreach activities on other campuses (Ministry of Tourism, OFPPT and Universities); • Organizing Najahi training workshops on other campuses; • Leveraging youth ambassadors for promotional activities; • Coordinating with trainers/faculty to facilitate organized visits to the career center; • Offering online services via the VCC;14 • Organizing career center summer camps when youth are freer; • Planning to establish career center sub-offices on other campuses; • Planning to open new centers. 14 USAID Career Center reports that 18 percent of VCC users come from regions other than where the centers are based. Feedback on Youth Ambassador and Summer Camp programs Among the strategies to extend career center services to a greater number of youth are the peer-to￾peer Youth Ambassador and the Summer Camp programs. KII and FGD participants from university centers in particular highlighted the important outreach role youth ambassadors play. For example, in Casablanca, the university vice-president (VP) shared plans to open career center sub-offices to be staffed by one administrator and youth ambassadors. Feedback from youth ambassadors on their role was generally positive but some felt that they needed more formal recognition from their institutions to be effective. A few ambassadors reported initiatives being “blocked” by his/her institution. This sentiment was especially true of vocational training center ambassadors. Summer camp participants indicated the approach facilitated access to career center services by some youth who would not be able to participate during the regular school year. Some participants remarked, in its first edition, demand for workshops outstripped capacity. Program managers acknowledged this issue; they indicated they had plans to address the capacity issue this summer One regional manager, however, doubted the availability of funding for special summer activities post￾USAID funding. “Distance is a big problem. When you want to attend a meeting, you have to spend the whole day.” – University Youth, Casablanca Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 13 Delivery of Soft Skills Trainings and Workshops Najahi Training. Based on the USAID Career Center data, more than 13,000 youth have participated in Najahi training activities as of March 31, 2018. Based on the evaluation team’s benchmarking, the program’s Najahi soft skills training package aligns with what experts in the field generally consider essential for effective soft skills acquisition.15 Based on feedback from USAID Career Center program managers and EFE trainers, however, career centers are not delivering all of the Najahi modules to youth. Several core modules covering the foundations of soft skill development only are offered as supplemental modules and currently are not included in the three to five-day training delivered by EFE. According to the USAID Career Center performance data through March 2018, around 70 percent of the total 13,263 Najahi participants attended workshops focused on job search skills, “My Employment Kit” while, in contrast, only 234 youth have received training based on the supplemental modules. In discussions with 14 EFE trainers in Casablanca and Marrakech, there was overall consensus that modules relating to personal development should have been introduced at the very beginning of the training, to make the rest of it more relevant and effective. EFE trainers also commented on the quality of the French translation of Najahi training materials, indicating that revisions are required for them to be more readily usable by trainers. Soft Skills Workshops. In addition, to Najahi, training, youth career center beneficiaries participated in a variety of other soft skills workshops in the centers or during outreach activities. Based on data the USAID Career Center provided, more than 12,417 youth took part in these workshops. Fifty percent (6,178) of the total 12,417 participating youth attended workshops focused on job search and orientation topics including preparing a job interview, improving your CV, discovering the VCC, finding a career path, discovering the world of work. The remaining 50 percent (6,239 youth) attended workshops covering broader personal development themes addressing topics including time management, team work, public speaking and communication and self-confidence, as well as languages, computers skills, debates, etc. The centers also have offered one-on-one sessions to more than 500 youth, 1,300 youth have taken part in enterprise visits and guest speakers workshops, and 1,600 youth have participated in job fairs. Timely Orientation Services Key stakeholders across all categories were almost unanimous that youth needed to benefit from orientation services at an earlier stage than provided in the career centers. Participants in youth FGD representing university and vocational training institutions expressed keen interest in the orientation modules16 but some of them also expressed frustration that they were trapped by choices already made. Some youth indicated that center workshops and trainings on career choices were not too late to be helpful when choosing among potential job options within their field of study. Some career centers have sought to reach youth earlier through secondary school student outreach activities, although USAID Career Center personnel indicated that they are limited in their capacity to support this strategy as it is beyond the program’s current scope of work. The evaluation team notes that the Moroccan education system is considerably less able to accommodate career path redirection than its North American counterparts17; various 15 The evaluation team used the Global Partnership for Youth Employment document “Strengthening Life Skills for Youth” as its primary benchmark. 16 “Knowing myself” and “Preparing for my career” 17 The Economist, Special Report: Choice of sectors by Abashi Shamamba, Edition N° 5057 (04 July 2017) http://www.leconomiste.com/article/1014491-choix-de-filieres-rebondir-apres-une-erreur-d-aiguillage “I think that training should be given to high school students like my younger sister. They need help with their orientation.” – University Summer Camp Participant, Marrakech Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 14 participants in KIIs affirmed that the orientation’s timing is an element of the current model that requires adaptation to the Moroccan context. Retention USAID Career Center program managers highlighted high turn-over among center staff as a challenge to maintaining quality personnel. Program managers said that they had found some candidates for center director, which the career center hosting institutions proposed, to be lacking some qualities required of a good director. This contributed to some of the career center director turnover. Program managers and center host institution administrators expressed overall satisfaction with the qualifications and performance of current career center personnel at the time of the evaluation. EVALUATION QUESTION 2: VALIDATION OF THE CAREER CENTER MODEL Evaluation question two measures the extent to which the career center model has been tested and validated to ensure relevance to Moroccan stakeholders and career center beneficiaries to improve performance. The findings presented highlight the effectiveness of the USAID Career Center’s engagement with its main stakeholders during the design, implementation and adaptation of the career center model as well as key stakeholder feedback on the career center model’s relevance to their needs. ENGAGEMENT WITH KEY STAKEHOLDERS AND BENEFICIARIES Engagement: Design Phase Various GOM and other national level informants affirmed that USAID and USAID Career Center managers used participatory methods to engage and collaborate with most key stakeholders in the development of the Career Center model. USAID Career Center personnel indicated that the engagement with the university host institutions would have been stronger initially if consultations with their administrations had occurred prior to the selection of their institution to host a career center. For vocational training institutions, the program selected career center host institutions through an open bid; according to their reports, this process, strengthened engagement and ownership at the start among these institutions. USAID Career Center personnel also involved other stakeholders including youth in career center design, according to KIIs. Relevant assessment reports indicate private sector representatives were involved in the development of soft skills training programs through structured assessments and enterprise-based tests. According to USAID Career Center reports, Najahi program designers also reviewed existing training materials used in university and vocational training centers to assess how work readiness and soft skills have been integrated into the program counterpart’s existing study and training programs. Based on informant feedback and project reports, it does not appear that CSOs, including those with specific missions related to gender and persons with disabilities, were consulted during the design phase of the USAID Career Center model. Rather they were later integrated as activity partners. USAID notes that inclusion of persons with disabilities was not part of the USAID Career Center’s original scope; the informant also noted that once USAID had sensitized USAID Career Center managers on the issue during the implementation phase, they quickly integrated activities to address it. One representative of a CSO working with persons with disabilities believes that the needs of persons with disabilities should have been integrated in the program in the design phase. EQ2: How effectively has USAID career center tested and validated the career center model to ensure relevance to Moroccan stakeholders (students and their immediate environment, educators, government counterparts, private sector, and civil society organizations [CSOs])? Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 15 Engagement: Implementation and Oversight Overall, feedback from most of the USAID Career Center’s institutional partners showed a high degree of satisfaction with USAID Career Center management efforts to keep them informed. Program reports cite many good communication practices such as forming a National Steering Committee, outreach activities and a newsletter. The evaluation team found program reporting to be detailed and easy-to￾understand. Stakeholder perceptions on how they had been engaged by the program during implementation and oversight, however, were somewhat mixed: • University administrators indicated satisfaction with the regular contact and ongoing collaboration they had with USAID Career Center program managers to discuss and find solutions to the challenges the career centers face. They also indicated that their direct oversight of the career centers was minimal at first but that this relationship was changing as the program matures and center directors report to them more frequently. • One GOM counterpart expressed disappointment with the program’s efforts to keep national stakeholders involved in decisions about the future of the centers. • The majority of participating EFE trainers indicated that they had provided the program office feedback, via email to program managers, on necessary improvements to be made on the Najahi training modules, including the need for additional foundational content and problems in the translation of workshop presentations. However, this feedback had not been taken into consideration by the program. • Some career center directors indicated they and their institutions’ decision makers needed more information about the financial requirements of running a center. They also feared that institutional leadership was not sufficiently sensitized about the importance of allocating an operational budget for activities (such as enterprise visits, job fairs, workshops). One center director was adamant, “The center will die later on if there is not enough financial support. OFPPT wants to open 300 centers, but they don’t know how much it is going to cost them and if they do, they will change their minds. Even I don’t know how much it costs.” One GOM stakeholder indicated that during a recent presentation, the cost of installing a center was shared, the ongoing operational and management costs to sustain the centers, however, was not. USAID Career Center program managers note that vacancies in the leadership of the Minister of National Education, Higher Education, Vocational Training and Scientific Research, OFPPT, and the State Secretary for Vocational Training affected the USAID Career Center’s ability to fully discuss and evaluate how the model would be adapted by the institution, including its associated costs, in the process of scaling the model. The kinds of collaboration I would like to have with a career center are around preparing youth for internships/apprenticeships." – Private Sector Enterprise, Tangiers Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 16 Engagement: Evaluation and Adaptation The USAID Career Center sends satisfaction surveys to gather feedback from youth and private sector partners after completing each activity. According to the USAID Career Center, while youth rates of return are relatively good (30 percent), there is insufficient feedback from employers in general and specifically related to the outcomes of sourcing activities. Currently, the USAID Career Center is using a proxy indicator which measures how many private sector partners collaborate with the centers two or more times to provide an indication of satisfaction. Based on USAID Career Center reporting for fiscal year three, 50 percent of private organizations partner with career centers two times or more. One challenge in following up with youth who receive services from the career centers is the accuracy of their contact information in the project’s database. Although the program maintains a youth beneficiary database as part of its management information system, during efforts to contact youth for this evaluation, program personnel reported challenges mobilizing them because of erroneous contact information, noting youth frequently change their telephone numbers and email addresses. According to USAID Career Center program managers, youth manage their profiles directly in the VCC and are able to update their contact information from the site. While career center staff also can update participants contact information through the Management Tool, youth ultimately are responsible for updating their own profiles and contact information. RELEVANCE OF CURRENT MODEL TO KEY STAKEHOLDER NEEDS EQ1 findings show youth feedback on the model’s relevance to their needs to be overall very positive. It also highlighted aspects of the model that need to be adapted to the Moroccan context such as the size of the student body and geographic distribution of the campuses. Other stakeholder feedback about the model’s relevance, described below, is likewise primarily positive. Career Center Host Institutions Overall, senior-level university stakeholders were positive about the project. One university vice president (VP) described the career center as providing “undeniable value-addition” because it was enabling the university to fill gaps in students’ soft skills. Another VP agreed but added that the relevance of current services needed to be assessed. Engaging ANAPEC: Gaps in Consultation and Communication ANAPEC, the national institution charged with the economic insertion of vocational training and university graduates (among its other mandates), indicated that while it was informed of planned USAID Career Center activities in an initial meeting and later through participation on the steering committee, it was disappointed in the quality of consultations during the development of the model. According to one official, the Agency had expressed interest in exploring synergies between its planned collaboration with Higher Education to establish university agencies and the career centers but had not been given an appropriate opportunity despite its formal requests. One Higher Education official indicated that there had been discussions to consider whether or not the development of USAID Career Centers and ANAPEC agencies should be coordinated. In the end, the Ministry decided that this debate needed to be postponed until after the initial piloting of USAID Career Centers. According to a KII with ANAPEC, this decision had never been clearly communicated to the Agency. The latter noted that in Tangiers and Casablanca, there will be both a USAID Career Center and an ANAPEC university agency; and that ways to coordinate and capitalize on the other’s services had not been discussed yet. (According to USAID Career Center managers, the Agency in a university in Casablanca opened and closed before the first career center was open in the city. In Tangiers, the ANAPEC agency is scheduled to open in 2019.) Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 17 Faculty involved in USAID Career Center Najahi Training of Trainer programs. Participants in university TOT18 indicated that the training on Najahi training modules the USAID Career Center provided was useful, although some of them said the content was similar to their existing courses. The faculty members strongly agreed with the relevance of integrating soft skills into the university curriculum but highlighted many obstacles to do so effectively, most significantly class sizes and challenges evaluating the acquisition of soft skills. Faculty FGD participants highlighted that professional development opportunities tended to be few and far between and that they were very grateful to update their instruction techniques. They were not entirely satisfied, however, with the level of support that the USAID Career Center provided following the TOT, highlighting that they had received no follow-up training or coaching and that copies of Najahi modules arrived after they had already begun duplicating the training. According to a USAID Career Center program manager, all three universities had been offered three months of follow￾up support. To date, the university in Tangiers, where the majority of the evaluation’s TOT KIIs were conducted, is the only university that had not yet accepted the USAID Career Center’s follow-up support. GOM Stakeholders and Counterparts According to one national counterpart KII, the Ministry of Higher Education had been considering following the American/Anglo-Saxon career center model for some time and was pleased that USAID had provided needed technical and financial support. Private Sector Partners KIIs with private sector partners highlighted positive interactions with the centers. Private sector representatives appreciated the flexibility of the centers to respond to specific needs such as helping with recruitment drives and providing professional development opportunities for human resource managers. Several indicated that their interaction had been limited and they would have liked to be more-involved and better-informed on career center activities. Private sector KIs agreed on the importance of the USAID Career Center’s mission to develop work readiness and soft skills in youth. Some private sector informants wanted center services to be even more tailored to their specific needs such as providing pre￾selection services for intern and employee recruitment or, while beyond the scope of the USAID Career Center, offering soft skills training to their employees. A representative of an employers’ association underlined the importance of evaluating the impact of the model on beneficiaries’ soft skills. CSO Partners CSO partners found a win/win arrangement in their collaboration with the USAID Career Center. CSO KII participants reported benefits such as increased visibility among youth and a space for their activities while career center beneficiaries appreciated for CSO programs. Campus-based student clubs linked with national and international CSOs in particular seemed engaged in collaboration with the university career centers. Based on a variety of KIs, through collaboration with persons with disabilities and women’s rights CSOs, career centers integrated activities addressing some specific needs of women and persons with disabilities. RELEVANCE OF THE USAID CAREER CENTER MODEL TO THE LOCAL MOROCCAN CONTEXT USAID Career Center partners face challenges to adapt the North American center model to fit the Moroccan context. EQI findings highlighted obstacles to operate the model on a scale to meet the needs 18 Unfortunately, no participants in OFPPT TOT training were available to speak to the evaluation team. “The USAID Career Center has become even more than what we expected. It has become an interface between the training centers and employers.” - GOM Official Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 18 of the large numbers of students and trainees served by an institution as well as the need for earlier orientation services given the relative rigidity of the education system. Model relevance to the Moroccan context is also affected by characteristics of the pilot program’s implementation environment. For example, the USAID Career Center model has been tested in locations with relatively vibrant economies and with a limited set of private sector partner profiles (large structured companies and multinationals). According to USAID Career Center program managers, the program intended to adapt its services to the needs of sectors with high growth potential in cities with large numbers of youth. The selection of these cities and choice of private sector partners is well-founded in GOM strategies for economic growth and development.19 As the model is expanded to additional locations, including cities with few or no emerging industries, however, the model will require adjustments. Very small, small, and medium-sized enterprises comprise 80 percent of Morocco’s private sector, making them the country’s the largest employer. 20 One university VP in Marrakech, where the economy is dominated by tourism, remarked, “The career center should propose services more adapted to SMEs because the big enterprises are looking for specific kinds of profiles rather than the generalists who have the greatest difficulty finding employment. The career center should tailor its services according to the enterprise. The big enterprises are not recruiting a lot.” One USAID Career Center manager recognized that in a future expansion of the model, existing career center services would need to be adapted to the needs of smaller cities with less vibrant economies. EVALUATION QUESTION 3: SUSTAINABILITY AND SCALABILITY Findings under EQ3 focus on the factors that currently contribute to or limit USAID Career Center sustainability based on the feedback of a variety of key stakeholders. SECURING COMMITMENTS TO SUSTAIN AND SCALE Physical Career Centers In year three, the program reported that the center hosting institutions contributed 63 percent of the centers’ operational costs, more than double the target of 30 percent for the period. All host-institutions contribute to operational costs of the career centers and fund the salaries of center directors (with the addition of a deputy director position in Marrakech University). OFPPT and the Ministry of Tourism also fund the salaries for center career counselors. According to multiple KIs, center host institutions have the mechanisms to generate revenue to cover center operational costs through training programs for external clients. This model is the main one university administrators propose to cover centers’ operational costs. According to USAID Career Center program managers, several strategies are also on the table for assuring the continuity of programs that currently depend on donor funding and expertise. These plans include identifying an existing or creating a new organization to manage some services through public-private partnerships with partner institutions, moving towards an income generation model and/or full integration into host-institutions’ management structures and budgets existing institutional hierarchy oversee. According to several KIIs, the income generation model has not been tested yet because USAID regulations restrict the practice. 19 Information about the GOM strategy to develop economic growth sectors can be found here: http://www.mcinet.gov.ma/fr/content/plan-d%E2%80%99acceleration-industrielle 20 http://www.cfcim.org/wp-content/uploads/2016/10/conjoncture-985-octobre-novembre-2016.pdf, p.3 EQ3: How likely are the career centers to be sustainable? Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 19 Feedback from a small sample of private sector partners indicates limited interest in paying the career centers for soft skills training for existing employees. Two companies indicated that they have their own soft skills training programs, while another one said that unless the services were free, they would not be interested. The same informants did indicate interest in collaboration with the career centers for sourcing activities and a willingness to pay. One regional activity staff person with a background in the private sector likewise believes there is a market for sourcing support but added “the centers were not yet very effective” in engaging with the private sector to explore sourcing support. Virtual Career Center According to USAID Career Center program staff, a variety of institutional models are under consideration for developing and managing the VCC. These ideas include giving over management to a central ministry partner and/or creating a new or delegating VCC management to an existing CSO or enterprise. One central institutional partner indicated that to take over the management of the VCC, it would need additional capacity building. Najahi Soft Skills Training EFE currently delivers Najahi training. According to USAID Career Center program managers, the main strategy for ensuring the continuity of the training is to train trainers within the center host institutions. As of the third quarter of year three, the USAID Career Center has not met its targets yet related to the integration of soft skills into training courses curriculum in year three; however, it anticipates progress in the final quarter of year three. 21 According to an OFPPT administrator, Najahi training modules are currently being adapted by the Office, which plans to mainstream 60 hours of soft skills training into all its vocational training programs starting in September 2018 (30 hours more than previously). Training of OFPPT master trainers already has taken place. Personnel in one OFPPT career center viewed Najahi workshops as an important center service and believed delinking it from the center would create a gap. According to some EFE trainers who provided the training of master trainers, some of the OFPPT trainers do not have the ideal profile to deliver the soft skills training. One university VP doubted the feasibility of integrating the training program into the curriculum of open access undergraduate degree programs because of the large number of students. The administrator added that decisions to integrate training program modules into the bachelor curriculum have to be done at the national level. According to USAID Career Center program managers, a temporary vacancy within the Ministry of Higher Education had slowed down soft skills integration, as university leadership wanted to wait for approval from the ministry on some aspects. Based on feedback from the university VP and participants in Najahi TOT, however, both the universities in Tangier and in Marrakech already have integrated soft skills into the master’s curriculum, and trainings are being delivered in Tangier as of February 2018. Scaling the Model OFPPT administrators indicate that it already has committed to open three new career centers in other regions/areas before the end of the program and it also has committed to disseminate the model in all the training centers across Morocco, as stated in the memorandum of understanding (MOU) signed between OFPPT and USAID. Similar commitments have not been made yet by any of the career centers host universities. In Casablanca, however, various informants reported that plans are underway to open career center sub offices to be staffed by one university employee with support from Youth Ambassadors. The university VP proposed to structure the Youth Ambassadors as a student association based on the 21 There were no targets for integrating soft skills training in years one and two. In the first three quarters of year three, the USAID Career Center had achieved six education institutions (six departments of the Abdelmalek Essaadi University in Tangiers) against its target of 15 for the full year. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 20 ENACTUS model, which has a strong system for renewing its leadership as students enter and then leave university. Based on feedback from USAID Career Center management, other GOM entities with vocational training departments (such as agriculture, handicrafts, and health) have a keen interest in replicating the USAID Career Center model. One participant in the meeting between the USAID Career Center and these GOM departments said that some participants found the project cost of establishing a center based on the USAID model to be excessively expensive. FACTORS CONTRIBUTING TO AND LIMITING SUSTAINABILITY AND SCALABILITY Relevance to Key Stakeholders Priorities and Needs As presented above, the evaluation findings and analysis indicate that the USAID Career Center has established a brand and set of services that broadly respond to national counterparts’ and partners institutions’ priorities and beneficiaries’ needs. Model relevance and buy-in from key stakeholders will be a key factor contributing to career center sustainability. One caveat potentially affecting university career centers is upcoming changes in their leadership. The mandates of the university presidents come to an end at the end of 2018. According to one VP, the changing of the president would not affect the continuity of the existing career center because the program is already well-anchored. However, according to another VP, the position of the new president will be decisive for the continuity and scaling of the program. Maintaining the Quality and Dynamic Nature of the Career Centers’ Human Resources One dimension of sustainability cited in various KIIs is existing career centers’ capacity to recruit current staff. One GOM official was adamant, “The question of sustainability really rides on the quality of our human resources.” A regional activity manager cited counselors turn-over as a major problem “A student creates a relationship of trust with the counselor and then the person leaves.” S/he was pessimistic that those not already recruited by the center would stay. Another center director likewise indicated that the process for hiring counselors in his/her university had been slow and was worried that existing staff might look elsewhere. Relevant institutional stakeholders report various degrees of progress addressing the matter. All OFPPT/Ministry of Tourism current personnel are already employees of their respective institutions. In universities, the host institution covers the director position, but not the career counselors. According to one university VP, these positions will be integrated within the university, but most likely at a lower salary level than they currently are paid. A Ministry of Tourism official expressed concern about the relatively low salaries of current personnel and possible challenges to keeping current staff and hiring in the future. Current center personnel also are worried that their positions have not been integrated formally into their institution’s human resource structure yet, making it difficult for them to be evaluated and receive promotions. The other challenge to maintaining qualified career center personnel is establishing a sustainable mechanism for career counselor professional development. KIIs with administrators of center host institutions indicated that current personnel will play an important role in training additional personnel who may be recruited during an anticipated expansion. In addition, according to USAID Career Center program managers, the program is developing a strategy to address this issue in collaboration with the University of Laval in Quebec, Canada. In Casablanca, the career center director reported that the university already had agreed on the implementation of a joint program with Laval and that a visit was programmed in the coming weeks to begin setting up the program. Several existing personnel of vocational training institutions indicated that they already felt able to manage their center independently with one career center staff member indicating that “we are well-equipped to manage our center ourselves.” In contrast, however, a Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 21 regional manager disagreed noting that “the centers are still babies” and “they are not ready to function without outside support.” Private Sector Partnership Building According to half of the participating career center directors, the career centers capacity to develop and maintain partnerships with the private sector is a weak point in the current model. From their perspective, the problem is that they lack capacity to mobilize the private sector within their teams and depend on USAID Career Center personnel. It is now the responsibility of the regional business developer. The career center directors indicated that they think it should be a core function of the center, but that currently they do not have the time or expertise to take on this function. One regional manager likewise affirmed, “There are some strong partnerships between the university and the private sector, but they need to be cultivated. University personnel don’t have time for this.” According to one private sector stakeholder, the issue of developing and maintaining private sector partnerships is particularly acute for the university centers; most vocational training centers have at least some experience in this area. USAID Career Center managers shared that they propose the business developer as a key staff person in their career center tool kit guidelines, which offers strategies for creating and managing of future centers. They also indicated that they are looking at options to recreate the regional business developer position in one of their GOM partner’s regional offices. CONCLUSIONS AND RECOMMENDATIONS The evaluation found that USAID Career Center has made substantial progress towards the achieving its primary objectives. EVALUATION QUESTION 1: WORKFORCE READINESS The Career Centers have made progress to help youth acquire the work readiness and soft skills required by the private sector. The pilot career centers have succeeded to create a youth￾friendly space and set of services that correspond, to its young users’ expectations and needs. The majority of youth beneficiaries who participated in the evaluation, expressed high levels of satisfaction with the USAID Career Center and highlighted positive outcomes as a result of their participation including improved self-confidence, a more realistic understanding of the job market, and awareness of the importance of soft skills. The current career center model demonstrates strengths in the provision of small group and individualized services and connecting students/trainees with the private sector. The program has produced motivated and qualified personnel to manage the centers. The USAID Career Center helps participants across the six career centers to acquire the work readiness and soft skills required by the private sector; yet some gaps remain. As a pilot, USAID Career Center is limited in its capacity to provide services to the large number of students and trainees spanning across the multiple campuses that the centers are mandated to serve. The USAID Career Center is currently testing strategies (such as online services, outreach events, creation of sub-offices, peer-to-peer counseling, summer camps, and adapting services to persons with disabilities) to extend the reach of existing centers and ensure their continued relevance after the USAID Career Center ends. The USAID Career Center and its institutional counterparts need to pursue these strategies in the time that remains for project implementation, and the institutions that will manage the centers after USAID support need to refine the strategies to ensure sustainability. Narrow selection and delivery of Najahi training modules may limit the training’s impact on participants’ soft skills development. Career centers and partner institutions’ tendency to focus narrowly on modules Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 22 promoting technical job search and job preparedness skills may not be fostering the development of some key foundational soft skills, limiting the potential impact of the training if not reconsidered. Workshops offered by the career centers address a fuller range of soft skills and partially address the limitations of Najahi core training as it is currently delivered. Differences in levels of satisfaction between vocational training and university beneficiaries highlight the importance of the center providing adequate links to private sector employers.22 Additionally, university youth from centers operating for more than a year, indicated the need for innovation in services. Their comments suggest that continued youth satisfaction is contingent on renewing current approaches regularly. The timing of career center orientation services is problematic in Morocco where the educational system limits youth’s options to rethink and change educational paths. While orientation is relevant at every educational stage, the current career center model is inheriting a problem (weak orientation services in secondary education) that is outside of its scope to address effectively. Recommendations for consideration to strengthen the effectiveness of career center services to help youth acquire work readiness and soft skills required by the private sector include the following: 1. Expand the foundational Najahi training core curriculum. USAID Career Center should integrate existing foundational soft skills modules such as “Managing my Emotions and my Stress,” “Working as a Team,” “Me and Problem Solving” and “Negotiate with Gusto” into the core (versus supplemental) curriculum of Najahi training and advocate for expanding planned institutional offerings to likewise include these foundational modules. (High priority, to be addressed in the short term) 2. Elevate business developers as key personnel alongside career counselors. USAID Career Center should integrate the post of business developer as key center personnel alongside the career counselors in order to ensure strong linkages with the private sector and CSOs. (High priority, to be addressed in the short term) 3. Assess private sector demand for career center services among SMEs. USAID Career Center has tested engagement with structured, large and multinational companies in its pilot activities. Additional engagement with SMEs would be useful to more fully test the model with smaller companies that make up a significant portion of the Moroccan economy and employers. (High priority, to be addressed in the short term) 4. Build the capacity of host institutions to strengthen their institutional communication activities to mobilize youth. To mobilize a greater number of services users, USAID Career Center should advocate with host institutions to strengthen institutional communication activities (include information about the center in orientation materials, on the institutions’ websites, in existing outreach activities, in faculty/trainer orientation and directives). The program should maintain and expand current practices of joint activities with student clubs, youth-serving CSOs and existing university and training institution entrepreneurship programs to make existing centers multi-service trainee and student hubs. (Medium priority, to be addressed in the medium term) 22 Comparative data presented in the findings section of this report under EQ1 and in annex 3, summarizing the evaluation online survey of youth show that youth in vocational training centers generally indicated higher levels of satisfaction with career center services. They also demonstrate a greater sense that they were gaining valuable knowledge about future career paths and were able to make positive connections between career center services and outcomes like finding a job or an internship. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 23 EVALUATION QUESTION 2: VALIDATION OF THE CAREER CENTER MODEL During the first phase of the activity, USAID Career Center has effectively tested the career center model to ensure relevance to Moroccan stakeholders. The program has been effective in creating and/or responding to a sense of urgency to address work readiness and soft skill deficits in university and vocational training programs among key institutional partners. It largely has been effective in mobilizing relevant stakeholders to participate in piloting a solution. USAID and USAID Career Center practices for informing and obtaining feedback from key stakeholders have been effective overall with some gaps. Additional efforts are needed to more fully engage GOM central ministries and offices as well as host institutions stakeholders in the model’s evolution. As the USAID Career Center enters the midpoint of its period of performance, additional evaluation and assessment are needed to validate some parts of the model, and to ensure its continued adaption to the Moroccan context. As a number of KIIs indicated, USAID Career Center needs to do more to assess and communicate to the future owners of the model about what is required to run a center on an operational level, especially what it costs and what services are essential. Progress has been delayed on this front as a result of vacancies of key leadership positions among USAID Career Center partner institutions. More evidence of the model’s impact on meeting the private sector’s expectations for soft skills is needed. USAID Career Center program managers need to encourage host institutions’ on-going changes to the model so that the centers evolve and hopefully respond more effectively to key differences between the operating environment of North American educational institutions and their Moroccan counterparts. In light of the critical need to maintain or expand private sector linkages with career centers, these stakeholder needs, and expectations require close attention in the transition phase from a USAID pilot to a host institution service. It was not easy to get private sector feedback in the context of this evaluation; USAID Career Center reports a similar challenge. This indicates that the program may need to explore new strategies to engage and obtain feedback from the private sector. Although based on limited feedback, it is clear from this evaluation that private sector partners, although willing to participate in center events (including guest speakers, enterprise visits), they primarily are interested in career center services tailored to their business needs. Given ANAPEC’s mandate, to assist vocational training center and university graduates with job insertion, its ongoing plans to open university agencies, and the positive synergies between soft skills and work readiness training and insertion activities to meet both youth and private sector employers’ expectations, more attention to building bridges with the USAID Career Center activities is needed. Recommendations for consideration to ensure the relevance of the career center model include the following: 5. Strengthen the strategic planning and oversight roles of its institutional partners for center management. Continue the emerging good practices observed among some universities of reinforcing the institutional hierarchy’s oversight of centers through regular meetings and ongoing reporting. The role of regional coordinator for center oversight, should be redefined as is planned for year four and limited to transferring capacity to career center staff. (High priority, to be addressed in the short term) “When you invite youth to a workshop to learn something, you may not get many participants. When you invite them because a business is recruiting, you will get participants.” – Regional Program Manager, Tangiers Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 24 6. Engage key stakeholders in activities to evaluate the current career center model. The USAID Career Center should continue and expand its collaboration efforts, engaging with key stakeholders to monitor the career center model’s performance. Members of the steering committee should be involved through participation in career center activities, FGDs with youth and center managers, review of internal evaluation data, and validation of assessment reports modeling center sustainability. (High priority, to be addressed in the medium term) 7. Consider ways to maintain the current model of delivering soft skills and work readiness training through extra-curricular services after USAID funding ends. To reach undergraduates and preserve the current catalogue of center services in universities and vocational training institutions, USAID Career Center should consider ways to maintain the current model of delivering soft skills and work readiness training through extra-curricular services after USAID funding ends. Training services should be delivered through the career centers using center personnel, through partnerships between the center and external partners, and/or through faculty with specialized training. (High priority, to be addressed in the medium term) EVALUATION QUESTION 3: SUSTAINABILITY AND SCALABILITY At the mid-point of implementation, USAID Career Center has established a good foundation for the sustainability and scalability of the USAID Career Center model. The host institutions of the career centers and their relevant ministry departments have taken ownership of the career centers and are committed to pursuing the effort, an important factor favoring sustainability. It is surpassing its targets on integrating center operational costs into its institutional counterparts’ budget, a noteworthy accomplishment. While USAID Career Center and its institutional partners are discussing potential solutions to replace USAID funding and expertise, these strategies remain mostly theoretical. Solutions proposed by the program and its partners to ensure the current model’s relevance and sustainability - the income generation model, the private-public partnership model, and the full integration model show creative thinking and promise. They have not been fully modeled or tested yet, however, which limits the ability of the project and its counterparts to evaluate the feasibility and compatibility of these proposed strategies. This shortcoming is especially true for current plans to sustain the VCC post the USAID Career Center. Host institutions are likely to sustain the existing physical career centers in some form. Likewise, it is likely that additional centers and/or sub offices will be opened based on counterparts’ current intentions, extending services to additional students and trainees. According to key stakeholders, proposed changes in the institutional management structure of existing centers and strategies to scale the model are likely to significantly alter the USAID career center model. Significant challenges exist to maintaining the quality of career center services (such as sustaining current personnel in university centers, recruiting and training new counselors, funding activities post-USAID, and diversifying training offerings) although the project has made progress addressing some of these issues. Recommendations for consideration to improve the sustainability of the career centers include the following: 8. Develop a business plan that estimates the cost of maintaining USAID Career Center services post USAID funding and assesses alternative income generation strategies. The business model should be developed in collaboration with institutional partners taking into consideration how similar activities are managed within the universities and vocational training institutions and how these institutions currently organize continuing education programs for private sector clients for a fee. (High priority, to be addressed in the short term) Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 25 9. Before continuing the development of the VCC, the post program management model should be determined. Future development of online services should be done in collaboration with the entities that will assume the service’s management. In this way, the VCC will develop according to the capacity of its future owner(s) to manage its evolution. (High priority, to be addressed in the short term) 10. Upon identifying an appropriate VCC management model, consider the following improvements: – Improving the website’s design and functionality and offering content in Arabic; – Integrating online modules with university e-learning strategies and potential strategies to offer credits for soft skills courses. – Consider blended learning models that integrate learning on the VCC with Najahi training programs (which is being planned for year four); – Reinforcing social networking strategies to ensure all new content on the VCC is diffused via existing center social media platforms. (Medium priority, to be addressed in the medium term) 11. Build the capacity of each university to sustain and scale the career center model. To ensure strong buy-in from university leadership, USAID Career Center should focus their sustainability strategy on building the capacity of each university to sustain and scale the career center model within the university structure. USAID Career Center should continue to collaborate with the State Secretariat for High Education to scale the model within other universities. In the case of the OFPPT and Ministry of Tourism, it should continue to work with the central office to develop a plan to replicate the model in additional sites. (High priority, to be addressed in the short term) Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 26 ANNEX 1: EVALUATION DESIGN MATRIX Researchable Questions Information Required and Score(s) Scope and Methodology Limitations What this evaluation will likely allow the evaluator to say EQ1: To what extent are USAID Career Center participants – men and women between the ages of 15 and 29 – acquiring the work readiness and soft skills required by the private sector? What are the enabling and disabling factors contributing to workforce-ready students? Sub-Evaluation Questions: • Which Career Center program activities do you consider particularly successful/unsuccessful in terms of promoting the acquisition of work readiness and soft skills required by the private sector by students/trainees? Why? • What have been the main intervention strategies of the USAID Career Center program to promote the acquisition of work readiness and soft skills required by the private sector by students/trainees? • To what extent has the program been able to contribute to increasing opportunities within your institution for youth to acquire soft skills required by the private sector? How? Required Project Documents: • Project annual workplans • Project MELP • Project progress reports (Annual reports and most recent Quarterly Report) • Project needs assessments and other reports used to design services • Training curricula, training format, and Training of Trainers approach/agenda • Project data from training activity participant self￾assessments • Project data from private sector/civil society partner satisfaction surveys • Project data from Social Network Analysis Data Collection Methods: Online Survey: • Registered users of online career center (if applicable) • In person participants in two or more career center services (youth) • Sample size: survey to be sent to 10,000 youth, ages 15-29 Direct Observation • Center visits • Center Activity observation • 6 centers in 3 cities (Casablanca, Tangiers, Marrakech) to be visited • # of activities: TBD KIIs (up to 1.5 hours) • USAID • Implementers Limitations: Survey response rates: Response rates to the e￾survey may be low. The evaluation team is proposing a large sample size with the objective of having a reasonable number of responses. Logistical Limitations: Transport and scheduling limitations to be overcome by scheduling interviews that are near one another on the same day; conducting remote or phone interviews; using holidays as analysis or travel days. Availability of Participants: Fieldwork includes both a university/training institution holiday and exam period. FGD will be Evaluation Themes: • Description of the IPs’ main strategies to deliver workforce readiness services (including both physical and virtual centers and outreach activities) to target users. The extent that these are in line with general good practices used by career centers to facilitate workforce readiness elsewhere? • The extent that the project is meeting its related output and outcome indicator targets. • The extent that primary recipients of center /outreach program activities are in line with expectations and are reaching marginalized populations (women, persons with disabilities, Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 27 (Soft skills instructional activities integrated into existing curricula? Development of new extracurricular activities?) Please provide examples. What are the main obstacles/constraints? • To what extent do you think that the services that are offered in the physical and virtual career centers are expanding youth users’ understanding of employment trends, demand for skills, and available career pathways? How might services be improved? What are the main obstacles/constraints? • What is your perception of the work done by the USAID Career Center program in the area of increasing the work readiness and soft skills of youth? Do you have any suggestions for how the program could be more effective? • To what extent do you think that the services that are offered in the physical and virtual career centers are meeting the human resource needs of private sector employers? • Report on Organizational Performance Index evaluation Other documents for background and benchmarking: • A Model for Career Centers to Support Student Career Development: A Delivery Tool for Modern Career Centers, Jones Christopher • Youth and Transferable Skills: An evidence gap map, International Initiative for Impact Evaluation, 2015 • University/Training Institution Administrators/Educators • Employers • Intermediary Organizations • GOM institutions • Sample size: 86 KIIs FGDs (up to 2 hours) • Youth • Instructors in TOT • Sample size: 12 FGD of between 8-12 participants • KII and FGD Locations: Rabat, Casablanca, Tangiers, and Marrakech Data Analysis Techniques: • Content Analysis • Trend Analysis • Gap Analysis • Comparative Analysis • Gender Analysis arranged and participants confirmed in advance. Invitations will be extended to more than desired # of participants to account for last minute dropouts. Bias Recall Bias: CC beneficiaries may have received training or services from multiple sources. The evaluation team will ask respondents about any other programs they have been involved in or received assistance, drawing a distinction between what respondents may remember from the CC activity as opposed to another program. Response Bias: Key informants may be motivated to provide responses that would be considered socially desirable or influential in obtaining donor support. The evaluation team will explain the evaluation purpose and process, emphasizing the learning nature of the exercise, highlighting the value of honest feedback and assuring participants that their responses will not and other less advantaged populations? • How CC users perceive the quality and relevance of CC interventions/ services and follow-up to build their work readiness and soft skills. • Which services/ interventions are perceived to be the most and least successful and why. • What contextual factors have inhibited or facilitated program achievement to date and how effective the project has been to adapt. • What are the main lessons learned and emerging good practices to date. • How might services/interventions be improved in the time that remains before the project’s end. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 28 Definitions of key terms: • Work readiness: extent that implementing partner’ (IPs’) strategies and interventions are contributing to youth understanding of employment trends, demand for skills, potential for employment generation in growth sectors, and available career pathways. • Soft skills required by the private sector include extent that IPs’ strategies and interventions are contributing to the development of communications/ interpersonal skills, cooperation/ teamwork, self￾confidence, creative thinking, critical thinking, empathy, responsibility, respect for self and others, managing emotions, decision-making, and conflict management. affect their access to future services. Selection Bias: Implementers may orient the evaluation team to the most active, positive respondents. Where possible, the evaluation team will select interviewees from beneficiary lists provided by USAID Career Centers and arrange KIIs and FGDs independently. EQ2: How effectively has the USAID Career Center tested and validated the USAID Career Center model to ensure relevance to Moroccan stakeholders (students and their immediate environment, educators, government counterparts, private sector, and civil society organizations)? Sub-Evaluation Questions: • Which USAID Career Center program activities do you consider particularly successful/unsuccessful in involving and collaborating Required Project Documents: • Project annual workplans • Project MELP Data Collection Methods: Online Survey: • Registered users of online career center (if applicable) Limitations: Survey response rates: Response rates to the e￾survey may be low. The evaluation team is proposing a large sample Evaluation Themes: • Description of the IP’s main strategies for testing and validating the CC model. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 29 with its target population and key stakeholders in participatory planning and decision-making? Why? • To what extent did the Career Center program engage you or your institution in the design of its model and activities? Please describe the process. • Does the Career Center have on-going mechanisms to regularly obtain your feedback? Please describe. How could this be improved? • To what extent is your feedback (or feedback from your institution) being used by the Career Center to improve Centers’ performance? • Which activities in your Center do you consider particularly successful/unsuccessful in collaborating with and involving key stakeholders in participatory planning and decision-making? Why? • How relevant are the Career Center’s activities to you (your institution)? To Other stakeholders (students, educators, government counterparts, private sector, • Project progress reports (Annual reports and most recent Quarterly Report) • Project needs assessments and other reports used to design services • Training curricula, training format, and Training of Trainers approach/agenda • Project data from training activity participant self￾assessments • Project data from private sector/civil society partner satisfaction surveys • Project data from Social Network Analysis • Report on Organizational Performance Index evaluation • Reports of relevant stakeholder workshops • In person participants in two or more career center services (youth) • Sample size: survey to be sent to 10,000 youth, ages 15-29 Direct Observation • Center visits • Center Activity observation • 6 centers in 3 cities (Casablanca, Tangiers, Marrakech) to be visited • # of activities: TBD KIIs (up to 1.5 hours) • USAID • Implementers • University/Training Institution Administrators/Educators • Employers • Intermediary Organizations • GOM institutions • Sample size: 86 KIIs FGDs (up to 2 hours) • Youth • Instructors in TOT • Sample size: 12 FGD of between 8-12 participants • KII and FGD Locations: Rabat, Casablanca, Tangiers, and Marrakech Data Analysis Techniques: size with the objective of having a reasonable number of responses. Logistical Limitations: Transport and scheduling limitations to be overcome by scheduling interviews that are near one another on the same day; conducting remote or phone interviews; using holidays as analysis or travel days. Availability of Participants: Fieldwork includes both a university/training institution holiday and exam period. FGD will be arranged and participants confirmed in advance. Invitations will be extended to more than desired # of participants to account for last minute dropouts. Bias Recall Bias: CC beneficiaries may have received training or services from multiple sources. The evaluation team will ask respondents about any other programs they have been involved in or received assistance, drawing a distinction • The extent that the project is meeting its related output and outcome indicator targets. • The extent to which the IP has effectively involved all relevant parts of the youth employment eco-system. • How various CC users/stakeholders perceive the relevance of the CC model (including testing and validation process) relative to their needs and expectations. • The extent to which the IPs’ have adapted the CC model in response to feedback from key stakeholders. • The extent that the IP has implemented strategies to share learning from CC with relevant stakeholders. • Identification of successful and unsuccessful strategies for engaging all relevant stakeholders. • Perception of inhibiting and facilitating factors for engaging all relevant stakeholders. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 30 and civil society organizations). How could they be improved to be more relevant? Definitions of key terms: Relevance includes: -to what extent the IP engaged in participatory planning and decision-making processes that involve their target population and key stakeholders; -to what extent the IP includes stakeholder’s feedback in activity design • Content Analysis • Trend Analysis • Gap Analysis • Comparative Analysis • Gender Analysis between what respondents may remember from the CC activity as opposed to another program. Response Bias: Key informants may be motivated to provide responses that would be considered socially desirable or influential in obtaining donor support. The evaluation team will explain the evaluation purpose and process, emphasizing the learning nature of the exercise, highlighting the value of honest feedback and assuring participants that their responses will not affect their access to future services. Selection Bias: Implementers may orient the evaluation team to the most active, positive respondents. Where possible, the evaluation team will select interviewees from beneficiary lists provided by USAID Career Centers and arrange KIIs and FGDs independently. • Identification of lessons learned and good practices in engaging with key stakeholders. • Suggestions for how process for ensuring relevance might be improved in time that remains before project end. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 31 EQ3: How likely are the career centers to be sustainable? Sub-Evaluation Questions: • Which USAID Career Center activities do you consider particularly successful/unsuccessful in promoting the sustainability and scaling of the USAID Career Center model? How do you define success? Why? • To what extent and how has the USAID Career Center built the capacity of its partners to sustain activities beyond the conclusion of the USAID Career Center program? What are the challenges? What are the outcomes of the USAID Career Center capacity building to date? What do you think are the prospects for sustainability? • What types of support have you received from the USAID Career Center central program managers and specialists? To what extent has support been sufficient to meet your needs? Are there ways that the support might be improved or adapted to be more effective in building Required Project Documents: • Project annual workplans • Project MELP • Project progress reports (Annual reports and most recent Quarterly Report) • Project needs assessments and other reports used to design services • Training curricula, training format, and Training of Trainers approach/agenda • Project data from training activity participant self￾assessments • Project data from private sector/civil society partner satisfaction surveys • Project data from Social Network Analysis • Report on Organizational Performance Index evaluation • If available, assessment reports related to sustainability (capacity building needs assessments, center Data Collection Methods: Online Survey: • Registered users of online career center (if applicable) • In person participants in two or more career center services (youth) • Sample size: survey to be sent to 10,000 youth, ages 15-29 Direct Observation • Center visits • Center activity observation • 6 centers in 3 cities (Casablanca, Tangiers, Marrakech) to be visited • # of activities: TBD KIIs (up to 1.5 hours) • USAID • Implementers • University/Training Institution Administrators/Educators • Employers • Intermediary Organizations • GOM institutions • Sample size: 86 KIIs FGDs (up to 2 hours) Limitations: Survey response rates: Response rates to the e￾survey may be low. The evaluation team is proposing a large sample size with the objective of having a reasonable number of responses. Logistical Limitations: Transport and scheduling limitations to be overcome by scheduling interviews that are near one another on the same day; conducting remote or phone interviews; using holidays as analysis or travel days. Availability of Participants: Fieldwork includes both a university/training institution holiday and exam period. FGD will be arranged and participants confirmed in advance. Invitations will be extended to more than desired # of participants to account for last minute dropouts. Bias Evaluation Themes: • Description of IPs’ main strategies and initiatives to promote CC (including virtual CC) sustainability to date. • Description of IPs’ phase out strategy (if applicable). • The extent that the project is meeting its related output and outcome indicator targets. • Stakeholder perceptions of effectiveness of project sustainability interventions to date (engagement with and capacity building of stakeholders who may be involved in/contribute to sustaining CC services). • The extent of evidence of stakeholder commitment to sustain CC services measured by verbal statements, signed agreements, investments to date, etc. • The extent of evidence of stakeholder capacity to sustain CC (includes Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 32 your capacity as career center staff? • How confident are you that you will be able to continue your work when the support provided by USAID Career Centers program concludes? • What additional efforts are needed to ensure that you are able to continue your work after the end of the USAID Career Center program support? • What role if any do you think organizations like yours could have in ensuring the sustainability of USAID Career Center program services after the end of USAID support? • To what extent is the USAID Career Center program relevant to the political and programmatic priorities of your institution? Please explain how the program responds/ doesn’t respond to priorities. • What role if any do you think private enterprises like yours could have in ensuring the sustainability of USAID Career Center program services after the end of USAID support? management cost/revenue projections) • Any memorandum of understanding (MOU) signed with partners related to sustainability • If available, assessment reports related to sustainability (capacity building needs assessments, center management cost/revenue projections) • Any MOU signed with partners related to sustainability • Youth • Instructors in TOT • Sample size: 12 FGD of between 8-12 participants • KII and FGD Locations: Rabat, Casablanca, Tangiers, and Marrakech Data Analysis Techniques: • Content Analysis • Trend Analysis • Gap Analysis • Comparative Analysis • Gender Analysis Recall Bias: CC beneficiaries may have received training or services from multiple sources. The evaluation team will ask respondents about any other programs they have been involved in or received assistance, drawing a distinction between what respondents may remember from the CC activity as opposed to another program. Response Bias: Key informants may be motivated to provide responses that would be considered socially desirable or influential in obtaining donor support. The evaluation team will explain the evaluation purpose and process, emphasizing the learning nature of the exercise, highlighting the value of honest feedback and assuring participants that their responses will not affect their access to future services. Selection Bias: Implementers may orient the evaluation team to the most active, positive respondents. Where analysis of quantitative data from IPs’ counterpart capacity assessments (if applicable). • Inhibiting and facilitating factors for CC (including virtual CC) sustainability. • Suggestions for how sustainability may be reinforced in time that remains before project end. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 33 Definitions of key terms: Sustainability: • host institutions and their respective Ministry counterparts’ commitments and capacity to sustain and scale the physical and virtual model of career centers. • private sector and other stakeholder willingness to embrace a sustainable, demand-driven model including willingness to contribute financial and other resources for CC services. possible, the evaluation team will select interviewees from beneficiary lists provided by USAID Career Centers and arrange KIIs and FGDs independently. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 34 ANNEX 2: FINDINGS MATRIX Findings Analysis and Conclusions Recommendations EQ1: To what extent are USAID Career Center participants – men and women between the ages of 15 and 29 – acquiring the work readiness and soft skills required by the private sector? What are the enabling and disabling factors contributing to workforce-ready students? Progress meeting outcome & output indicator targets (in which areas is the project exceeding targets, in which is it behind) • By December 31, 2017, USAID Career Center had successfully established five of six planned pilot career centers as well as the virtual career center. • In years 2 & 3, USAID Career Center reached or surpassed its indicator targets measuring Centers’ effectiveness such as the number of youth receiving services, and percentage of youth who return to use two or more services, and percentage of youth satisfied with the quality and relevance of training and services. • By the end of year three, the Activity had developed and tested workforce readiness modules, known as Najahi – Prêt pour l’Emploi. • The activity largely surpassed its targets for the number of courses, workforce initiatives, and career services using labor market • The CC has made substantial progress toward the achievement of its primary objectives. • The CCs correspond to the majority of beneficiaries’ expectations and needs. There have been many positive outcomes. • Overall the quality of CC human resources are high. • The CC model is contributing to helping CC participants to acquire the work readiness and soft skills required by the private sector with some gaps: • The CC is limited in its future capacity to impact its target population because of its capacity (number of counselors, number of centers, distance between training institutions and the center) to reach students in mass (number of students, 1. Expand the Foundational Najahi Training Core Curriculum. USAID Career Center should integrate existing foundational soft skills modules such as “Managing my Emotions and my Stress,” “Working as a Team,” “Me and Problem Solving” and “Negotiate with Gusto” into the core (versus supplemental) curriculum of Najahi training and advocate for expanding planned institutional offerings to likewise include these foundational modules. (high priority, to be addressed in the short term) 2. Elevate Business Developers as Key Personnel alongside Career Counselors. USAID Career Center should integrate the post of business developer as key center personnel alongside the career counselors in order to ensure strong linkages with the private sector and CSOs. (high priority, to be addressed in the short term) 3. Assess private sector demand for career center services among SMEs. USAID Career Center has tested engagement with structured, large and Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 35 Findings Analysis and Conclusions Recommendations EQ1: To what extent are USAID Career Center participants – men and women between the ages of 15 and 29 – acquiring the work readiness and soft skills required by the private sector? What are the enabling and disabling factors contributing to workforce-ready students? information in year two but only achieved 70% of its third-year target • In both years two and three, the Activity surpassed its targets for the number of private organizations partnering with universities and vocational training centers; in year two it partnered with 89 organizations, more than double the number planned. It was slightly under its objective of a 50% rate of return (partners who engage in more than one collaboration) in year two but met the target in year three Perceptions of Youth • Project indicator on youth satisfaction shows high levels of satisfaction • Evaluation youth online survey shows high levels of satisfaction with some difference between males and females, university students vs. vocational education trainees • Youth FGDs: Several youth in Casablanca, both university and OFPPT: Feel the quality of service is very high. “We did not believe it could be a free service.” “This kind of service costs a lot of money” number of counselors per student, depth of training). • The current tendency by CC and partner institutions to focus narrowly on technical job search may not be fostering some key foundational soft skills, limiting potential impact of training. • The timing of CC orientation services is problematic in Morocco. Education is not as flexible – to make informed choices, youth need orientation before they finish their secondary education. (This may out of the scope of USAID Career Center but should be considered in future programming. multinational companies in its pilot activities. Additional engagement with SMEs would be useful to more fully test the model with smaller companies that make up a significant portion of the Moroccan economy and employers. (high priority, to be addressed in the short term) 4. Build the capacity of host institutions to strengthen their institutional communication activities to mobilize youth. In order to mobilize a greater number of users of USAID Career Center services, USAID Career Center should advocate with center host institutions to strengthen institutional communication activities (include information about the center in orientation materials, on the institutions’ websites, in existing outreach activities, in faculty/trainer orientation and directives). The program should maintain and expand current practices of joint activities with student clubs, youth￾serving CSOs as well as existing university and training institution entrepreneurship programs with the aim of making existing centers a multi-service trainee/student hub. (medium priority, to be addressed in the medium term) Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 36 Findings Analysis and Conclusions Recommendations EQ1: To what extent are USAID Career Center participants – men and women between the ages of 15 and 29 – acquiring the work readiness and soft skills required by the private sector? What are the enabling and disabling factors contributing to workforce-ready students? • Youth from FGD in all 3 cities: appreciation for individualized coaching, contact with private sector employers. • Youth from ENACTUS – Entrepreneurship development is not so strong within center. Needs to link with entrepreneurship ecosystem. Need for Constant Innovation • CC staff (Casa, Tangiers): youth expect new things all the time. • Youth FGD Casa: Summer camp was great, looking forward to it. But we hope that it will not repeat the same workshops. There is a need to keep innovating. • Youth FGD Tangiers: Our center has been open the longest. We think there needs to be more variety in CC activities. It should not always be the same thing. Feedback on CC Counselors • Many informants (USAID, Activity managers, youth): quality of career center personnel is the most important factor determining the quality of center services. Additional recommendations for consideration include: • While beyond the scope of the USAID Career Center, in the future CC should consider the feasibility of providing orientation services to secondary school students through outreach events. Before starting, it should assess what already exists for secondary students in order to identify the appropriate niche and implementation mechanism. • USAID CC should consider ways to strengthen placement services in order to respond to private sector and youth demand: internships, recruitment services, job fairs. - Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 37 Findings Analysis and Conclusions Recommendations EQ1: To what extent are USAID Career Center participants – men and women between the ages of 15 and 29 – acquiring the work readiness and soft skills required by the private sector? What are the enabling and disabling factors contributing to workforce-ready students? • Activity managers indicated that it was a challenge to keep quality personnel and that several good counselors had already left the program. • Youth FGD Casablanca: the counselors and the trainers are doing excellent work. Qualities are capacity to listen, respond, form personal relationships with students, their flexibility, their skills. • Youth FGD Tangiers (university): Counselors generally good. Some could be more competent. • Youth FGD Tangiers (OFPPT): The coach’s support helped me and my friends to change our attitude. We are more serious in our studies. Feedback from CC staff • Most career center personnel expressed satisfaction with training and exchange opportunities offered by the program. KII. • KII CC staff: visit to CC in US was “eye opening” and very helpful. Participants feel they learned a lot. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 38 Findings Analysis and Conclusions Recommendations EQ1: To what extent are USAID Career Center participants – men and women between the ages of 15 and 29 – acquiring the work readiness and soft skills required by the private sector? What are the enabling and disabling factors contributing to workforce-ready students? • Several (across all cities): Identified gaps in their training that needed to be addressed before the end of the program: finance and administrative management, training program design, coaching and counseling techniques and certification. CC staff: we would like certification of our skills KII CC directors: we need a business plan; we need financial and admin management training • CC Activity managers, CC staff, regional CC staff indicated that management roles were changing with more responsibility given to center managers. CC staff in Casablanca and Tangiers expressed need for greater autonomy. • KII regional staff Tangiers: About the financial management capacity “it would be a good idea to let the centers start to manage themselves before the end of the program” • KII CC staff: the function of the business developer belongs in the CC team Youth Feedback on Outcomes of Services • FGD Casablanca: “I learned a lot in terms of self￾confidence and how to approach the job market.” Another youth: “I changed my way of thinking. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 39 Findings Analysis and Conclusions Recommendations EQ1: To what extent are USAID Career Center participants – men and women between the ages of 15 and 29 – acquiring the work readiness and soft skills required by the private sector? What are the enabling and disabling factors contributing to workforce-ready students? Having a diploma isn’t enough. We need other things – especially soft skills.” • FGD youth summer school Marrakech: “I have friends that were able to find internships thanks to the CC.” • FGD youth all cities: Youth find the space “youth friendly.” Importance of Private Sector Connection • KII Business Dev Tangiers: “When you invite youth to a workshop to learn something, you may not get many participants. When you invite them because a business is recruiting, you will get participants.” • CC Director: the weak link in the chain of CC services: we are not able to help with internships and job placement. It is a structural problem – the business developer is not close enough to the center to be effective. We think the business developer should be part of the team. Feedback on VCC • According to most stakeholders (youth, IP, University admin, institutional partners), the Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 40 Findings Analysis and Conclusions Recommendations EQ1: To what extent are USAID Career Center participants – men and women between the ages of 15 and 29 – acquiring the work readiness and soft skills required by the private sector? What are the enabling and disabling factors contributing to workforce-ready students? VCC is a relevant strategy to reach youth who are far from centers • FGD Youth: Youth who are farther away from the center appreciated the VCC • CC team: a youth that visited the center was more likely to use the VCC – it was a complementary tool rather than a substitute. OFPPT youth use it to register but do not systematically return to the site to consume content; lack of motivation for online learning. For some youth, Arabic content would be more accessible. • Institution admin: youth do not use websites – they are on social media. • The CC director remarked that there are many passive subscribers of the site; active users usually had an experience in the physical center. • CC director: it is a challenge to get youth to take advantage of online services. It is not a reflex to check the site; even to check their mail Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 41 Findings Analysis and Conclusions Recommendations EQ1: To what extent are USAID Career Center participants – men and women between the ages of 15 and 29 – acquiring the work readiness and soft skills required by the private sector? What are the enabling and disabling factors contributing to workforce-ready students? • CC staff: OFPPT trainees change their telephone number and email often – the data base is not accurate. Private sector level of satisfaction with youth • Project reports high levels of private sector satisfaction with youth who have received CC training based on survey data and verbal feedback following pilot testing. • Employer KII: informants indicated they could not evaluate the CC youth recruits’ soft skills. Two said that they were no different than others. • Private sector partners were satisfied with career center sourcing services, indicating it made their recruitment efforts more efficient. • CC activity managers report that obtaining employers’ feedback to be a challenge. • KII CC Director OFPPT Tangiers consider that the placement services are not part of the CC mission, but it contributes to attract the youth to the other activities organized by the CC. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 42 Findings Analysis and Conclusions Recommendations EQ1: To what extent are USAID Career Center participants – men and women between the ages of 15 and 29 – acquiring the work readiness and soft skills required by the private sector? What are the enabling and disabling factors contributing to workforce-ready students? • KII BD Tangiers “In our approach, we are looking into how to mobilize graduates. It is perhaps something we can develop in the program.” Challenges Orientation • KII BD Tangiers: Cultural aspect “There is a problem that youth do not always choose their field of study well…Youth see a field of study in terms of a job not so much their personal aptitudes.” Diploma = job • FGD Youth Casa, Tangiers: youth felt frustration to discover their aptitudes and potential links to professional options after they had already been enrolled in a program. • FGD Marrakech: “Thanks to the trainer, we were able to open our spirit and imagine a many more career choices.” • FGD youth Tangiers: Orientation should start sooner. Also said by youth in Casa, Marrakech. • KII CC Director – OFPPT Tangiers: the orientation modules would have been more useful if there were bridges between the ‘filières’ Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 43 Findings Analysis and Conclusions Recommendations EQ1: To what extent are USAID Career Center participants – men and women between the ages of 15 and 29 – acquiring the work readiness and soft skills required by the private sector? What are the enabling and disabling factors contributing to workforce-ready students? Communication • The Career center in UHII is not in a central place. • FGD Youth: There are no orientation programs offered by the universities to inform students of the existence of the center. • FGD youth, all centers: the centers were largely unknown and unused by many of their fellow students/trainees. Communication not adequate. Need to use social media better. • KII vocational training career center manager: Attracting youth was less of a problem in recent months; rather, their challenge was now how to organize to satisfy demand. • USAID CC managers: Wanted to be prudent initially in their communication efforts during the center start-up and testing phase so as not to attract demand that they could not meet. Capacity and Reach • KII institution, FGD youth: The capacity of the career center relative to the size of the student body and the distance between campus is a challenge. Students who are close Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 44 Findings Analysis and Conclusions Recommendations EQ1: To what extent are USAID Career Center participants – men and women between the ages of 15 and 29 – acquiring the work readiness and soft skills required by the private sector? What are the enabling and disabling factors contributing to workforce-ready students? to the center have more possibilities to use the center. • KII CC staff: the center is reaching a variety of majors; the priority is given to students from faculties that have more difficulty finding a job. • KII university CC staff (Tangiers and Marrakech): Center is located on “elite” campus; some students from open access degree programs don’t feel comfortable entering campus. • KII university CC staff, youth: The cost of transportation for youth from farther away campuses prohibits their participation. • KII CC Director OFPPT Tangiers: The most served category is Techniciens (niveau Bac) et techniciens spécialisés (bac) and those who are trained in close institutions (the CC is located between 2 OFPPT institutions) Strategies to overcome distance challenge Summer Camp • FGD youth: Summer Camp was allowed youth who were not close to the CC and did not have time to visit to benefit from CC services. However, there was a problem of capacity. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 45 Findings Analysis and Conclusions Recommendations EQ1: To what extent are USAID Career Center participants – men and women between the ages of 15 and 29 – acquiring the work readiness and soft skills required by the private sector? What are the enabling and disabling factors contributing to workforce-ready students? Outreach • CC direction – learned from the visit in the US to put in place outreach strategies in the trainer centers to recruit youth – involving parents, involving instructors, creating structured programs for the same group of youth. Youth Ambassadors • University VP Casablanca: Youth Ambassadors – means to conduct outreach to distant campuses, part of communication strategy • KII BD Tangiers “The strategy of youth Ambassadors is important – it develops a greater sense of attachment to the university. For example, we have an ex ambassador who is now working who has been very helpful.” • FGD w/ambassadors: No institutional recognition, “we are students like others” We are not supported by the institution for our work – we can even be blocked. They are limited by their class hours…sometimes there is a conflict. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 46 Findings Analysis and Conclusions Recommendations EQ1: To what extent are USAID Career Center participants – men and women between the ages of 15 and 29 – acquiring the work readiness and soft skills required by the private sector? What are the enabling and disabling factors contributing to workforce-ready students? • KII University VP: An association is being created modeled after ENACTUS. They will have a key role in the planned sub-offices. • CC director: will create a student club to structure Youth Ambassadors. Meeting special needs of women people w/disabilities • KII CC staff, FGD youth UHII: the CC has organized activities relative to gender. It has invited women entrepreneur guest speakers. They have been sensitized about the needs of people with disabilities. • KII CSO partner (Rabat): organized workshop on sexual violence against women in OFPPT CC in Casablanca. • KII CSO partner (Tetouan): CC organized workshops for association beneficiaries in CC in Tangiers and facilitated participation of association youth in job fair in Casablanca. Issues with website accessibility and physical accessibility for some disabilities Extent that soft skills training offerings are in line with good practices elsewhere Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 47 Findings Analysis and Conclusions Recommendations EQ1: To what extent are USAID Career Center participants – men and women between the ages of 15 and 29 – acquiring the work readiness and soft skills required by the private sector? What are the enabling and disabling factors contributing to workforce-ready students? • Based on ET benchmarking, the program’s “Najahi” soft skills training package aligns with what experts in the field generally consider as essential for effective soft skills acquisition. Benchmark: Global Partnership for Youth Employment document “Strengthening Life Skills for Youth” • Based on feedback from CC program manager and EFE trainers, in the majority of training workshops, career centers only delivering some Najahi modules to youth. 70% of training Najahi workshops use My Employment Kit which mainly covers technical job search skills. (source table No of participants by activities, provided by CC managers) • Several core modules covering the foundations of soft skill development are only offered as supplemental modules. Out of 13,000+ participants in Najahi training workshops, only 234 used the supplemental modules. • According to CC project management, other workshops and activities organized in CCs complement Najahi training. 12,400 youth have taken part in workshops. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 48 Findings Analysis and Conclusions Recommendations EQ1: To what extent are USAID Career Center participants – men and women between the ages of 15 and 29 – acquiring the work readiness and soft skills required by the private sector? What are the enabling and disabling factors contributing to workforce-ready students? • Based on CC table, most workshops also focus on technical job search skills (3500+ participants). In addition, workshops on time management and organization (932 participants), public speaking (113), team work • The Najahi modules that university and vocational training institutional partners have selected to integrate into their programs also mainly focus on technical job search and orientation. (source University VP, TOT participants, CC program managers) Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 49 Findings Analysis and Conclusions Recommendations EQ2: How effectively has USAID Career Center tested and validated the USAID Career Center model to ensure relevance to Moroccan stakeholders (students and their immediate environment, educators, government counterparts, private sector, and civil society organizations Engagement with stakeholders: Design Phase • KII project management: USAID proposed the model of Career Center; agreed with Ministry of Higher Education for the selection of universities; it was necessary to convince the universities. The approach was corrected with OFPPT using a call for proposals and selecting institutions based on their proposals • ANAPEC: informed of planned USAID Career Center activities in an initial meeting and later, through participation on the Activity steering committee. Disappointed in the quality of consultations during the development of the model. According to one official, the Agency had expressed interest in exploring synergies between its planned collaboration with Higher Education to establish university agencies and the career centers but had not been given an appropriate opportunity despite its formal requests. • One Higher Education official indicated that there had been discussions to consider whether the development of USAID Career Centers and ANAPEC • USAID and USAID Career Center practices for informing and obtaining feedback from key stakeholders have been effective in the overall with some gaps. • Additional efforts are needed to more fully engage some key stakeholders in the model’s evolution. • Evaluation and assessment is likewise required at this stage to validate some parts of the model and to ensure its fuller adaption to the Moroccan context • USAID Career Center has effectively tested the career center model to ensure relevance to Moroccan stakeholders using participative methods. • The program has been effective in creating and/or responding to a sense of urgency to address work readiness and soft skill deficits in university and vocational training programs among key institutional partners. • It has likewise been largely effective in mobilizing relevant stakeholders to participate in piloting a solution. 5. Strengthen the strategic planning and oversight roles of its institutional partners for center management. Continue the emerging good practices in some universities of reinforcing oversight of centers by the institutional hierarchy through regular meetings and ongoing reporting. The role of regional coordinator for center oversight, should be redefined as is planned for Year Four and limited to transferring capacity to career center staff. (high priority, to be addressed in the short term) 6. Engage key stakeholders in activities to evaluate the current career center model. The USAID Career Center should continue and expand collaboration efforts, engaging with key stakeholders to monitor the performance of the career center model. Members of the steering committee should be involved through participation in career center activities, focus group discussion with youth and center managers, review of internal evaluation data, and validation of assessment reports modeling center Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 50 Findings Analysis and Conclusions Recommendations EQ2: How effectively has USAID Career Center tested and validated the USAID Career Center model to ensure relevance to Moroccan stakeholders (students and their immediate environment, educators, government counterparts, private sector, and civil society organizations agencies should be coordinated. In the end, the Ministry decided that this debate needed to be postponed until after the initial piloting of USAID Career Centers. • According to KII with ANAPEC, this decision had never been clearly communicated to the Agency. Noted that in Tangiers and Casablanca, there will be both a USAID Career Center and an ANAPEC university agency; and that ways to coordinate and capitalize on the other’s services had not been discussed yet. • KII EFE: initial decisions about the design of the modules were made in Washington and written in English; Afterwards the design was done with EFE personnel and the work readiness specialist of FHI360 • KII CSO: Concerns of persons with disabilities were not taken into consideration at the very start of the program. Engagement with stakeholders: Implementation and Oversight • Feedback from most Career Center institutional partners showed a high • USAID Career Center needs to do more to assess and communicate to the future owners of the model on what is required to run a center on an operational level, especially what it costs and what services are essential. • More evidence of the model’s impact on beneficiaries’ insertion in the job market and on meeting the private sector’s expectations for soft skills is needed to more fully “validate” the model. • On-going changes to the model by host institutions need to be encouraged by USAID Career Center program managers so that the centers evolve and hopefully respond more effectively to key differences in the operating environment of North American educational institutions and their Moroccan counterparts. • Given the ANAPEC’s mandate (economic insertion of vocational training center and university graduates), its ongoing plans to open university agencies, and the positive synergies between soft skills and work readiness training and insertion activities to meet both youth and private sector employers’ expectations, more attention to building bridges between USAID Career Center activities and Agency activities were needed. sustainability. (high priority, to be addressed in the medium term) 7. Consider ways to maintain the current model of delivering soft skills and work readiness training through extra-curricular services after the end of USAID funding. To reach undergraduates and preserve the current catalogue of center services in universities and vocational training institutions, USAID Career Center should consider ways to maintain the current model of delivering soft skills and work readiness training through extra-curricular services after the end of USAID funding. Training services should be delivered through the career centers using center personnel, through partnerships between the center and external partners, and/or through faculty with specialized training. (high priority, to be addressed in the medium term) Additional recommendations include: • USAID Career Center should explore potential collaboration with ANAPEC Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 51 Findings Analysis and Conclusions Recommendations EQ2: How effectively has USAID Career Center tested and validated the USAID Career Center model to ensure relevance to Moroccan stakeholders (students and their immediate environment, educators, government counterparts, private sector, and civil society organizations degree of satisfaction with USAID Career Center management efforts to keep them informed. • Program reports cite many good communication practices such as forming a National Steering Committee, outreach activities and a newsletter. • The evaluation team found program reporting to be detailed and easy-to￾understand. • KII regional coordinator: the steering committee meetings are good but there is not sufficient follow-up between meetings • KII project management: youth involved in the selection of the logo, colors and architecture of the centers; youth involved in implementation of program through Youth Ambassador Feedback Mechanisms: • KII project management: youth receive a feedback survey after each training; private sector partners also – 30% return from youth; much less from employers • KII CC management: difficulty getting feedback from enterprise partners university agencies as part of its sustainability strategy. • To ensure strong buy-in from university leadership, USAID Career Center should focus their sustainability strategy on building the capacity of each university to sustain and scale the career center model within the university structure. In the case of the OFPPT and Ministry of Tourism, it should continue to work with the central office to develop a plan to replicate the model in additional sites. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 52 Findings Analysis and Conclusions Recommendations EQ2: How effectively has USAID Career Center tested and validated the USAID Career Center model to ensure relevance to Moroccan stakeholders (students and their immediate environment, educators, government counterparts, private sector, and civil society organizations • KII university admin: until recently, the university admin was not sufficiently involved with CC managers; now there is a monthly coordination meeting which has facilitated oversight and increased implication of institutional actors. • A lesson learned was about the importance of integrating the university leadership in the supervision of the CC. Before it was more the CC →USAID project • One GOM counterpart expressed disappointment with the program’s efforts to keep national stakeholders involved in decisions about the future of the centers. • EFE trainers indicated that they had provided the program office feedback on needed improvements in Najahi training modules that had not been taken into consideration by the program. Engagement with stakeholders: Evaluation and Adaptation • KII CC management team, institution admin: Key stakeholders are not aware of Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 53 Findings Analysis and Conclusions Recommendations EQ2: How effectively has USAID Career Center tested and validated the USAID Career Center model to ensure relevance to Moroccan stakeholders (students and their immediate environment, educators, government counterparts, private sector, and civil society organizations how much it costs to run a center: center managers do not know how much it costs to run their center; Institutions are likewise not aware of how much it costs. How can they validate a model if they do not know what resources are needed to create and implement? • KII professional association: Not sufficiently involved in evaluating the model; invited to CC inauguration but have not been asked to participate in a CC activity since. CC not well known by private sector – would be useful to promote collaboration with CC through CGEM 33 industry/sector associations Relevance of current model to key stakeholder needs Feedback from central GOM counterparts • According to one national counterpart KII, the Ministry of Higher Education had been considering following the American/Anglo￾Saxon career center model for some time and was pleased that USAID had provided needed technical and financial support. Feedback from career center host institutions • One university VP described the career center as providing “undeniable value- Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 54 Findings Analysis and Conclusions Recommendations EQ2: How effectively has USAID Career Center tested and validated the USAID Career Center model to ensure relevance to Moroccan stakeholders (students and their immediate environment, educators, government counterparts, private sector, and civil society organizations addition” because it was enabling the university to fill gaps in students’ soft skills. Another VP agreed but added the relevance of current services needed to be assessed. • One GOM counterpart affirmed, “I am conscious of the objective of the project to strengthen the links between our trainees and the private sector. The USAID Career Center has become even more than what we expected. It has become an interface between the training centers and employers.” Feedback from faculty involved in USAID Training of Trainer (TOT) programs • Participants in university TOT indicated that the training on Najahi training modules was useful, although some said the content was similar to their existing courses. • The faculty members strongly agreed with the relevance of integrating soft skills into the university curriculum but highlighted many obstacles to do so effectively, most significantly class sizes and challenges evaluating the acquisition of soft skills. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 55 Findings Analysis and Conclusions Recommendations EQ2: How effectively has USAID Career Center tested and validated the USAID Career Center model to ensure relevance to Moroccan stakeholders (students and their immediate environment, educators, government counterparts, private sector, and civil society organizations • Faculty FGD participants highlighted that professional development opportunities tended to be few and far between and that they were very grateful to update their instruction techniques. • Not satisfied with the level of support that they had been provided by the Activity following the TOT • According to a USAID Career Center program manager, the universities had been offered a follow-up support program but did not accept. Feedback from private sector partners • KII with private sector partners highlighted positive interactions with the centers. • Private sector representatives appreciated the flexibility of the centers to respond to specific needs such as helping with recruitment drives and providing professional development opportunities for Human Resource managers. (Hilton, AMITHM, Siemens) • Several indicated that their interaction had been limited and would have liked to be more-involved and better-informed on Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 56 Findings Analysis and Conclusions Recommendations EQ2: How effectively has USAID Career Center tested and validated the USAID Career Center model to ensure relevance to Moroccan stakeholders (students and their immediate environment, educators, government counterparts, private sector, and civil society organizations career center activities. (CGEM, AMITHM) • Most private sector KIIs; importance of the USAID Career Center mission to develop work readiness and soft skills in youth. • Some wanted center services to be even more tailored to their specific needs such as providing pre-selection services for intern and employee recruitment or offering soft skills training to their employees. • A representative of an employers’ association underlined the importance of evaluating the impact of the model on beneficiaries’ soft skills. Feedback from CSO partners • CSO partners found a win/win arrangement in their collaboration with the USAID Career Center. • CSO KII participants reported benefits such as increased visibility among youth and a space for their activities while career center beneficiaries reported appreciation for CSO programs. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 57 Findings Analysis and Conclusions Recommendations EQ2: How effectively has USAID Career Center tested and validated the USAID Career Center model to ensure relevance to Moroccan stakeholders (students and their immediate environment, educators, government counterparts, private sector, and civil society organizations • Campus-based student clubs linked with national and international CSOs in particular seemed actively engaged in collaboration with the university career centers. • Based on a variety of KII reports, through collaboration with persons with disabilities and women’s rights CSOs, career centers integrated activities addressing some specific needs of women and persons with disability. • KI UHII admin: The university administration had a favorable impression of the center services. Emphasized the need to adapt the model to fit the Moroccan context, especially the ratio counselors/student Ex. University of Penn 30k students 60 counselors University of Casablanca 102k and 3 counselors • KII BD Tangiers: One of the differences between the US/N America and Morocco that require the model to be adapted is the management style (in Morocco authoritarian, need of the hierarchy support) Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 58 Findings Analysis and Conclusions Recommendations EQ2: How effectively has USAID Career Center tested and validated the USAID Career Center model to ensure relevance to Moroccan stakeholders (students and their immediate environment, educators, government counterparts, private sector, and civil society organizations Contextual Factors Affecting Relevance Focus on large structured enterprise, big city economies • USAID Career Center model has been tested in locations with relatively vibrant economies and with a limited set of private sector partner profiles (large structured companies, multinationals). • GoM reports: 80% of the Moroccan private sector is made up of very small, small and medium-sized enterprises (VSME-SME) and is still the largest employer in Morocco • Activity manager & USAID official: intention of the program to adapt its services to the needs of sectors with high growth potential in cities with large numbers of youth • KII Center director Marrakech: SMEs have more employment potential • KII University VP: SMEs more able to absorb generalist profiles. They are the ones that need help finding a job. Program Management structure: • According to one career center director, center personnel will need to adapt to a Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 59 Findings Analysis and Conclusions Recommendations EQ2: How effectively has USAID Career Center tested and validated the USAID Career Center model to ensure relevance to Moroccan stakeholders (students and their immediate environment, educators, government counterparts, private sector, and civil society organizations new management style in the future (more authoritarian, greater attention to hierarchy). Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 60 Findings Analysis and Conclusions Recommendations EQ3: How likely are the career centers to be sustainable? Sense of ownership/political will • The mandates of the university presidents come to an end at the end of 2018. According to one VP, the changing of the president would not affect the continuity of the existing Career Center because the program is already well￾anchored. According to another VP, the position of the new president will be decisive for the continuity and scaling of the program. • Issue was also mentioned by Activity manager. Factors favoring sustainability • KII CC Director OFPPT Tangiers: The CC provides a better framework to integrate the WR and orientation modules, in continuity of the past 10 years OFPPT’s strategy • KII BD Tangiers “The sense of ownership of the institutions for the CC is increasing. We think the faculty are getting more involved.” The university made additional staff available to the CC (responsable d’orientation, chargé de formation et filières) Factors not favoring sustainability • KII institution: the integration of the CC has been done at the national level but has not yet been integrated into the institutional • The host institutions of the career centers and their relevant Ministry departments have taken ownership of the career centers and are committed to pursuing the effort, an important factor favoring sustainability. • Few institutional and financial management mechanisms have been tested to take over some key functions of the current model which rely on the program management team’s expertise and donor funding. • While USAID Career Center and its institutional partners are discussing many potential solutions to replace USAID funding and expertise, these have not been fully modeled or tested yet, which limits the ability of the project and its counterparts to evaluate. As such it is too early to evaluate the feasibility and compatibility of these proposed strategies. This is especially true of current plans to sustain the VCC post Activity. 8. Develop a business plan that estimates the cost of maintaining USAID Career Center services post USAID funding and assesses alternative income generation strategies. The business model should be developed in collaboration with institutional partners taking into consideration how similar activities are managed within the universities and vocational training institutions and how these institutions currently organize continuing education programs for private sector clients for a fee. (high priority, to be addressed in the short term) 9. Before continuing the development of the VCC, the post program management model should be determined. Future development of online services should be done in collaboration with the entities that will assume the management of the service. In this way, the VCC will develop according to the capacity of its future owner(s) to manage its evolution. (high priority, to be addressed in the short term) 10. Upon identifying an appropriate VCC management model, consider the following improvements: – Improving the design and functionality of the website and offering content in Arabic; Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 61 Findings Analysis and Conclusions Recommendations EQ3: How likely are the career centers to be sustainable? organigram; there is not yet an “operational” work plan for how to replicate the model • KII CC staff: Not sure that top leadership of institution has fully bought into model; not sure they understand what is required to implement model ($, competence, resources) • KII CC regional coordinator Need a champion(s) within private sector to promote the model. Challenge of Maintaining Quality HR • KII institution: OFPPT personnel are already fully paid by institution • University- only the directors were selected and are currently paid by the institution • KII CC IP: Having qualified CC personnel are the key to a successful CC. Unless good working conditions of current CC staff are ensured, high risk current personnel will leave. • CC staff are concerned about what working conditions will be like after the project are concerned that they will not have resources to cover important operational expenses (transport, activity costs) • The existing physical career centers are likely to be sustained by their host institutions in some form. • It is likewise likely that additional centers and/or sub offices will be opened based on counterparts’ current intentions, extending services to additional students/trainees. • Proposed changes in the institutional management structure of existing centers and strategies to scale the model are likely to significantly alter the USAID career center model. • There are significant challenges to maintaining the quality of career center services (sustaining current personnel in university centers, recruiting and training new counselors, funding activities post￾USAID, diversifying and diversifying training offerings) although the project has made progress addressing some of these issues. – Integrating online modules with university e-learning strategy and strategies to offer credits for soft skills courses. – Considering blended learning models that integrate learning on the VCC with Najahi training programs (which is being planned for Year Four); – Reinforcing social networking strategies to ensure all new content on the VCC is diffused via existing center social media platforms. (medium priority, to be addressed in the, medium term ) 12. Build the capacity of each university to sustain and scale the career center model. To ensure strong buy-in from university leadership, USAID Career Center should focus their sustainability strategy on building the capacity of each university to sustain and scale the career center model within the university structure. USAID Career Center should continue to collaborate with the State Secretariat for High Education to scale the model within other universities. In the case of the OFPPT and Ministry of Tourism, it should continue to work with the central office to develop a plan to replicate the model in additional sites. (high priority, to be addressed in the short term) Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 62 Findings Analysis and Conclusions Recommendations EQ3: How likely are the career centers to be sustainable? • KII Activity managers: the initial selection of personnel by universities did not align with the ideal profile of a center manager • KII university admin: there are administrative challenges to hire current staff – may need to go through an open recruitment process; possibility to contract personnel but the administrative mechanisms are not yet in place • The work of career counselor is not yet recognized as a specific category of human resource; not able to distinguish from the categories in terms of salary and benefits • KII regional staff Tangiers: there is a problem with the basic skills/profile of the CC staff members “I am not sure that even with training, it is possible to change the way directors relate”. This is why it is important to integrate the CC into professional association (as a member of the board of tourism HR association) or the students associations (BDE/alumni). How to Replace USAID Financial Resources • KII institutions, CC staff, private sector assoc.: See potential for CC to generate income through training/continuing education services targeting companies who want to offer soft skills training to personnel; both OFPPT and • USAID Career Center has established a good foundation for the sustainability and scalability of the Career Center model. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 63 Findings Analysis and Conclusions Recommendations EQ3: How likely are the career centers to be sustainable? university already generate income this way. CC IP staff: have not been able to experiment with income generation due to USAID rules • CC regional coordinator.: when global budgets cost to establish a CC, running costs of a CC were shared with institutional stakeholders – first reaction was that it would not be possible to allocate the same level of resources • KII regional staff Tangiers: use the VCC to generate revenues through publicity banners and the installation of terminals where the youth can save their CV after the CC treatment and validation in specific areas (Zone Franche à Tanger). Private companies can pay membership/subscription fees Future of Soft Skills Training • Inst Admin & CC team: the trainers will be recruited among OFPPT trainers and University PhD. students and faculty. • At the moment, it is not clear who/which entity will ensure follow-up and quality assurance with trainees. • KII Activity Manager: EFE trainers have significant experience as coaches “the majority Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 64 Findings Analysis and Conclusions Recommendations EQ3: How likely are the career centers to be sustainable? trainers from OFPPT …. don’t have the right profile.” • KII Activity Manager: EFE is based in Morocco, will exist after the end of the program and will be available to support our institutional partners. It is an opportunity for EFE. • KII CC staff: Soft skills training is done outside the center; the CC staff currently coordinate with the trainers to arrange the training. If/when the training is institutionalized, will the CC have a role, or will it be separate? Coherence of overall sustainability strategy • KII private sector association: “once USAID stops supporting the project, everything will stop”. Universities most challenging because of their autonomy –need to convince each institution; up to now, universities are having difficulties respecting their initial commitments – what will be the case when all USAID support ends? OFPPT may be better positioned because it is centralized and has experience working with the private sector. • KII institution: pilot projects are historically difficult to sustain. We have had many good Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 65 Findings Analysis and Conclusions Recommendations EQ3: How likely are the career centers to be sustainable? pilot programs that are not sustained once donor funding ends • KII institution: incoherence – ANAPEC agency and CC – two centers in same space – from the perspective of a student – what is the difference? One is supported by a budget already allocated by the State while the other is not yet anchored in an institution. • KII BD Tangiers: About the institutional issue “They need to work out a mechanism so that CC can have funding to organize activities. There is demand for CC services from the private sector, but it seems complex to ensure that the money that is generated through CC comes back to the CC.” (une gestion déléguée? ONG ou autre institution) Progress Scaling the model • KII institution, CC director, CC IP staff: UHIICC in the process of creating sub offices to be staffed by one university employee (administrator) and youth ambassadors • KII institution, CC IP staff: OFPPT has decided to create 3 more centers; final selection of sites in progress – mostly likely 1 in Settat, 1 in Al Hoceima, Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 66 Findings Analysis and Conclusions Recommendations EQ3: How likely are the career centers to be sustainable? • KII institution, CC IP staff: OFPPT is adapting soft skills curriculum (My Najahi) for integration into the training program of all its centers (300+ across Morocco). 60 course hours. Following same model as entrepreneurship training (also 60 course hours). Will be integrated officially in program starting September 2018. OFPPT has already trained master trainers; master trainers are training other trainers. • KII institution (Casablanca); trained doctoral students as soft skills trainers • Organized meeting at national level with other institutions involved in training to present CC model • In process of developing a toolkit “how to create a CC” • In process of negotiating a partnership with Univ. Laval QC to create a Masters degree “Career Orientation” • KII CC Activity managers: Project has ideas but has not yet determined how shared resources and services– VCC, MIS, tool updates, training of personnel will be managed after the end of the program. As a result, no Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 67 Findings Analysis and Conclusions Recommendations EQ3: How likely are the career centers to be sustainable? capacity building of the “entity” has been carried out at this stage. • KII institution: imagined that current CC staff will train staff in new centers • KII regional staff Tangiers: About the VCC management and sustainability “In the future, there is a risk that the content will not be enriched. I am thinking about a “gestion deleguée” for the centers – for both the centers and the VCC so that the CC are managed using a model that is closer to an association or enterprise.” • KII BD Tangiers: Training is not enough there is a need to follow up. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 68 ANNEX 3: YOUTH SURVEY RESULTS All Respondents Male Respondents Female Respondents University Respondents Vocational Institution Respondents Number % of Total Number % of Total Number % of Total Number % of Total Number % of Total Respondent Demographics n = 218 110 108 149 69 Sex Male 110 50.5% 110 50.5% - - 75 50.3% 35 50.7% Female 108 49.5% - - 108 49.5% 74 49.7% 34 49.3% Age Range 17 - 46 - 18 – 40 - 17 - 46 - 17 – 46 - 18 - 29 - Average 22 - 23 - 21 - 23 - 21 - 15 - 19 50 22.9% 16 14.5% 34 31.5% 27 18.1% 23 33.3% 20 - 24 126 57.8% 66 60.0% 60 55.6% 89 59.7% 37 53.6% 25 - 29 29 13.3% 18 16.4% 11 10.2% 20 13.4% 9 13.0% 30 and above 13 6.0% 10 9.1% 3 2.8% 13 8.7% 0 0.0% Residence Casablanca 85 39.0% 38 34.5% 47 43.5% 59 39.6% 26 37.7% Marrakech 43 19.7% 29 26.4% 14 13.0% 37 24.8% 6 8.7% Tanger 58 26.6% 30 27.3% 28 25.9% 27 18.1% 31 44.9% Other 32 14.7% 13 11.8% 19 17.6% 26 17.4% 6 8.7% Occupation Public University 135 61.9% 68 61.8% 67 62.0% 117 78.5% 18 26.1% Private University 5 2.3% 2 1.8% 3 2.8% 3 2.0% 2 2.9% Public Vocational Training 48 22.0% 21 19.1% 27 25.0% 6 4.0% 42 60.9% Private Vocational Training 2 0.9% 2 1.8% 0 0% 1 0.7% 1 1.4% High School Student 9 4.1% 5 4.5% 4 3.7% 7 4.7% 2 2.9% Middle School Student 3 1.4% 2 1.8% 1 0.9% 1 0.7% 2 2.9% Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 69 All Respondents Male Respondents Female Respondents University Respondents Vocational Institution Respondents Number % of Total Number % of Total Number % of Total Number % of Total Number % of Total FT Employee 10 4.6% 7 6.4% 3 2.8% 10 6.7% 0 0.0% PT Employee 4 1.8% 3 2.7% 1 0.9% 2 1.3% 2 2.9% Job Seeker 25 11.5% 17 15.5% 8 7.4% 17 11.4% 8 11.6% Educational Institute Universite HASSAN II 60 27.5% 28 25.5% 32 29.6% 60 40.3% - - ISTA 36 16.5% 19 17.3% 17 15.7% - - 36 52.2% Universite Cadi Ayyad de Marrakech 52 23.9% 27 24.5% 25 23.1% 52 34.9% - - ISTAHT 7 3.2% 5 4.5% 2 1.9% - - 7 10.1% Universite Abdelmalek Essaadi 37 17.0% 20 18.2% 17 15.7% 37 24.8% - - ISMONTIC 26 11.9% 11 10.0% 15 13.9% - - 26 37.7% I participated in the following Career Center Services or activities (check as many as apply): One-On-One Counselling 38 17.4% 21 19.1% 17 15.7% 23 15.4% 15 21.7% Soft Skills Training 32 14.7% 12 10.9% 20 18.5% 18 12.1% 14 20.3% Workshops 54 24.8% 30 27.3% 24 22.2% 39 26.2% 15 21.7% Job Fair 22 10.1% 11 10.0% 11 10.2% 15 10.1% 7 10.1% Internship 14 6.4% 9 8.2% 5 4.6% 6 4.0% 8 11.6% Job Interview 23 10.6% 12 10.9% 11 10.2% 9 6.0% 14 20.3% Youth Ambassador Program 13 6.0% 4 3.6% 9 8.3% 7 4.7% 6 8.7% Summer Camp 13 6.0% 5 4.5% 8 7.4% 7 4.7% 6 8.7% Online Services 30 13.8% 14 12.7% 16 14.8% 20 13.4% 10 14.5% None of the Above 14 6.4% 5 4.5% 9 8.3% 10 6.7% 4 5.8% Other 15 6.9% 7 6.4% 8 7.4% 12 8.1% 3 4.3% Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 70 All Respondents Male Respondents Female Respondents University Respondents Vocational Institution Respondents Number % of Total Number % of Total Number % of Total Number % of Total Number % of Total Utilization of Career Centers n = 135 61.9% 73 66.4% 62 56.9% 91 61.1% 44 63.8% How long has it been since you participated in your first Career Center service or activity? More than one year 34 25.2% 23 31.5% 11 17.7% 22 24.2% 12 27.3% 1 - 12 months 61 45.2% 30 41.1% 31 50.0% 41 45.1% 20 45.5% Less than one month 40 29.6% 20 27.4% 20 32.3% 28 30.8% 12 27.3% How long has it been since you last participated in a Career Center service or activity? More than one year 22 16.3% 18 24.7% 4 6.5% 16 17.6% 6 16.3% 1 - 12 months 62 45.9% 30 41.1% 32 51.6% 41 45.1% 21 47.7% Less than one month 51 37.8% 25 34.2% 26 41.9% 34 37.4% 17 38.6% Based on your experience with Career Center services, to what extent do you agree with the following statement: “The career center staff was attentive to my needs.” Strongly Disagree 3 2.2% 1 1.4% 2 3.2% 1 1.1% 2 4.5% Disagree 10 7.4% 6 8.2% 4 6.5% 7 7.7% 3 6.8% Neutral 17 12.6% 10 13.7% 7 11.3% 15 16.5% 2 4.5% Agree 34 25.2% 18 24.7% 16 25.8% 22 24.2% 12 27.3% Strongly Agree 60 44.4% 33 45.2% 27 43.5% 37 40.7% 23 52.3% Do not know 11 8.1% 5 6.8% 6 9.7% 9 9.9% 2 4.5% Based on your experience with Career Center services, to what extent do you agree with the following statement: “The Career Center space was welcoming for youth.” Strongly Disagree 4 3.0% 2 2.7% 2 3.2% 1 1.1% 3 6.8% Disagree 6 4.4% 6 8.2% 0 0% 5 5.5% 1 2.3% Neutral 12 8.9% 6 8.2% 6 9.7% 9 9.9% 3 6.8% Agree 39 28.9% 21 28.8% 18 29.0% 29 31.9% 10 22.7% Strongly Agree 68 50.4% 33 45.2% 35 56.5% 42 46.2% 26 59.1% Do not know 6 4.4% 5 6.8% 1 1.6% 5 5.5% 1 2.3% Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 71 All Respondents Male Respondents Female Respondents University Respondents Vocational Institution Respondents Number % of Total Number % of Total Number % of Total Number % of Total Number % of Total Based on your experience with Career Center services, to what extent do you agree with the following statement: “The Career Center has useful tools and content.” Strongly Disagree 3 2.2% 2 2.7% 1 1.6% 2 2.2% 1 2.3% Disagree 8 5.9% 5 6.8% 3 4.8% 5 5.5% 3 6.8% Neutral 16 11.9% 8 11.0% 8 12.9% 14 15.4% 2 4.5% Agree 44 32.6% 22 30.1% 22 35.5% 32 35.2% 12 27.3% Strongly Agree 54 40.0% 28 38.4% 26 41.9% 30 33.0% 24 54.5% Do not know 10 7.4% 8 11.0% 2 3.2% 8 8.8% 2 4.5% Based on your experience with Career Center services, to what extent do you agree with the following statement: “I have a better understanding of what employers are looking for in their employees.” Strongly Disagree 3 2.2% 2 2.7% 1 1.6% 1 1.1% 2 4.5% Disagree 8 5.9% 5 6.8% 3 4.8% 7 7.7% 1 2.3% Neutral 20 14.8% 12 16.4% 8 12.9% 13 14.3% 7 15.9% Agree 48 35.6% 28 38.4% 20 32.3% 37 40.7% 11 25.0% Strongly Agree 40 29.6% 19 26.0% 21 33.9% 22 24.2% 18 40.9% Do not know 16 11.9% 7 9.6% 9 14.5% 11 12.1% 5 11.4% Based on your experience with Career Center services, to what extent do you agree with the following statement: “I know more about what kinds of jobs and careers suit my personality and skills.” Strongly Disagree 3 2.2% 2 2.7% 1 1.6% 2 2.2% 1 2.3% Disagree 5 3.7% 3 4.1% 2 3.2% 3 3.3% 2 4.5% Neutral 23 17.0% 13 17.8% 10 16.1% 18 19.8% 5 11.4% Agree 55 40.7% 30 41.1% 25 40.3% 35 38.5% 20 45.5% Strongly Agree 38 28.1% 16 21.9% 22 35.5% 23 25.3% 15 34.1% Do not know 11 8.1% 9 12.3% 2 3.2% 10 11.0% 1 2.3% Based on your experience with Career Center services, to what extent do you agree with the following statement: “I know more about what sectors may be offering jobs in the future that may interest me.” Strongly Disagree 6 4.4% 5 6.8% 1 1.6% 4 4.4% 2 4.5% Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 72 All Respondents Male Respondents Female Respondents University Respondents Vocational Institution Respondents Number % of Total Number % of Total Number % of Total Number % of Total Number % of Total Disagree 7 5.2% 4 5.5% 3 4.8% 5 5.5% 2 4.5% Neutral 24 17.8% 13 17.8% 11 17.7% 19 20.9% 5 11.4% Agree 54 40.0% 29 39.7% 25 40.3% 35 38.5% 19 43.2% Strongly Agree 33 24.4% 19 26.0% 14 22.6% 20 22.0% 13 29.5% Do not know 11 8.1% 3 4.1% 8 12.9% 8 8.8% 3 6.8% Based on your experience with Career Center services, to what extent do you agree with the following statement: “I feel that I am ready to look for a job.” Strongly Disagree 6 4.4% 5 6.8% 1 1.6% 3 3.3% 3 6.8% Disagree 14 10.4% 8 11.0% 6 9.7% 9 9.9% 5 11.4% Neutral 22 16.3% 11 15.1% 11 17.7% 17 18.7% 5 11.4% Agree 44 32.6% 26 35.6% 18 29.0% 31 34.1% 13 29.5% Strongly Agree 38 28.1% 20 27.4% 18 29.0% 24 26.4% 14 31.8% Do not know 11 8.1% 3 4.1% 8 12.9% 7 7.7% 4 9.1% Based on your experience with Career Center services, to what extent do you agree with the following statement: “Through the Career Center, I have learned about the importance of developing skills such as controlling my emotions, self-esteem, communicating clearly and thinking critically” Strongly Disagree 9 6.7% 6 8.2% 3 4.8% 6 6.6% 3 6.8% Disagree 6 4.4% 4 5.5% 2 3.2% 5 5.5% 1 2.3% Neutral 20 14.8% 11 15.1% 9 14.5% 16 17.6% 4 9.1% Agree 47 34.8% 27 37.0% 20 32.3% 32 35.2% 15 34.1% Strongly Agree 41 30.4% 20 27.4% 21 33.9% 21 23.1% 20 45.5% Do not know 12 8.9% 5 6.8% 7 11.3% 11 12.1% 1 2.3% Based on your experience with Career Center services, to what extent do you agree with the following statement: “I have acquired new skills through the Career Center such as controlling my emptions, self-esteem, communicating clearly and thinking critically.” Strongly Disagree 8 5.9% 5 6.8% 3 4.8% 6 6.6% 2 4.5% Disagree 6 4.4% 5 6.8% 1 1.6% 4 4.4% 2 4.5% Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 73 All Respondents Male Respondents Female Respondents University Respondents Vocational Institution Respondents Number % of Total Number % of Total Number % of Total Number % of Total Number % of Total Neutral 30 22.2% 16 21.9% 14 22.6% 21 23.1% 9 20.5% Agree 39 28.9% 23 31.5% 16 25.8% 29 31.9% 10 22.7% Strongly Agree 39 28.9% 20 27.4% 19 30.6% 21 23.1% 18 40.9% Do not know 13 9.6% 4 5.5% 9 14.5% 10 11.0% 3 6.8% Based on your experience with Career Center services, to what extent do you agree with the following statement: “The virtual career center is easy to use and was responsive to youth needs.” Strongly Disagree 3 2.2% 1 1.4% 2 3.2% 2 2.2% 1 2.3% Disagree 7 5.2% 4 5.5% 3 4.8% 6 6.6% 1 2.3% Neutral 17 12.6% 10 13.7% 7 11.3% 12 13.2% 5 11.4% Agree 50 37.0% 32 43.8% 18 29.0% 34 37.4% 16 36.4% Strongly Agree 47 34.8% 21 28.8% 26 41.9% 29 31.9% 18 40.9% Do not know 11 8.1% 5 6.8% 6 9.7% 8 8.8% 3 6.8% Based on your experience with Career Center services, to what extent do you agree with the following statement: “The virtual career center has useful tools and content.” Strongly Disagree 2 1.5% 1 1.4% 1 1.6% 1 1.1% 1 2.3% Disagree 9 6.7% 6 8.2% 3 4.8% 5 5.5% 4 9.1% Neutral 14 10.4% 9 12.3% 5 8.1% 12 13.2% 2 4.5% Agree 47 34.8% 27 37.0% 20 32.3% 32 35.2% 15 34.1% Strongly Agree 47 34.8% 24 32.9% 23 37.1% 30 33.0% 17 38.6% Do not know 16 11.9% 6 8.2% 10 16.1% 11 12.1% 5 11.4% Based on your experience with Career Center services, to what extent do you agree with the following statement: “If I can, I will participate in more Career Center activities and services.” Strongly Disagree 5 3.7% 4 5.5% 1 1.6% 2 2.2% 3 6.8% Disagree 5 3.7% 3 4.1% 2 3.2% 5 5.5% 0 0.0% Neutral 11 8.1% 6 8.2% 5 8.1% 7 7.7% 4 9.1% Agree 37 27.4% 22 30.1% 15 24.2% 25 27.5% 12 27.3% Strongly Agree 71 52.6% 35 47.9% 36 58.1% 48 52.7% 23 52.3% Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 74 All Respondents Male Respondents Female Respondents University Respondents Vocational Institution Respondents Number % of Total Number % of Total Number % of Total Number % of Total Number % of Total Do not know 6 4.4% 3 4.1% 3 4.8% 4 4.4% 2 4.5% Based on your experience with the Career Center, is the following statement true or false: “The Career Center helped me to access an internship.” True 65 48.1% 35 47.9% 30 48.4% 36 39.6% 29 65.9% False 70 51.9% 38 52.1% 32 51.6% 55 60.4% 15 34.1% Based on your experience with the Career Center, is the following statement true or false: “The Career Center helped me to get a job.” True 50 37.0% 26 35.6% 24 38.7% 24 26.4% 26 59.1% False 85 63.0% 47 64.4% 38 61.3% 67 73.6% 18 40.9% Based on your experience with the Career Center, is the following statement true or false: “The Career Center helped me to network with potential employers.” True 63 46.7% 31 42.5% 32 51.6% 37 40.7% 26 59.1% False 72 53.3% 42 57.5% 30 48.4% 54 59.3% 18 40.9% Based on your experience with the Career Center, is the following statement true or false: “The Career Center connected me to associations or agencies that can help with my ‘employability’.” True 61 45.2% 34 46.6% 27 43.5% 32 35.2% 29 65.9% False 74 54.8% 39 53.4% 35 56.5% 59 64.8% 15 34.1% Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 75 ANNEX 4: LIST OF PARTICIPANTS Annex 4 has been removed to ensure the protection of human subjects who participated in the mid-term performance evaluation of the USAID Career Center. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 76 ANNEX 5: DESK STUDY BIBLIOGRAPHY • Alison Doyle, Top 10 In-Demand Customer Service Soft Skills (Updated 22 March 2018) • Career Center Workshop – Know Yourself • Career Center Workshop – The World of Work • Career Center Workshop – My Professional Project (Personal/Professional Development Planning) • Career Center Workshop – Improve my CV • Career Center Workshop – My Letter of Introduction • Career Center Workshop – How to Introduce Myself in 60 seconds • Career Center Workshop – Preparing for a Job Interview • Career Center Workshop – Improve my Communication Skills • Career Center Workshop – Work Life Balance • Career Center Workshop – Time Management and Organizational Skills • Career Center Workshop – Team Working • Career Center Workshop – Public Speaking • Christopher Jones II, A Model for Career Centers to Support Student Career Development: A Delivery Tool for Modern Career Centers (Fall 2012) • Database list of beneficiaries – Marrakech UCA Summer Camp 2017 • Database list of beneficiaries – Sampling 5000 • Initial Monitoring, Evaluation and Learning Plan MELP (October 2016) • International Youth Foundation (IYF), Strengthening life skills for youth: A practical guide to quality programming (2014) • Mariya Pylayev, 10 Skills Everyone Needs To Thrive In Today's Job Market (10 April 2014) • MELP USAID Note – Proposed Revision of Program Indicators and Targets (March 2018) • Michelle Benjamin, 10 Soft Skills Manufacturers Need from Employees (02 October 2014) • Monster International, HospitalitySchools.com, Top 10 Qualities of a Great Hospitality Employee • Morocco Career Center Activity Final and Fully Executed Award • MOU USAID-OFPPT • Najahi Prêt pour l’emploi – EFE Maroc Trainers by Region • Najahi Prêt pour l’emploi – Modules A, B, C, D, E, F, G • Najahi Prêt pour l’emploi – Najahi Catalogue • Najahi Prêt pour l’emploi – Training of Trainers ToT List of Participants • Najahi Prêt pour l’emploi – Training of Trainers ToT OFPPT, ToT Master Trainers for 13 Modules (September 2017) • Najahi Prêt pour l’emploi – Training of Trainers ToT OFPPT, ToT-Basic OFPPT (14 – 16 February 2018) • Najahi Prêt pour l’emploi – Training of Trainers ToT OFPPT, ToT-Supplementary (20 – 23 February 2018) • Najahi Prêt pour l’emploi – Training of Trainers ToT Universities, ToT Casablanca FSJES (October 2017) • Najahi Prêt pour l’emploi – Training of Trainers ToT Universities, ToT FST-ENSA (July 2017) • Najahi Prêt pour l’emploi – Training of Trainers ToT Universities, ToT UCA (26 – 28 February 2018) • Najahi Prêt pour l’emploi – Training of Trainers ToT Universities, ToT UCA (October – November 2017) • Najahi Prêt pour l’emploi – Training of Trainers ToT Universities, ToTs UHIIC (27 – 28 December 2017; 17 – 18 January 2018; and 1 – 2 February 2018) • Newsletters to Stakeholders No. 1, 2, 3 and 4 Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 77 • Oregon State University, College of Agricultural Sciences Academic Programs 2, The Value of Soft Skills (2018) • Partnerships table (31 December 2017) • Partnerships table (28 February 2018) • Partnerships selected with contact details, 2 documents • Program Organigram, revised (13 March 2018) • Rapid Industry Analysis Methodology • Revised Monitoring, Evaluation and Learning Plan MELP (March 2018) • Series of 12 industries analyses (6 in French and 6 in English) • Series of Whole Systems in the Room reports (4 reports) • Draft Social Network Analysis of Employment Stakeholders, Tangier Marrakech and Casablanca, (8 March 2018) • Summary of USAID Career Center Staff Training Curriculum • 30 Success Stories and Highlights • Survey – feedback enterprise • Survey – sourcing follow-up • Survey – management tool data (April – June 2017) • Survey – management tool data (July – September 2017) • Survey – evaluation template • USAID Career Center Donor Mapping Report • USAID Career Center Soft Skills Review • USAID Career Center Gender Analysis • USAID Career Center Workplan Year 1 • USAID Career Center Workplan Year 2 • USAID Career Center Workplan Year 3 • USAID Career Center Year 1, Annual Report • USAID Career Center Year 1, Q1 Quarterly Report (April – June 2015) • USAID Career Center Year 1, Q2 Quarterly Report (July – September 2015) • USAID Career Center Year 1, Q3 Quarterly Report (Oct – December 2015) • USAID Career Center Year 2, Annual Report • USAID Career Center Year 2, Q1 Quarterly Report (April – June 2016) • USAID Career Center Year 2, Q2 Quarterly Report (July – September 2016) • USAID Career Center Year 3, Q2 Quarterly Report (Jan – March 2017) • USAID Career Center Year 3, Q3 Quarterly Report (April – June 2017) • USAID Career Center, Q3 Quarterly Report (Oct – Dec 2017) • USAID Career Center, National Steering Committee Report, Rabat (24 March 2016) • USAID Career Center, National Steering Committee Report, Rabat (30 November 2016) • USAID Career Center, National Steering Committee Report, Rabat (12 October 2017) • Youth Ambassadors Outreach, 2 documents Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 78 ANNEX 6: PROJECT PROGRESS AGAINST MELP INDICATORS The data included below, comprising Annex 6, is a compilation of performance data gathered from the USAID Career Center’s reports including the MELP Revised (March 2018), Quarterly Report (Oct – Dec 2017), and Annual Report, Year 2. Performance data reflects the following reporting periods: • MELP Year 1 (April 2015 – March 2016) - Year 1 Results against Year 1 Target • MELP Year 2 (April 2016 – March 2017) - Year 2 Results against Year 2 Target • MELP Year 3 (April 2017 – December 2017) 23 - Year 3 Results to date against Year 3 Target INDICATOR 1 Number of local workforce development (WFD) programs using labor market information to improve the quality and relevancy of offerings DESCRIPTION Number of courses, workforce initiatives, career services using the labor market information UNIT OF MEASURE Number of workforce initiatives ● As Career Centers were due to open in year 2, progress against targets was expected to be reported in year 2. Thus for year 1, the program reports 0 workforce initiatives against target 0. ● For year 2, the result of 26 workforce initiatives exceeded almost three-fold target 10. Results include the opening of the University and Vocational Training Institute Career Centers in Casablanca and the Vocational Career Center in Tangier and 14 supplemental work readiness modules. ● As of the third quarter of Year 3, the USAID Career Center has administered seven workforce initiatives. While this is below the Year 3 target of 10, the USAID Career Center is on track towards 23 Year 3 performance data covers April 2017 through December 2017, to coincide with the period under evaluation. The evaluation team did not analyze nor report on performance data during the final quarter of Year 3, January – March 2018. In light of this, the evaluation team has indicated that performance during Year 3, reflects only a portion of the year’s performance against the annual targets. 26 7 0 5 10 15 20 25 30 Year 1 Year 2 Year 3 (dec 2017) target result Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 79 achieving the target. There were 3 workforce initiatives during the 1st quarter. In Casablanca, after sharing labor market information about private IT business CGI, two groups of youth received training on two modules with the Career Center before integrating the company. In Tangier, labor market information was used during the job fair organized by the Free Trade Association (May 18-19). In Marrakech, labor market information was used during the Job fair organized from Marrakech Generation (Apr 20-21). In addition, there were 4 workforce initiatives during 4th quarter. In Nov 2017, the program’s labor market assessments were shared with two doctoral students conducting research on the labor market. And in Dec 2017, the Najahi soft skills trainings, which were developed based on labor market information, were integrated into the National School of Applied Sciences (ENSA) and the School of Sciences and Technology (FST) in the University in Tangier. INDICATOR 2 Percentage cost of CCs supported by universities, vocational training centers and companies DESCRIPTION The cost of CCs that will be supported by universities, vocational training centers and companies is defined as follow: • Operating costs include the local, electricity, Internet and phone expenses. • Services cost includes training and expertise costs. • Human resource to ensure the functioning of the career center and to supervise it. The university must take in charge 50% of the costs mentioned above, at the end of the project duration. UNIT OF MEASURE % Cost of CCs supported by other institutions ● Annual results against targets were reported as of the third quarter of Year 3. During Year 1 and 2 the targets and performance were zero. ● As of the third quarter of Year 3, the USAID Career Center achieved a result of 62.83% cost of career centers supported by other institutions was more than double the target of 30%. ● The Ministry of Tourism pledged to provide staff for the Vocational Career Center in Marrakech. 62.83% 0% 10% 20% 30% 40% 50% 60% 70% Year 1 Year 2 Year 3 (dec 2017) target result Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 80 ● All host-institutions continued to contribute to operational costs of the career centers and fund the salaries of Career Center Directors with the addition of a Deputy Director positioned at the Marrakech University. Additionally, OFPPT and the Ministry of Tourism also continued to fund the salaries for the Vocational Center staff. INDICATOR 3 Density of connection links in the labor market network DESCRIPTION Measure 3 types of connections: • Institutional relations: when an institutional framework exist between actors (conventions, memoranda of understanding, financial transactions, etc.) • Spontaneous or informal relations: this can be measured through email exchange, phone calls, etc. • Perception: estimates of trust, professionalism, efficiency UNIT OF MEASURE Density of connection links ● Annual results against targets were reported in year 3. During Year 1 and 2 the targets and performance were zero. • In alignment with the MELP Year 3, the density of connection links in the labor market network had a target of 0.059 and a result of 0.070 at the end of the third quarter of Year 3. INDICATOR 4 Number of private organizations partnering with educational institutions as a result of USG assistance DESCRIPTION We will count the number of private organizations partnering with universities and VTCs. Partnerships will be measured as follow: • -Formal partnership such as conventions and agreements, which may have research and insertion as the object of these agreements. • -Informal partnership such as the participation of the private sector in the organization of events. Both private businesses and educational institutions can also undertake activities together such as integrating young people into the workplace. 0.07 0 0.02 0.04 0.06 0.08 Year 1 Year 2 Year 3 (dec 2017) target result Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 81 UNIT OF MEASURE Number of private organizations • As Career Centers were due to open in year 2, results against targets were expected to be reported in year 2. (In Year 1, target N/A, result 0). • In Year 2, the result of 20 private businesses partnerships exceeded the target of 15. Partnerships were developed through the co-organization of events and activities, with organizations. • The USAID Career Center recorded a total of 89 private businesses partnerships as of the end of the third quarter of Year 3, far exceeding (more than double) the target of 40. During the last quarter, 42 new partnerships were developed. In October 2017, there were 5 partnerships in Casablanca and 2 partnerships in Tangier. In November 2017, there were 4 partnerships in Casablanca and 1 in Tangier. Finally, in December 2017, there were 21 partnerships in Casablanca, 5 in Marrakech and 3 in Tangier. INDICATOR 5 % of organizations partnering with educational institutions two or more times as a result of USG assistance DESCRIPTION % of private organizations partnering more than one time with universities and Vocational Career Centers. UNIT OF MEASURE Partnerships are measured as follow: • Formal partnership such as conventions and agreements, which may have research and insertion as the object of these agreements. • Informal partnership such as the participation of private organizations in the organization of events. Both can also undertake activities together such as integrating Young people into the workplace 20 89 0 20 40 60 80 100 Year 1 Year 2 Year 3 (dec 2017) target result Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 82 • Results against targets are reported annually. As Career Centers were due to open in year 2, results against targets were expected to be reported in year 2. (In MELP year 1, target 0, result N/A). • In Year 2, the result of 41% of organizations partnering with educational institutions was lower than the target of 50%. • In Year 3, the result of 50% of organizations partnering with educational institutions, as of the end of the third quarter of Year 3, exceeded the target of 40%. 50% (30/60) of private organizations are partnering more than one time with the Career Center. INDICATOR 6 Number of educational institutions who adopted the USG-funded career center model DESCRIPTION Number of universities who adopted the USG-funded career center model Number of vocational training centers who adopted the USG-funded career center model Note: no further definition UNIT OF MEASURE Number of educational institutions • As Career Centers were due to open in year 2, results against targets were expected to be reported in year 2 (In Year 1, target 0, result 0). • In year 2, the program reports that 5 educational institutions adopted the USG-funded career center model. This result is on par with the target of 5. 41% 50% 0% 10% 20% 30% 40% 50% 60% Year 1 Year 2 Year 3 (dec 2017) target result 5 5 0 1 2 3 4 5 6 Year 1 Year 2 Year 3 (dec 2017) target result Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 83 • In year 3, the program reported a total of 5 educational institutions, as of the end of the third quarter of Year 3, who had adopted the USG-funded career center model. This result to date has far exceeded the target of 1. INDICATOR 7 Number of workforce development initiatives completed as a result of USG participation in public-private partnerships DESCRIPTION Workforce development initiatives are cooperative investments in the human resources of a country or area through training or retraining for available or anticipated near-term workforce skills requirements. The objective is to endow workers with the skills needed to productively perform the tasks required for new formal sector jobs. Initiatives may include events, guest speakers and job fairs. UNIT OF MEASURE Number of workforce development initiatives • In year 2, the program result of 32 workforce development initiatives exceeded the target of 21. Training pilots have taken place in the three regions. Initiatives include: • Job Fair with Aksal Group in Casablanca and in Tangier; Job panel in Tangier; 14 developed supplemental Soft Skills Modules for “My 1st Step into the Professional Environment;” One More Step Towards Success • In year 3, the program has observed a total of 47 workforce development initiatives, as of the third quarter of Year 3. While slightly below the annual target of 50, the USAID Career Center is on track to achieve this target. During the last quarter, 12 workforce development initiatives have been completed. In October 2017, there were 3 Job Fair in Tangier and Casablanca and 2 Guest speakers in Casablanca. In November 2017, there were 2 Guest speakers in Casablanca. And in December 2017, there were 1 Engineers Job Fair in Casablanca and 4 Guest speakers in Marrakech. INDICATOR 8 Number of trainees satisfied with the quality and relevance of the training and services received through USG-supported career services. Sub indicator: Proportion of female trainees satisfied with the quality and relevance of the training and services received through USG-supported career services.” DESCRIPTION Number of persons satisfied with the training and services received by the Career Center 32 47 0 10 20 30 40 50 60 Year 1 Year 2 Year 3 (dec 2017) target result Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 84 UNIT OF MEASURE Number of trainees satisfied with the quality of the training • As Career Centers were due to open in year 2, progress against targets were expected to be reported in year 2 (In year 1: Target 50%, results not available) • In year 2, the result of 98.67% number of trainees satisfied with the quality of the training exceeded the target of 80%. Satisfaction rate is very high for all the service offerings of the Career Centers. However, the Career Center staff had not yet developed habit of conducting satisfaction surveys automatically after workshops and interviews. • As of the end of quarter 3, the USAID Career Center has met the percentage of trainees satisfied with the quality of training, with a result to date of 80.21% against a Year 3 target of 80%. It is important to note that the program started to conduct electronic surveys in May 2017 (The program used Survey Monkey to send to participants however, the response rate was less than 10%. As such, the program developed an IT solution in which each training participant is automatically sent a post-training link via email. Upon completion of the survey, participants receive a certificate of completion. This functionality was launched in April). INDICATOR 9: Number of persons receiving quality career services as result of USG assistance. Sub indicators differentiate between registered, unregistered, and proportion of female participants (3 sub-indicators). Note: quality has not been defined. DESCRIPTION This indicator is used to count the number of individuals who utilize physically and virtually career services. This includes registered, unregistered and proportion of female participants. UNIT OF MEASURE Number of individuals receiving quality career services. This includes registered, unregistered and proportion of female participants. 98.67% 80.21% 0% 20% 40% 60% 80% 100% Year 1 Year 2 Year 3 (dec 2017) target result Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 85 • As Career Centers were due to open in year 2, progress against targets were expected to be reported in year 2 (In year 1, target 2000; result 0). • In year 2, the result of 30,764 individuals receiving career services far exceeded the target of 18,000. • At the conclusion of the third quarter of year 3, the result of 53,218 individuals receiving career services far exceeded the target of 38,000. This includes both registered and unregistered beneficiaries. INDICATOR 10: % of individual registered beneficiaries obtaining two or more quality career services as a result of USG assistance. Sub indicators differentiate between % of individual registered beneficiaries obtaining two, three, four, five or more quality career services as a result of USG assistance (4 sub indicators). Note: quality has not been defined. DESCRIPTION This indicator measures the return rate of youth, i.e. youth who have obtained two or more Career Center Services. UNIT OF MEASURE % of individual registered beneficiaries • This indicator was added in May 2017. The program started to report on it in year 3. • At the end of the third quarter of Year 3, the result of 28.09% individuals registered beneficiaries slightly exceeded the target of 25%. 30764 53218 0 10,000 20,000 30,000 40,000 50,000 60,000 Year 1 Year 2 Year 3 (dec 2017) target result 28.09% 0% 10% 20% 30% 40% 50% Year 1 Year 2 Year 3 (dec 2017) target result Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 86 • Indeed 28.09% (5,668 out of 20,178) of registered beneficiaries are coming back to the physical or Virtual Career Centers in order to obtain more services. The return rate of young women is higher than young men. INDICATOR 11: Number of persons completing USG-supported work readiness training programs DESCRIPTION This indicator is used to count the number of individuals who benefit from work readiness training programs physically and virtually. UNIT OF MEASURE Number of individuals who completed work readiness training programs • As Career Centers were due to open in year 2, progress against targets were expected to be reported in year 2 (In year 1, target 500, result 0) • In year 2, the result of 6,044 individuals who completed work readiness training programs exceeded the target of 4,500. The achieved result includes training with EFE and training in the Career Centers. • The rate of women slightly exceeds that of men (51% women and 49% men) • At the conclusion of the third quarter of Year 3, the result of 11,215 individuals who completed work readiness training programs was lower than the target of 13,000. This counts the number of individuals who benefitted from work readiness training programs physically and virtually. INDICATOR 12: % youth who participate in gender-related training who express a change in their perception of gender in the workplace following training DESCRIPTION Percentage of youth who participate in gender-related training who express a change in their perception of gender in the workplace following the training. This includes both male and female participants. UNIT OF MEASURE Percentage of youth 6044 11215 0 2,000 4,000 6,000 8,000 10,000 12,000 14,000 Year 1 Year 2 Year 3 (dec 2017) target result Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 87 • This indicator was added in May 2017. The program organized several gender related events before this date, but it could not introduce a satisfaction survey in order to inform this indicator. • After this date, a satisfaction survey was developed and used for one event which was organized on July 13-14th 2017. 16 youth participated in the event. 100% reported an improvement in their awareness of gender issues in the workplace. 50% were women. • At the conclusion of the third quarter of Year 3, the program reported a result of 100% youth and doubled the target of 50%. • Note: Due to the very small sample, this indicator is not being effectively measured. INDICATOR 13: Number of institutions who integrated soft skills in the training courses curriculum DESCRIPTION Count the number of institutions who integrated soft skills courses in their curricula UNIT OF MEASURE Number of institutions that integrated soft skills • As Career Centers were due to open in year 2, progress against targets were expected to be reported in year 2. Thus, for year 1, the program reported a result of 0 educational institutions that integrated soft skills against a target of 0. • In year 2, the program was still doing outreach work with universities. Thus, it reports a result of 0 against a target of 0. 100% 0% 20% 40% 60% 80% 100% Year 1 Year 2 Year 3 (dec 2017) target result 6 0 5 10 15 20 Year 1 Year 2 Year 3 (dec 2017) target result Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 88 • At the conclusion of the third quarter of Year 3, work readiness trainings (Najahi) were integrated during the quarter Oct-Dec 2017 into (6) departments of the Abdelmalek Essaadi University in Tangier at the FST (Faculté des Sciences et Techniques). The program reports a result of 6 educational institutions that integrated soft skills, through December 2017, against a target of 15. INDICATOR 14: % of individuals who report improved skills following participation in USG- assisted workforce development programs DESCRIPTION % of individuals who participate in USG workforce program who express that the program improved the skills needed for the workplace success. UNIT OF MEASURE % of individuals • The program began reporting on this indicator at the start of year 3, which began in April 2017. Therefore, the program does not have data to inform this indicator for October-December 2016 and January-March 2017. • At the conclusion of the third quarter of Year 3, the result to date of 90.58% youth who expressed that they improved skills following Career Center training far exceeded the target of 50%. • For trainings, electronic surveys were conducted. The response rate of these surveys was between 20% and 50%. Improved skills were self-reported. INDICATOR 15: % of employers satisfied with youth employees that were recruited through the Career Center. Note: Satisfied had not been defined. DESCRIPTION Number of companies who are satisfied by employing youth who benefit from the Career Center services. UNIT OF MEASURE Percentage of satisfied employers 90.58% 0% 20% 40% 60% 80% 100% Year 1 Year 2 Year 3 (dec 2017) target result Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 89 • As Career Centers were due to open in year 2, progress against targets were expected to be reported in year 2. Thus, in year 1, the program reports a result of 0 against a target of 0%. • In year 2, all of the companies interviewed six months after recruiting youth who benefited from Career Center services have expressed their satisfaction. 11 companies recruited 93 young people through the Career Center. The program achieved a result of 100% satisfied employers, double the target of 50%. • At the conclusion of the third quarter of Year 3, the program achieved a result to date of 92.30% satisfied employers and exceeded the target of 70%. • Note: The Annual Report, Year 2 (October 1st, 2016 - September 30, 2017) further states that the program received feedback from 13 companies. Those companies hired youth via the Career Center or youth who were trained by the Career Center. 12 of 13 companies were very satisfied with those youth, in terms of reduced turnover, better integration and improved teamwork and would like to collaborate with the Career Center on upcoming recruiting activities. The dissatisfaction of the last company was regarding the trainer and not the trainings modules. INDICATOR 16: % of companies with youth employees that have received Career Center services, reporting an improvement in productivity (e.g. lower rate of absenteeism, decreased employee turnover, increased production volumes) DESCRIPTION This indicator is used to define the percentage of companies reporting an improvement in productivity (e.g. lower rate of absenteeism, decreased employee turnover, increased production volumes) UNIT OF MEASURE % of companies 100% 92.30% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Year 1 Year 2 Year 3 (dec 2017) target result Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 90 • As Career Centers were due to open in year 2, progress against targets were expected to be reported in year 2. Thus, in year 1, the program results against targets are both N/A. • In year 2, trainings were organized with private companies during Aug. and Sept. 2016. Indicators to measure and evaluate desired changes have been developed and surveys were to be done within one or two months. Thus, the program results against targets are both N/A. • At the conclusion of the third quarter of Year 3, the program received feedback from ten companies with youth that have received Career Center services. 9 out of 10 companies reported an improvement in productivity. The program achieved a result to date of 90% companies far exceeding the target of 30% companies. • Note: this is based on a very small sample; moreover, the program indicates that it is not able to get the feedback to measure this indicator from the majority of its private sector partners/ recruiters. CROSS-CUTTING INDICATOR CC-3: Number of organizations with improved organizational performance as a result of USG assistance DESCRIPTION Number of CSOs/Career Centers/ other local organizations which have increased Organizational Performance Index (OPI) score when compared with baseline. The OPI is based on a 32-point scale and utilizes the IDRC/ framework Universalia & OECD which defines organizational performance in terms of effectiveness, efficiency, relevancy and sustainability. UNIT OF MEASURE Number of organizations with OPI scores which are higher at midline and endline compared to baseline 90% 0% 20% 40% 60% 80% 100% Year 1 Year 2 Year 3 (dec 2017) target result Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 91 • In year 1 and year 2, the result was zero against a target of zero. • At the conclusion of the third quarter of Year 3, the result to date was said to be reported annually against a baseline target of 5. CROSS-CUTTING INDICATOR CC-8: Number of people receiving gender-related training as a result of USG- assistance DESCRIPTION Gender disaggregated data collected during any training or program in any sector which include goals or objectives related to women’s empowerment. Include programs designed to raise broad awareness of human rights; programs that train journalists to report more responsibly on gender issues; education programs designed to change social norms and gender roles; programs designed to increase the political participation of women, youth development and empowerment with a discussion of gender. UNIT OF MEASURE Number of participants in a program, which includes gender-related training. • In year 1, the program reported 0 participants in a program which includes gender-related training against a target of 100 participants • In year 2, the program reported a result of 299 participants in a program which includes gender-related training, almost reaching the target of 300 participants. • At the conclusion of third quarter of Year 3, the program reports a result to date of 16 participants in a program which includes gender-related training against a target of 200 participants. 5 0 1 2 3 4 5 6 7 Year 1 Year 2 Year 3 (dec 2017) target result 299 16 0 50 100 150 200 250 300 350 Year 1 Year 2 Year 3 (dec 2017) target result Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 92 ANNEX 7: EVALUTION SCOPE OF WORK Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 93 Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 94 Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 95 Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 96 Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 97 Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 98 Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 99 Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 100 Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 101 Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 102 Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 103 Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 104 Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 105 Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 106 Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 107 Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 108 ANNEX 8: SUMMARY OF EVALUATION TEAM The following section highlights the evaluation team’s expertise as well as the roles and responsibilities of each team member. JEAN PIERRE ISBENDJIAN – TEAM LEADER Jean-Pierre Isbendjian is a multilingual youth development specialist with over 25 years of experience designing and leading multi-donor financed youth programs focused on building, strengthening and evaluating youth knowledge, competencies and skills. Mr. Isbendjian has successfully implemented multiple youth programs throughout the world, including in Morocco, Algeria, and Tunisia and has an in-depth understanding of these countries’ socio-economic conditions and political environment. Mr. Isbendjian also brings experience in evaluation and has served as a team lead/program manager on several youth assessments. Mr. Isbendjian brings extensive experience in Morocco and the Middle East and North Africa region. Working for the International Youth Foundation, he designed the TunisiaWorks project, a two￾year $1.9 million program supported by the Middle East Partnership Initiative (MEPI). In Morocco, he oversaw a program to enhance private sector employment prospects of underprivileged youth. He also worked with the International Organization for Migration to implement the Favorable Opportunities to Reinforce Self-Advancement for Today’s Youth, to improve life skills of marginalized youth. Mr. Isbendjian also worked with the Ministry of Education’s School Life Department in Morocco, designing participatory interview guides and conducting field visits to identify opportunities to integrate youth-focused skills training into extra-curricular clubs. Mr. Isbendjian has experience developing work readiness, life skills training, and job insertion programs for youth. These initiatives included developing and assessing the effectiveness of youth programming in delivering technical content, engaging the private sector, and sustaining overall impact. Mr. Isbendjian holds a Bachelor of Arts and is fluent in French and English. As the team leader, Mr. Isbendjian coordinated and directed the overall evaluation efforts and was responsible for day-to-day management of the team, evaluation design, workplan development, and presentations. He managed all evaluation activities, including the collection of data through online surveys, KIIs, FGDs, and workshops, and provided technical inputs and oversight to the team. He was responsible for the preparation and submission of all drafts of the evaluation report as well as the final evaluation report to Banyan Global, ensuring that all deliverables were completed on time and of the highest quality. SANDY WARK – SENIOR EVALUATION SPECIALIST Ms. Sandra Wark is an evaluation specialist with extensive experience in youth, workforce development and employment, international labor rights, and child labor programming. Ms. Wark has 25 years of experience designing and evaluating programs on youth skills development and employment, vocational training for youth, and civil society strengthening. She implemented six projects between 1995 – 2015 in Morocco, including 2 USAID-funded projects, and has a sound understanding of the country’s socio￾economic and political situation. Ms. Wark has served as a team leader for USAID and other donor-funded evaluations. She recently completed a USAID-funded, mid-term performance evaluation of the Workers’ Empowerment Program in Bangladesh, where she led teams through a participatory process to assess project impact and effectiveness. She provided technical assistance to the USAID International Organization on Migration FORSATY project in Tangiers and Tetouan to develop the project monitoring and evaluation (M&E) framework and a computerized management information system to collect data on beneficiaries and project activities. Ms. Wark designed and managed the cross-cutting information communication and technology component of the USAID-funded SANAD project in Morocco that focused on youth development, education and local governance. In this role, Ms. Wark collaborated with youth and youth organizations to develop innovative communication and advocacy strategies to raise awareness Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 109 on children’s rights issues. Additionally, she oversaw the development of the youth employment web portal providing information and tools to youth on career orientation, vocational training and employment in Morocco. Ms. Wark is a graduate of Georgetown University with a Master of Science in Foreign Service, as well as a graduate of Whitworth College with a Bachelor in International Studies. She is a US citizen but resides in Morocco. Ms. Wark is fluent in English and French. As the senior evaluation specialist team leader, Sandy Wark took the lead in managing the evaluation design and methodology as well as the development of the data collection tools and instruments and the data collection process including coding, cleaning and analyzing the data. She ensured that the evaluation team adhered to USAID evaluation policy and data collection best practices. Ms. Wark also led the desk study, identifying and summarizing key themes and findings from the relevant literature. She also served as the lead writer in the development of the draft and final evaluation reports. KHADIJA SAOUDI – SENIOR RESEARCH SPECIALIST Ms. Khadija Saoudi brings over 14 years of experience providing M&E support to USAID, international NGOs, and youth employment and entrepreneurship programs throughout Morocco. She has expertise in result-based management, qualitative and quantitative studies, knowledge management, capacity building, and youth training. Ms. Saoudi has designed and conducted a qualitative evaluation on the existing and potential rural youth clientele, staff, and partners of Al Barid Bank, including the administration of questionnaires and discussion guides, facilitation of focus groups, and interviews with youth and staff in French and Arabic. Additionally, while working with SILATECH on the BODOUR project, she designed and conducted a quantitative and qualitative market assessment on youth financial needs, to identify the financial and non-financial services available to create or develop their own businesses. Prior to this, she worked for MEDA Maroc on the YouthInvest Project for five years, first as the M&E manager and then as the deputy director. As the M&E manager, Ms. Saoudi designed and conducted quantitative and qualitative surveys and youth readiness reviews. She also contributed to the design, implementation, and supervision of an impact evaluation study and developed and managed the annual M&E and communications plans. Ms. Saoudi holds a Master of Arts in Econometrics and is pursuing a Ph.D. in Economics. She is a native Arabic and Berber speaker and is proficient in French and English. As the senior research specialist, Khadija Saoudi supported the development of the evaluation design and methodology as well as the development of the data collections tools and instruments and the data collection process including coding, cleaning and analysis of the data. Ms. Saoudi also supported the desk study in collaboration with Sandy Wark, senior evaluation specialist, identifying and summarizing key themes and findings from the relevant literature. DALIYA YASSIEN – SENIOR YOUTH DEVELOPMENT SPECIALIST Daliya Yassien is a multilingual youth development and evaluation specialist who has worked extensively with the World Bank Group, UNICEF, Save the Children and universities to promote youth empowerment and to build the capacity of international and national stakeholders to integrate youth throughout the program cycle. She has extensive youth development experience in strategic planning, research, monitoring and evaluation, communication management, implementation and reporting. Additionally, she has experience negotiating and coordinating with international partners while providing strategic and technical support to projects she has worked on. Ms. Yassien oversaw multiple projects regarding underprivileged youth empowerment domestically and internationally. While working for Save the Children, she drove UN General Assembly 69 (UNGA69), Education Advocacy/ Youth Engagement, Private Sector Engagement around Post-2015 Agenda, and CRC@25 activities, developing strategy and program toolkits, surveys, key performance indicators, workplans, working group material, and reports. At UNICEF she advised on youth empowerment projects in civic engagement and digital empowerment in Haiti and Brazil. At Bayer, she Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 110 supported programs to empower, educate, and simplify the lives of children and youth with diabetes and oversaw production of diabetes publications, educational materials, training guides, and program tools. Ms. Yassien also has extensive experience in the education field. As a graduate of the Graduate Center of the City University of New York (CUNY) she worked with the university’s Weissman Center for International Business to develop youth programs for underprivileged, first-generation youth through the Work Abroad Program. The program prepared youth for careers in the global economy and marketplace through international traineeships. As part of the university’s Center for Advanced Study in Education, she co-led the design and training of Long Island University’s Health STEM Service-Learning Project to help under-represented students in the Collegiate Science and Technology Entry Program (CSTEP) develop career awareness and readiness skills in the STEM fields. Ms. Yassien speaks French as a native language, is fluent in English, and speaks Arabic and Spanish at an intermediate level. As the senior youth development specialist, Daliya Yassien provided inputs on the development of the evaluation design and methodology, as well as the data collection tools and instruments, to ensure that they reflected positive youth development and were aligned with youth engagement best practices. Ms. Yassien also supported the administration of the data collection tools and instruments as well as the analysis of data. ROQUIA EL BOUDRARI – YOUTH DEVELOPMENT SPECIALIST Roquia El Boudrari, is a youth development specialist with over ten years of experience supporting donor-funded youth entrepreneurship, empowerment, and education programs in Morocco. She has provided technical support in designing positive youth development strategies, developing monitoring and evaluation protocols, and implementing youth programs for several NGO-funded projects in Morocco, including a USAID-supported project. She has hands-on experience conducting evaluations, leading participatory research, and summarizing data into higher level findings and conclusions. Recently, Ms. El Boudrari, conducted a diagnostic youth analysis for L’Heure Joyeuse, an association that provides training and employment assistance for at-risk youth. During the youth assessment, she administered participatory research and data collection methods to gather qualitative and quantitative data to inform the analysis. In 2009—2010, as Manager with L’Heure Joyeuse, Ms. El Boudrari implemented a professional orientation and integration program for underqualified youth, providing a combination of soft skills, guidance, and other services to increase their employability. As a manager with the Association Al Jisr, Ms. El Boudrari oversaw its training center that provides soft skills and employment training for Moroccan youth. In this role she collaborated with the Moroccan private sector and government stakeholders to identify opportunities for collaboration and improve youth employability. Ms. El Boudrari holds an Master of Arts in humanitarian studies from the Institut des Relations Internationales et Stratégiques (IRIS) in France and is a Moroccan national with fluency in French and Arabic and proficiency in English. As the youth development specialist, Roquia El Boudrari provided inputs on the development of the evaluation design and methodology as well as the data collection tools and instruments to ensure that they reflected positive youth development and were aligned with youth engagement best practices. Ms. El Boudrari also supported the administration of the data collection tools and instruments as well as the analysis of data. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 111 ANNEX 9: DATA COLLECTION PROTOCOLS 1. Online Survey on use of Career Center program 2. Online survey on sustainability for private sector representatives 3. USAID KII Guide 4. Career Center Program Managers and Specialist KII Guide 5. University or Vocational Training Career Center Staff KII Guide 6. University or Training Institute Directors/Administrators and/or Dean of Faculty KII Guide 7. Private Sector Employers KII Guide 8. Intermediary Organizations KII Guide 9. Government of Morocco Officials KII Guide 10. Direct Observations Career Center Activities Protocol 11. Youth FGD Guide 12. Faculty involved in TOT FGD Guide Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 112 MIDTERM PERFORMANCE EVALUATION OF THE USAID CAREER CENTER CAREER CENTER UTILIZATION ELECTRONIC SURVEY RESPONDENT INFORMATION: Sex of Respondent: ☐Male ☐Female Age: City of Residence: ☐Casablanca ☐Marrakech ☐Tangiers ☐Other ___________ Occupation: Please select as many that apply: ☐University Student ☐Public ☐Private ☐Vocational Training Center Trainee ☐Public ☐Private ☐High School Student ☐Middle School Student ☐Full-Time Employee ☐Part-Time Employee ☐Job Seeker ☐Other (please specify) ___________ Educational Institution: If you attend one of the education institutions where there is a Career Center, please indicate which one: ☐Université HASSAN II ☐Institut Spécialisé de Technologie Appliquée Hay Hassani 1 - (ISTA) ☐Université Cadi Ayyad de Marrakech, ☐Institut Spécialisé de Technologie Appliquée Hôtelière et Touristique - (ISTAHT) ☐Université Abdelmalek Essaadi ☐Institut Spécialisé dans les Métiers de l’Offshoring et les Nouvelles Technologies de l'information et de la Communication (ISMONTIC) This questionnaire has been prepared as part of a broader evaluation commissioned by the United States Agency for International Development (USAID) for the midterm performance evaluation of the USAID Career Center in Morocco. The purpose of this mid-term performance evaluation is to analyze the performance of the USAID Career at the midway point and to make action-oriented, evidence￾based recommendations to improve ongoing and future programming. Responses to the questionnaire will not be directly attributed to any individual in the final report. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 113 Career Center Services Accessed 1. I participated in the following Career Center services or activities (check as many as apply): ☐One on one counseling in a Career Center facility ☐Soft skills training (Najahi-Prêt pour l’Emploi)”. ☐Workshop (Mon Kit pour l’Emploi) ☐Job fair ☐Internship placement ☐Job Interview ☐Youth Ambassador program ☐Summer Camp ☐Online services available at http://vcc.careercenter.ma/vcc/ ☐Other (please fill in) ☐None of the above 2. How long has it been since you participated in your first Career Center service or activity? ☐More than one year ☐1 – 12 months ☐Less than one month 3. How long has it been since you last participated in a Career Center service or activity? ☐More than one year ☐1 – 12 months ☐Less than one month Feedback on Career Center Services 4. Based on your experience with Career Center services, to what extent do you agree with the following statement: “The career center staff was attentive to my needs.” ☐Strongly disagree ☐Disagree ☐Neutral ☐Agree ☐Strongly agree ☐Do not know 5. Based on your experience with Career Center services, to what extent do you agree with the following statement: “The Career Center space was welcoming for youth.” ☐Strongly disagree ☐Disagree ☐Neutral ☐Agree ☐Strongly agree ☐Not applicable 6. Based on your experience with the Virtual Career Center, to what extent do you agree with the following statement: “The Virtual Career Center is user-friendly and easy to navigate for youth.” ☐Strongly disagree ☐Disagree ☐Neutral ☐Agree ☐Strongly agree ☐Not applicable Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 114 7. Based on your experience with Career Center services, to what extent do you agree with the following statement: “I know more about what sectors may be offering jobs in the future that may interest me.” ☐Strongly disagree ☐Disagree ☐Neutral ☐Agree ☐Strongly agree ☐Do not know 8. Based on your experience with Career Center services, to what extent do you agree with the following statement: “I know more about what kinds of jobs and careers suit my personality and skills.” ☐Strongly disagree ☐Disagree ☐Neutral ☐Agree ☐Strongly agree ☐Do not know 9. Based on your experience with Career Center services, to what extent do you agree with the following statement: “I feel that I am ready to look for a job.” ☐Strongly disagree ☐Disagree ☐Neutral ☐Agree ☐Strongly agree ☐Do not know 10. Based on your experience with Career Center services, to what extent do you agree with the following statement: “Through the Career Center, I have learned about the importance of developing skills such as controlling my emotions, self-esteem, communicating clearly and thinking critically” ☐Strongly disagree ☐Disagree ☐Neutral ☐Agree ☐Strongly agree ☐Do not know 11. Based on your experience with Career Center services, to what extent do you agree with the following statement: “I have acquired new skills through the Career Center such as controlling my emptions, self-esteem, communicating clearly and thinking critically.” ☐Strongly disagree ☐Disagree ☐Neutral ☐Agree ☐Strongly agree ☐Do not know 12. Based on your experience with Career Center services, to what extent do you agree with the following statement: “The virtual Career Center is easy to use, and youth friendly.” ☐Strongly disagree ☐Disagree ☐Neutral ☐Agree ☐Strongly agree ☐Do not know 13. Based on your experience with Career Center services, to what extent do you agree ☐Strongly disagree ☐Disagree ☐Neutral Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 115 with the following statement: “The virtual Career Center has useful tools and content.” ☐Agree ☐Strongly agree ☐Do not know 14. Based on your experience with Career Center services, to what extent do you agree with the following statement: “If I can, I will participate in more Career Center activities and services.” ☐Strongly disagree ☐Disagree ☐Neutral ☐Agree ☐Strongly agree ☐Do not know 15. Based on your experience with the Career Center, is the following statement true or false: “The Career Center helped me to access an internship.” ☐True ☐False 16. Based on your experience with the Career Center, is the following statement true or false: “The Career Center helped me to get a job.” ☐True ☐False 17. Based on your experience with the Career Center, is the following statement true or false: “The Career Center helped me to network with potential employers.” ☐True ☐False 18. Based on your experience with the Career Center, is the following statement true or false: “The Career Center connected me to associations or agencies that can help with my ‘employability’.” ☐True ☐False 19. Do you have any suggestions for how the Career Center could be more effective in helping you improve my employability? (open answer, not mandatory) 20. Have you participated in any other career orientation, life skills or job placement programs besides the USAID Career Center? Please list. \ Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 116 MIDTERM PERFORMANCE EVALUATION OF THE USAID CAREER CENTER SUSTAINABILITY ELECTRONIC SURVEY RESPONDENT INFORMATION: Sector of the Company: Function of the Company: Location of Headquarters: ☐ Region of Casablanca ☐ Region of Marrakech ☐ Region of Tangiers ☐Other ___________ 1. Please indicate the types of activities you have collaborated with the Career Center on: ☐ None ☐ Workshop ☐ Job Fair / Job Fair ☐ Conference / Guest Speaker ☐ Company / organization site visit ☐ Sourcing / Recruitment (employee or trainee) ☐ Other (please specify) ____________________ 2. Are you aware of the activities offered by the Career Center? ☐ Yes ☐ No 3. Please indicate the services that the Career Center could offer that would be useful to your business: ☐ Train potential candidates for soft skills ☐ Propose trainees ☐ Propose candidates for specific jobs ☐ Train the company's employees in soft skills ☐ Accompany the company's HR strategy ☐ Support the company's CSR policy ☐ Strengthen the network of partners of the company ☐ Other (please specify) ____________________ This questionnaire has been prepared as part of a broader evaluation commissioned by the United States Agency for International Development (USAID) for the midterm performance evaluation of the USAID Career Center in Morocco. The purpose of this mid-term performance evaluation is to analyze the performance of the USAID Career Center at the midway point and to make action-oriented, evidence-based recommendations to improve ongoing and future programming. Responses to the questionnaire will not be directly attributed to any individual in the final report. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 117 4. Have you recruited career center participants for an internship and/or employment position? ☐ Yes ☐ No 5. Do career center participants meet your needs and requirements? ☐ Yes ☐ No 6. Please check, among the statements below, which one (s) reflect your experience with candidates from the Career Center ☐ CVs are well presented / written ☐ Candidates are well prepared for interviews ☐ Candidates communicate well ☐ No particular remark 7. Would you like to be able to distribute your job offers to Career Centers without having to disseminate them more widely? ☐ Yes ☐ No 8. At the end of USAID funding, would you like the Career Centers to continue? ☐ Yes ☐ No 9. Please indicate which of the following types of collaboration you might consider between your company and the Career Center: ☐ None ☐ Purchase of services in soft skills training ☐ Purchase of recruitment services ☐ Sponsoring or co-sponsoring events ☐ Funding or in-kind donations ☐ Time of employees of our organization (workshops, contribution to the governance of the Career Center) ☐ Other (please specify) 10. We would be happy to further discuss options with the evaluation team? ☐ Yes Name: Phone Number: ☐ No Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 118 MIDTERM PERFORMANCE EVALUATION OF THE USAID CAREER CENTER KEY INFORMANT INTERVIEW GUIDE, USAID Please state the following information for the record: RESPONDENT INFORMATION: ID [write name in the coding sheet and enter the ID code on this interview form]: Date of Interview: Sex of Respondent: ☐Male ☐Female Organization: Function/job of the Respondent: Interviewer Name: Primary Notetaker Name: City: Have we obtained a completed informed consent form? ☐ Yes ☐ No Do we have permission to take their photos and use them in reporting? ☐ Yes ☐ No Informed Consent Protocol Use the informed consent procedures: read informed consent script and ask KII if they agree to participate. Initials of evaluator to indicate receipt of verbal consent: _____________________ Date: _________________________ This interview guide has been prepared as part of a broader evaluation commissioned by the United States Agency for International Development (USAID) for the midterm performance evaluation of the USAID Career Center in Morocco. The purpose of this mid-term performance evaluation is to analyze the performance of the USAID Career Center at the midway point and to make action-oriented, evidence-based recommendations to improve ongoing and future programming. Responses during this interview will not be directly attributed to any individual in the final report. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 119 1. Which Career Center program activities do you consider particularly successful/unsuccessful in terms of promoting the acquisition of work readiness and soft skills required by the private sector by students/trainees? Why? (EQ1) Probing Sub-Questions: • To what extent has the program increased collaboration among key stakeholders (universities, vocational training centers, employers and their professional associations, public agencies, NGOs and other employment/skill development intermediaries) in the youth employment ecosystem in Morocco? • To what extent are the services that are offered in the physical and virtual Career Centers expanding youth users’ understanding of employment trends, demand for skills, and available career pathways? • To what extent has the program been able to contribute to increasing opportunities within universities and vocational training centers for youth to acquire soft skills required by the private sector? 2. Which Career Center program activities do you consider particularly successful/unsuccessful in involving its target population and key stakeholders in participatory planning and decision-making? Why? (EQ2) Probing Sub-Questions: • To what extent has the program been able to involve key public, private civil society partners and youth in the development and testing of the Career Center model? • To what extend have CC allowed youth to involve themselves in the CCs operation, or support services, volunteering, youth-led events, etc. • Have some partners been more responsive than others? Why? • To what extent have feedback loops been created and capitalized upon to improve the model? • What else could CC do to improve collaboration and stakeholder feedback? • Overall, how relevant do you think the CC model and its activities are to meeting the needs of key stakeholders (students, educators, government counterparts, private sector, civil society organizations)? How could this be improved? 3. Which Career Center activities do you consider particularly successful/unsuccessful in promoting the sustainability and scaling of the Career Center model? How do you define success? Why? (EQ3) Probing Sub-Questions: • To what extent have program efforts to engage key stakeholders fostered a sense of ownership and commitment to the model? Are some partners more committed than others? • To what extent are relevant partners demonstrating willingness to invest the resources necessary to sustain and/or scale the existing centers? What types of investment pose the greatest challenges to the institutions that should maintain or scale the centers? • To what extent has the program been able to build the capacity of partner personnel responsible for the quality and relevance of Career Center services (both physical and virtual center services)? • What do you think are the prospects for sustainability? What could be done differently to improve the chances of sustainability? Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 120 4. What external factors are best facilitating the Career Center program achievement of the activity goals and objectives? Which factors most constrain the Career Center Program’s achievement of the activity goals and objectives? (EQ1,2, 3) Probing Sub-Questions: • How have political, economic, social, and/or institutional contextual factors helped or hindered program achievement in targeting youth to acquire the work readiness and soft skills required by the private sector? • How have political, economic, social and/or institutional contextual factors helped or hindered program achievement in engaging with Career Center stakeholders to improve the model? • How have political, economic, social and/or institutional contextual factors helped or hindered Career Center sustainability? 5. What changes (if any) to existing strategies and activities do you think could improve the Career Center program’s effectiveness? (EQ1, 2, 3) Probing Sub-Questions: • Is there anything that should be done in the time remaining in the program to improve the quality and relevance of physical and virtual Career Center Services? Increase program engagement with relevant stakeholders? Promote the Career Center model? Increase the sustainability and scalability of the Career Center model? 6. To what extent has the Career Center offered services to meet the specific needs of particular categories of youth? Are there any significant differences in approach relative to: • male vs. female students/trainees • mature and less mature students/trainees • university students vs. vocational training center trainees • students with disabilities, large • small & medium enterprise employers vs. very small enterprise employers • youth seeking employment vs. youth seeking to create an enterprise? To what extent has the program been able to address these differences? (EQ1, 2) 7. What unintended outcomes (positive or negative) from the Career Center program have you seen thus far, if any? (Cross Cutting) 8. To what extent and how do you think the IPs have effectively managed the Career Center program so far (programmatic, admin, finance)? Do you have any particular concerns about any of Career Center implementation modalities? (Cross Cutting) 9. What lessons have been learned in the implementation of the Career Center program to date? (Cross Cutting) 10. What (if any) have been the good practices used by the Career Center that you think might be useful to other organizations implementing a similar program? (Cross Cutting) 11. Do you have any other feedback or comments that you would like to share that were not covered by these questions? (Cross Cutting) Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 121 MIDTERM PERFORMANCE EVALUATION OF THE USAID CAREER CENTER KEY INFORMANT INTERVIEW GUIDE, CAREER CENTER PROGRAM MANAGERS AND SPECIALISTS Please state the following information for the record: RESPONDENT INFORMATION: ID [write name in the coding sheet and enter the ID code on this interview form]: Date of Interview: Sex of Respondent: ☐Male ☐Female Organization: Function/job of the Respondent: Interviewer Name: Primary Notetaker Name: City: Have we obtained a completed informed consent form? ☐ Yes ☐ No Do we have permission to take their photos and use them in reporting? ☐ Yes ☐ No Informed Consent Protocol Use the informed consent procedures: read informed consent script and ask KII if they agree to participate. Initials of evaluator to indicate receipt of verbal consent: _____________________ Date: _________________________ This interview guide has been prepared as part of a broader evaluation commissioned by the United States Agency for International Development (USAID) for the midterm performance evaluation of the USAID Career Center in Morocco. The purpose of this mid-term performance evaluation is to analyze the performance of the USAID Career Center at the midway point and to make action-oriented, evidence￾based recommendations to improve ongoing and future programming. Responses during this interview will not be directly attributed to any individual in the final report. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 122 1. What have been the main intervention strategies of the Career Center program to promote the acquisition of work readiness and soft skills required by the private sector by students/trainees? (EQ1) Area of Impact Strategies Work readiness: youth understanding of employment trends, demand for skills, potential for employment generation in growth sectors, and available career pathways. Soft Skills: development of communica￾tions/interpersonal skills, cooperation/ teamwork, self-confidence, creative thinking, critical thinking, empathy, responsibility, respect for self and others, managing emotions, decision-making, and conflict management 2. Which Career Center program activities do you consider especially successful/unsuccessful in terms of promoting the acquisition of work readiness and soft skills required by the private sector by students/trainees? Why? (EQ1) Probing Sub-Questions: • To what extent has the program increased collaboration among key stakeholders (universities, vocational training centers, employers and their professional associations, public agencies, NGOs and other employment/skill development intermediaries) in the youth employment ecosystem in Morocco? • To what extent are the services that are offered in the physical Career Centers expanding youth users’ understanding of employment trends, demand for skills, and available career pathways? • To what extent are the services that are offered in the Virtual Career Center expanding youth users’ understanding of employment trends, demand for skills, and available career pathways? • To what extent has the program been able to contribute to increasing opportunities within universities and vocational training centers for youth to acquire soft skills required by the private sector? • What are the main obstacles/constraints faced by the program in this area of intervention? 3. Which Career Center program activities do you consider particularly successful/unsuccessful in involving key stakeholders in participatory planning and decision-making? Why? (EQ2) Probing Sub-Questions: • To what extent has the program been able to involve and collaborate with key public, private and civil society partners, and youth in the development and testing of the Career Center model? Have some partners been more responsive than others? Why? • To what extent has stakeholder feedback been used to improve performance? Please provide examples. • To what extent have feedback loops been created and capitalized upon to improve the model? What are the main obstacles/constraints faced by the program in this area of intervention? • How relevant do you think the Career Center model/activities are to key stakeholders (students, educators, government counterparts, private sector, and civil society)? Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 123 4. Which Career Center program activities do you consider particularly successful/unsuccessful in promoting the sustainability and scaling of the Career Center model? Why? (EQ3) Probing Sub-Questions: • To what extent have program efforts to engage key stakeholders fostered a sense of ownership and commitment to the model? Are some partners more committed than others? • To what extent are relevant partners demonstrating willingness to invest the resources necessary to sustain and/or scale the existing centers? What types of investment pose the greatest challenges to the institutions that should maintain or scale the centers? • To what extent has the program been able to build the capacity of partner personnel responsible for the quality and relevance of Career Center services (both physical and virtual center services)? • What are the main obstacles/constraints faced by the program in this area of intervention? 5. To what extent has the Career Center offered services adapted to groups with particular needs? Are there any significant differences in approach relative to: • male vs. female students/trainees • mature and less mature students/trainees • university students vs. vocational training center trainees • persons with disabilities • large, small & medium enterprise employers vs. very small enterprise employers, youth seeking employment vs. youth seeking to create an enterprise? 6. To what extent has the program been able to address these differences? Could the program be more effective in adapting its services to the needs of these groups? (EQ1, 2) 7. How has the Career Center Program used its analyses to influence change in the programs & services of others at the national and/or international level through presentations, training and/or publications? (EQ2) 8. To what extent and how has the Career Center built the capacity of its partners to sustain activities beyond the conclusion of the Career Center program? What are the challenges? What are the outcomes of Career Center capacity building to date? What do you think are the prospects for sustainability? (EQ3) Capacity Building strategies Challenges Outcomes to date 9. What lessons have been learned in the implementation of the Career Center program to date? (Cross Cutting) Area of Intervention Lessons Learned Promoting the acquisition of work readiness and soft skills required by the private sector by students/trainees Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 124 Engaging in participatory planning and decision￾making processes that involve Career Center program target population Promoting sustainability Other (explain) 10. What unintended outcomes (positive or negative) from the Career Center program have you seen thus far, if any? (Cross Cutting) 11. What (if any) have been the good practices used by the Career Center that you think might be useful to other organizations implementing a similar program? (Cross Cutting) 12. Do you have any other feedback or comments that you would like to share that were not covered by these questions? (Cross Cutting) Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 125 MIDTERM PERFORMANCE EVALUATION OF THE USAID CAREER CENTER KEY INFORMANT INTERVIEW GUIDE, UNIVERSITY AND VOCATIONAL TRAINING CAREER CENTER STAFF Please state the following information for the record: RESPONDENT INFORMATION: Date of Interview: Organization: Interviewer Name: Primary Notetaker Name: City: Resp. 1 Resp. 2 Resp. 3 Resp. 4 Resp. 5 Resp. 6 ID [write name in the coding sheet and enter the ID code on this interview form]: Sex of Respondent: ☐Male ☐Female ☐Male ☐Female ☐Male ☐Female ☐Male ☐Female ☐Male ☐Female ☐Male ☐Female Function/job of the Respondent: Have we obtained a completed informed consent form? ☐ Yes ☐ No ☐ Yes ☐ No ☐ Yes ☐ No ☐ Yes ☐ No ☐ Yes ☐ No ☐ Yes ☐ No Do we have permission to take ☐ Yes ☐ No ☐ Yes ☐ No ☐ Yes ☐ No ☐ Yes ☐ No ☐ Yes ☐ No ☐ Yes ☐ No This interview guide has been prepared as part of a broader evaluation commissioned by the United States Agency for International Development (USAID) for the midterm performance evaluation of the USAID Career Center in Morocco. The purpose of this mid-term performance evaluation is to analyze the performance of the USAID Career Center at the midway point and to make action-oriented, evidence-based recommendations to improve ongoing and future programming. Responses during this interview will not be directly attributed to any individual in the final report. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 126 their photos and use them in reporting? Informed Consent Protocol Use the informed consent procedures: read informed consent script and ask KII if they agree to participate. Initials of evaluator to indicate receipt of verbal consent: _____________________ Date: _________________________ 1. In your role as a Career Center staff, what are your main objectives and activities? (EQ1, EQ2, EQ3) Probing Sub-Questions: • What are your main duties? How do you split your time between various duties? Do you have a Job Description? How do you relate your job description to your work? Do you have clear objectives and targets that you should meet? Established by whom? 2. What have been the most significant obstacles you have faced in the development and implementation of services to help target youth acquire the work readiness and soft skills required by the private sector? In engaging in dialogue with relevant Career Center stakeholders to improve the model? (EQ1, EQ2) Probing Sub-Questions: • Have you faced any resistance from stakeholders (youth, the administration, external partners)? • Have you been provided with appropriate tools to carry out your work? • Have you received adequate training to do your job? 3. Which activities in your career center do you consider especially successful/unsuccessful in terms of promoting the acquisition of work readiness and soft skills required by the private sector by students/trainees? Why? (EQ1) Probing Sub-Questions: • To what extent has your center increased collaboration among key stakeholders (universities, vocational training centers, employers and their professional associations, public agencies, NGOs and other employment/skill development intermediaries) in the youth employment ecosystem in Morocco? • To what extent are the services that are offered in your center and in the virtual Career Center expanding youth users’ understanding of employment trends, demand for skills, and available career pathways? • To what extent has your center been able to contribute to increasing opportunities within universities and vocational training centers for youth to acquire soft skills required by the private sector? • What are the main obstacles/constraints faced by the program in this area of intervention? 4. Which activities in your center do you consider particularly successful/unsuccessful in involving key stakeholders in participatory planning and decision-making? Why? (EQ2) Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 127 Probing Sub-Questions: • To what extent has your center been able to involve key public, private, civil society partners and youth in the development and testing of the Career Center model? Have some partners been more responsive than others? Why? • To what extent have feedback loops been created and capitalized upon to improve services in your center? • To what extent has stakeholder feedback been used to improve performance? Please provide examples. • What are the main obstacles/constraints faced by your center when trying to engage with key stakeholders? • How relevant do you think the Career Center model/activities are to key stakeholders (students, educators, government counterparts, private sector, and civil society)? 5. To what extent has the Career Center offered services adapted to groups with particular needs? How could they be adapted to be more effective? Are there any significant differences in approach relative to: • male vs. female students/trainees • mature and less mature students/trainees • university students vs. vocational training center trainees • persons with disabilities • large, small & medium enterprise employers vs. very small enterprise employers, youth seeking employment vs. youth seeking to create an enterprise? 6. What types of support have you received from the Career Center central program managers and specialists? To what extent has support been sufficient to meet your needs? Are there ways that the support might be improved or adapted to be more effective in building your capacity as Career Center staff? (EQ3) Probing Sub-Questions: • Have you received training? Mentoring? Other types of support? Which types of support have been especially impactful? • Have you received feedback on how to improve your performance? • How often do you meet with Career Center central program staff and specialists? What types of issues are discussed? Are there any critical issues that you have not discussed? • How well is the program balancing its program management needs (need for information, pressure to meet targets) with center needs (need for autonomy, need to be responsive to the host institution)? 7. In what ways, if any, has Career Center support influenced how you work? (EQ 3) Probing Sub-Questions: • In which ways have you been able to apply the skills you learned from the Career Center program managers and specialists in your day-to-day work with students/trainees? What are some examples? Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 128 8. How confident are you that you will be able to continue your work when the support provided by USAID Career Centers program concludes? (EQ 3) Probing Sub-Questions: • Do you think your University/Vocational Training Center will be able to provide you with adequate support to continue? What are your most important areas of concern? Funding? Coaching and Technical Assistance? 9. What additional efforts are needed to ensure that you are able to continue your work after the end of the USAID Career Center program support? (EQ3) 10. What lessons have been learned in the implementation of the Career Center program to date? (Cross Cutting) 11. What (if any) have been the good practices used by the Career Center that you think might be useful to other organizations implementing a similar program? (Cross Cutting) 12. Do you have any other feedback or comments that you would like to share that were not covered by these questions? (Cross Cutting) Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 129 MIDTERM PERFORMANCE EVALUATION OF THE USAID CAREER CENTER KEY INFORMANT INTERVIEW GUIDE, UNIVERSITY AND VOCATIONAL TRAINING DIRECTORS, ADMINISTRATORS AND/OR DEAN OF FACULTY Please state the following information for the record: RESPONDENT INFORMATION: ID [write name in the coding sheet and enter the ID code on this interview form]: Date of Interview: Sex of Respondent: ☐Male ☐Female Organization: Function/job of the Respondent: Interviewer Name: Primary Notetaker Name: City: Have we obtained a completed informed consent form? ☐ Yes ☐ No Do we have permission to take their photos and use them in reporting? ☐ Yes ☐ No Informed Consent Protocol Use the informed consent procedures: read informed consent script and ask KII if they agree to participate. Initials of evaluator to indicate receipt of verbal consent: _____________________ Date: _________________________ This interview guide has been prepared as part of a broader evaluation commissioned by the United States Agency for International Development (USAID) for the midterm performance evaluation of the USAID Career Center in Morocco. The purpose of this mid-term performance evaluation is to analyze the performance of the USAID Career Center at the midway point and to make action-oriented, evidence￾based recommendations to improve ongoing and future programming. Responses during this interview will not be directly attributed to any individual in the final report. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 130 1. Please describe your partnership with the USAID Career Center program. (EQ1, EQ2) Probing Sub-Questions: • To what extent has your institution been involved in developing, testing and validating the Career Center model? Please describe how. • Does the Career Center program have on-going mechanisms to regularly obtain your feedback? Please describe. Could these be improved? • Do you think that your perspective and inputs were ultimately reflected in the model? • How relevant do you think the Career Center model/activities are to key stakeholders (students, educators, government counterparts, private sector, and civil society)? 2. What types of support has your institution received from the USAID Career Center program as part of this partnership. (EQ1, EQ2, EQ3) 3. To what extent has the program increased your institution’s collaboration with key stakeholders (employers and their professional associations, public agencies involved in youth employment issues, NGOs and other employment/skill development intermediaries) in the youth employment ecosystem in Morocco? To the extent that collaboration has increased, have there been any noteworthy outcomes of the collaboration? Please provide examples. 4. To what extent has the program been able to contribute to increasing opportunities within your institution for youth to acquire soft skills required by the private sector? How? (Soft skills instructional activities integrated into existing curricula? Development of new extracurricular activities?) Please provide examples. What are the main obstacles/constraints? (EQ1) 5. To what extent do you think that the services that are offered in the physical and virtual Career Centers are expanding youth users’ understanding of employment trends, demand for skills, and available career pathways? How might services be improved? What are the main obstacles/constraints? (EQ1) 6. To what extent has your institution’s collaboration with the USAID Career Center program made you more aware of needs and issues affecting the work readiness of female students? Students with disabilities? What are ways that these concerns might be addressed? (Cross cutting) 7. To what extent do you think that the partnership between your institution and the USAID Career Center program has been effectively managed? (EQ1,2,3) Probing Sub-Questions: • Has the level of communication between your institution and USAID Career Center personnel been adequate? Have the roles and responsibilities for project implementation been clearly defined? To what extent has your institution been given the opportunity to contribute to key decisions related to the career center model and its implementation? 8. How confident are you that you will be able to sustain Career Center work when the support provided by the USAID Career Centers program concludes? What more needs to be done between now and the end of the program to enhance sustainability? (EQ3) Probing Sub-Questions: Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 131 • To what extent has the program been effective in building the capacity of your institution to sustain/scale Career Center services (both physical and virtual center services)? What are the main obstacles/constraints faced by your institution? (Financial? Human resources? Know-how?) 9. Are you receiving technical or financial support from other international organizations or NGOs to strengthen your institutions capacity to build work readiness and softs kills of students/trainees? Which other organizations and how have they supported you? (CC) Probing Sub-Questions: • How does the work of these other organizations complement or fill gaps in the support you receive from the Career Center? 10. What lessons have been learned in the implementation of the Career Center program to date? (Cross Cutting) 11. What (if any) have been the good practices used by the Career Center that you think might be useful to other organizations implementing a similar program? (Cross Cutting) 12. Do you have any other feedback or comments that you would like to share that were not covered by these questions? (Cross Cutting) Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 132 MIDTERM PERFORMANCE EVALUATION OF THE USAID CAREER CENTER KEY INFORMANT INTERVIEW GUIDE, PRIVATE SECTOR EMPLOYERS Please state the following information for the record: RESPONDENT INFORMATION: ID [write name in the coding sheet and enter the ID code on this interview form]: Date of Interview: Sex of Respondent: ☐Male ☐Female Company Name: Sector: Title of Respondent: Function/job of the Respondent: Interviewer Name: Primary Notetaker Name: City: Have we obtained a completed informed consent form? ☐ Yes ☐ No Do we have permission to take their photos and use them in reporting? ☐ Yes ☐ No Informed Consent Protocol Use the informed consent procedures: read informed consent script and ask KII if they agree to participate. Initials of evaluator to indicate receipt of verbal consent: _____________________ Date: _________________________ This interview guide has been prepared as part of a broader evaluation commissioned by the United States Agency for International Development (USAID) for the midterm performance evaluation of the USAID Career Center in Morocco. The purpose of this mid-term performance evaluation is to analyze the performance of the USAID Career Center at the midway point and to make action￾oriented, evidence-based recommendations to improve ongoing and future programming. Responses during this interview will not be directly attributed to any individual in the final report. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 133 1. Please briefly describe your enterprise. Probing Sub-Questions: • Size of enterprise? Sector of activity? Structure (number of offices, where)? Main types of employment offered? 2. How has your enterprise been involved with the USAID Career Center program? (EQ1, EQ2) Probing Sub-Questions: • Have you participated in any of Career Center activities i.e. training, meeting, workshop, or events organized by the Career Center program? • Were you assisted to find an intern? An employee? If so, what was your experience with the hire? 3. Have you been involved in providing input into the design of the Career Center model and activities? If so, what are examples of collaboration/interaction? Have there been any positive or negative outcomes of this collaboration/interaction for your enterprise? (EQ1, EQ2) Probing Sub-Questions: • Have you had opportunities to share information about the skills you look for in employees? The types of profiles you are likely to recruit in the coming month, years? The types of services that the CC should offer? How, with whom and how often? (youth, educators, Career Center personnel). • Does the CC have on-going mechanisms to regularly obtain your feedback? Can these be improved? • Do you think your feedback has been used by the CC to improve its performance? 4. What is your perception of the work done by the Career Center program in the area of increasing the work readiness and soft skills of youth? Do you have any suggestions for how the program could be more effective? (EQ1,2) Probing Sub-Questions: • Probe: If you provided employment or an internship to a CC youth, how would you rate their soft skills? Do you think the services offered by the Career Center program are relevant to the needs of employers in your sector of activities? Of students/trainees? Do you think it has changed students/trainees’ ability to access employment? Contributed to the productivity of your company? How? 5. What role if any do you think private enterprises like yours could have in ensuring the sustainability of Career Center program services after the end of USAID support? (EQ3) Probing Sub-Questions: • Are there any services offered by the Career Centers that your enterprise would be able and willing to contribute financial or in-kind resources? Which ones? What resources? Are there Career Center activities that your enterprise would be willing to sponsor? What types of activities? • Do you have any other thoughts about how to improve the sustainability of Career Centers? Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 134 6. What (if any) have been the good practices for involving employers used by the Career Center that you think might be useful to other organizations implementing a similar program? (Q2, Cross Cutting) 7. Do you have any other feedback or comments that you would like to share that were not covered by these questions? (Cross Cutting) Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 135 MIDTERM PERFORMANCE EVALUATION OF THE USAID CAREER CENTER KEY INFORMANT INTERVIEW GUIDE, INTERMEDIARY ORGANIZATIONS Please state the following information for the record: RESPONDENT INFORMATION: ID [write name in the coding sheet and enter the ID code on this interview form]: Date of Interview: Sex of Respondent: ☐Male ☐Female Organization: Title of Respondent: Function/job of the Respondent: Interviewer Name: Primary Notetaker Name: City: Have we obtained a completed informed consent form? ☐ Yes ☐ No Do we have permission to take their photos and use them in reporting? ☐ Yes ☐ No Informed Consent Protocol Use the informed consent procedures: read informed consent script and ask KII if they agree to participate. Initials of evaluator to indicate receipt of verbal consent: _____________________ Date: _________________________ This interview guide has been prepared as part of a broader evaluation commissioned by the United States Agency for International Development (USAID) for the midterm performance evaluation of the USAID Career Center in Morocco. The purpose of this mid-term performance evaluation is to analyze the performance of the Career Center at the midway point and to make action-oriented, evidence￾based recommendations to improve ongoing and future programming. Responses during this interview will not be directly attributed to any individual in the final report. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 136 1. Please briefly describe your organization and its mission? Probing Sub-Questions: • How is your organization involved in facilitating access to employment or skills for employment by youth? 2. How has your organization been involved with the USAID Career Center program? (EQ1, EQ2) Probing Sub-Questions: • Have you participated in any of Career Center activities i.e. training, meeting, workshop, or events organized by the Career Center program? 3. To what extent has your involvement in the USAID Career Center program increased your interaction/collaboration with any of the following groups:  Students/trainees  Public agencies involved in promoting youth employment  Universities or vocational training center administrators  Universities or vocational training center faculty  NGOs helping youth find employment  Professional associations in your sector If so, what are examples of collaboration/interaction? Have there been any positive or negative outcomes of this collaboration/interaction for your organization? (EQ1, EQ2) Probing Sub-Questions: • Have you had opportunities to share information about your services to relevant partners or potential beneficiaries? How and with whom? (youth, educators, Career Center personnel) 4. Have you been involved in providing input into the design of the Career Center model and activities? • Is the program provide services to youth that your organization serves? Were you able to draw attention/get recognition for your organization’s concerns and programs? What positive or negative outcomes? • Does the CC have on-going mechanisms to regularly obtain your feedback? Can these be improved? • Do you think your feedback has been used by the CC to improve its performance? 5. What is your perception of the work done by the Career Center program in the area of increasing the work readiness and soft skills of youth? Do you have any suggestions for how the program could be more effective? (EQ1) Probing Sub-Questions: • Do you think the services offered by the Career Center program are relevant to the needs of employers in your sector of activities? Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 137 • If you have are in contact with youth who have participated in Career Center programs, do you see any changes in the knowledge or behaviors that might improve their chances for employment? How? 6. What role if any do you think organizations like yours could have in ensuring the sustainability of Career Center program services after the end of USAID support? (EQ3) Probing Sub-Questions: • Are there any services offered by the Career Centers that your organization would be able and willing to contribute financial or in-kind resources? Which services? What kinds of resources? Are there Career Center activities that your organization would be willing to sponsor? Which activities? • Are there other opportunities to strengthen the sustainability of the Career Centers? 7. What (if any) have been the good practices for involving employers or civil society organizations used by the Career Center that you think might be useful to other organizations implementing a similar program? (Cross Cutting) 8. Do you have any other feedback or comments that you would like to share that were not covered by these questions? (Cross Cutting) Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 138 MIDTERM PERFORMANCE EVALUATION OF THE USAID CAREER CENTER KEY INFORMANT INTERVIEW GUIDE, GOVERNMENT OF MOROCCO OFFICIALS Please state the following information for the record: RESPONDENT INFORMATION: ID [write name in the coding sheet and enter the ID code on this interview form]: Date of Interview: Sex of Respondent: ☐Male ☐Female Ministry/Department: Title of Respondent: Function/job of the Respondent: Interviewer Name: Primary Notetaker Name: City: Have we obtained a completed informed consent form? ☐ Yes ☐ No Do we have permission to take their photos and use them in reporting? ☐ Yes ☐ No Informed Consent Protocol Use the informed consent procedures: read informed consent script and ask KII if they agree to participate. Initials of evaluator to indicate receipt of verbal consent: _____________________ Date: _________________________ 1. Please describe your relationship with the USAID Career Center program. (EQ1, EQ2) This interview guide has been prepared as part of a broader evaluation commissioned by the United States Agency for International Development (USAID) for the midterm performance evaluation of the USAID Career Center in Morocco. The purpose of this mid-term performance evaluation is to analyze the performance of the USAID Career Center at the midway point and to make action￾oriented, evidence-based recommendations to improve ongoing and future programming. Responses during this interview will not be directly attributed to any individual in the final report. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 139 Probing Sub-Questions: • To what extent has your institution been involved in developing, testing and validating the Career Center model? Please describe how. 2. What types of support has your institution received from the USAID Career Center program as part of this relationship. (EQ1, EQ2, EQ3) 3. To what extent has the USAID Career Center program increased your institution’s collaboration with key stakeholders in the youth employment eco-system:  Students/trainees  Public agencies involved in promoting youth employment  Universities or vocational training center administrators  Universities or vocational training center faculty  NGOs helping youth find employment  Professional associations To the extent that collaboration has increased, have there been any noteworthy outcomes (positive or negative) of the collaboration? Please provide examples. 4. To what extent do you think that the services that are offered in the physical and virtual Career Centers are expanding youth users’ understanding of employment trends, demand for skills, and available career pathways? Are there ways that services might services be improved? What are the main obstacles/constraints? (EQ1) 5. To what extent do you think that the services that are offered in the physical and virtual Career Centers are meeting the human resource needs of private sector employers? (EQ1) 6. To what extent is the USAID Career Center program relevant to the political and programmatic priorities of your institution? Please explain how the program responds/doesn’t respond to priorities. (EQ3) 7. To what extent do you think that the relationship between your institution and the USAID Career Center program has been effectively managed? (EQ1,2,3) Probing Sub-Questions: • Has the level of communication between your institution and USAID Career Center personnel been adequate? • Have the roles and responsibilities for project implementation been clearly defined? • To what extent has your institution been given the opportunity to contribute to key decisions related to the career center model and its implementation? • Are there on-going mechanisms that enable you to provide feedback to the Career Center? 8. To what extent do you think that the Career Center model is relevant and should be sustained and/or scaled in the future? 9. How confident are you that your institution will be able to sustain/scale USAID Career Center initiatives and outcomes when the support provided by the USAID concludes? What types of actions Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 140 and/or forms of support are needed between now and the end of the program to enhance sustainability? By whom? (EQ3) Probing Sub-Questions: • To what extent has the program been effective in building the capacity of your institution to sustain/scale Career Center services (both physical and virtual center services)? • What are the main obstacles/constraints faced by your institution? (Financial? Human resources? Know-how?) 10. Are you receiving technical or financial support from other international organizations or NGOs to strengthen your institution’s capacity to build work readiness and soft skills of students/trainees? Which other organizations and how have they supported you? (CC) Probing Sub-Questions: • How does the work of these other organizations complement or fill gaps in the support you receive from the Career Center? 11. What lessons have been learned in the implementation of the Career Center program to date? (Cross Cutting) 12. What (if any) have been the good practices used by the Career Center that you think might be useful to other organizations implementing a similar program? (Cross Cutting) 13. Do you have any other feedback or comments that you would like to share that were not covered by these questions? (Cross Cutting) Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 141 MIDTERM PERFORMANCE EVALUATION OF THE USAID CAREER CENTER CAREER CENTER DIRECT OBSERVATIONS Name of the Center: City: Date: Name of Observer/s: 1. Accessibility Yes Part No Notes Does the Career Center have a promotional messaging strategy targeting young people? Does the Career Center have a promotional messaging strategy targeting both young women and young men? Is there a confidentiality policy widely publicized to young people, their parents and support staff? Does the Career Center use creative, innovative strategies including technology and activity-based approaches to engage young people? Are services provided free, or at a cost affordable to young people? Are services expected to be provided free, or at a cost affordable to young people, after the end of the USAID support? Can young people access the service easily, without too much paperwork? Is the service open when young people want to access it? Is the service sensitive to the cultural and language needs of young people? Is it possible for young people to drop in and use some services without having made an appointment? This observation score card has been prepared as part of a broader evaluation commissioned by the United States Agency for International Development (USAID) for the midterm performance evaluation of the USAID Career Center in Morocco. The purpose of this mid-term performance evaluation is to analyze the performance of the USAID Career Center at the midway point and to make action-oriented, evidence￾based recommendations to improve ongoing and future programming. Personal identifiable information recorded as part of the observations will be kept confidential and observations made will not be attributable to any individuals in the final report. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 142 1. Accessibility Yes Part No Notes Is there capacity to offer longer sessions to deal with more complex issues/considerations that may arise? Do support staff appear to be welcoming and happy to help? (not busy on their computer or smart phone) Can young people come and chat among themselves without directly accessing specific services? Does the Career Center encourage the involvement of other stakeholders in their activities? Which stakeholders? Please take pictures and describe: Suggested actions: 2. Youth Participation Yes Part No Notes Does the Career Center involve young people in activity planning and other services? Does the Career Center enable/encourage any form of peer-to-peer learning among students? Was the Career Center busy during the visit? How many youth were present? How many females, how many males? How long did the youth stay in average? Shortest stay? Longest stay? Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 143 2. Youth Participation Yes Part No Notes Suggested actions: 3. Equipment Yes Part No Notes Does the center have adequate space? Can students access online tools? Does the Internet connection work? Describe equipment (pictures): 4. Professional Development Yes Part No Notes Is the CC Staff available to respond to the questions of center users? Do the staff respond to the needs of the visitors? When needed, are the staff able to redirect the visitors to outside services? Suggested actions: 5. Evaluation Yes Part No Notes Does the Career Center evaluate its services on a regular basis, including seeking feedback from young people? Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 144 5. Evaluation Yes Part No Notes Suggested actions: 6. Other Observations / Comments Yes No Are there other observations/comments that you want to include? Notes: Suggested actions: Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 145 MIDTERM PERFORMANCE EVALUATION OF THE USAID CAREER CENTER FOCUS GROUP DISCUSSION GUIDE, YOUTH FOCUS GROUP DISCUSSION INFORMATION: City: Career Center: Date: Interviewer Name: Primary Notetaker Name: FOCUS GROUP DISCUSSION SIGN-IN SHEET: Name: Age: Sex: Have you completed an informed consent form? Have you completed a photo permission form? ☐Male ☐Female ☐ Yes ☐ No ☐ Yes ☐ No ☐Male ☐Female ☐ Yes ☐ No ☐ Yes ☐ No ☐Male ☐Female ☐ Yes ☐ No ☐ Yes ☐ No ☐Male ☐Female ☐ Yes ☐ No ☐ Yes ☐ No Informed Consent Protocol Use the informed consent procedures: read informed consent script and ask KII if they agree to participate. Initials of evaluator to indicate receipt of verbal consent: _____________________ Date: _________________________ This focus group discussion guide has been prepared as part of a broader evaluation commissioned by the United States Agency for International Development (USAID) for the midterm performance evaluation of the USAID Career Center in Morocco. The purpose of this mid-term performance evaluation is to analyze the performance of the USAID Career Center at the midway point and to make action-oriented, evidence-based recommendations to improve ongoing and future programming. Responses during this focus group will not be directly attributed to any individual in the final report. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 146 Before starting with the focus group questions, could you each tell us your name, your age, and the career center activities in which you have participated, and when? 1. What is your general impression of the Career Center Program? Probing Sub-Questions: • Was this something you were expecting to receive as you registered with the University/Vocational Training Center? 2. What if anything has changed about how prepared you feel for the world of work? Please explain Probing Sub-Questions: • Could you tell us an example or anecdote that would help us to better understand your point of view? 3. What are the services, workshops or training activities that you considered to be most useful? Why? 4. What are the services, workshops or training activities that you think were less useful? Why? 5. To what extent, if at all, has your participation in Career center activities changed how you approach your job search? 6. What are the qualities, knowledge and abilities of a good career counselor? To what extent do the career center personnel you have interacted with have these qualities, knowledge and abilities? What are their strengths and weaknesses? 7. To what extent do you believe career center services and activities will influence your ability to get a good job and a better life? Why? 8. To what extent have Career Center activities influenced your ideas about what kind of job or profession you would like to do in the future? 9. Do you have any suggestions for how Career Center activities might be improved? 10. For the Youth Ambassadors: can you tell us what are the main challenges that you have been facing when trying to engage other youth in the Career Center activities? 11. For the Youth Ambassadors: what strategies worked better in engaging other youth? Were the strategies different for girls and boys? 12. Do you have any other comments that you would like to share about the Career Center activities? Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 147 MIDTERM PERFORMANCE EVALUATION OF THE USAID CAREER CENTER FOCUS GROUP DISCUSSION GUIDE, TOT FACULTY FOCUS GROUP DISCUSSION INFORMATION: City: Career Center: Date: Interviewer Name: Primary Notetaker Name: FOCUS GROUP DISCUSSION SIGN-IN SHEET: Name: Sex: Have you completed an informed consent form? Have you completed a photo permission form? ☐Male ☐Female ☐ Yes ☐ No ☐ Yes ☐ No ☐Male ☐Female ☐ Yes ☐ No ☐ Yes ☐ No ☐Male ☐Female ☐ Yes ☐ No ☐ Yes ☐ No ☐Male ☐Female ☐ Yes ☐ No ☐ Yes ☐ No Informed Consent Protocol Use the informed consent procedures: read informed consent script and ask KII if they agree to participate. Initials of evaluator to indicate receipt of verbal consent: _____________________ Date: _________________________ This focus group discussion guide has been prepared as part of a broader evaluation commissioned by the United States Agency for International Development (USAID) for the midterm performance evaluation of the USAID Career Center in Morocco. The purpose of this mid-term performance evaluation is to analyze the performance of the USAID Career Center at the midway point and to make action-oriented, evidence-based recommendations to improve ongoing and future programming. Responses during this focus group will not be directly attributed to any individual in the final report. Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 148 1. What is your overall impression of the Training of Trainers Workshop that you attended? 2. How would you qualify the facilitation techniques used by the trainers? 3. From your point of view, what were the inputs or activities during the Training Workshop that you feel were going to be more difficult to apply? Why? 4. By the end of the training workshop, did you have any unresolved concerns on providing the training to your students? What kind of concerns? 5. How comfortable are you now in using with your students the training materials developed by the Career Center program? 6. As a result of the training, to what extent do you feel that you are better prepared to run interactive and engaging training sessions with your students? 7. If you have used the training materials with your students, which training modules in particular do you think are most effective and why? 8. Are there any training modules that you feel are less effective with your students or not received so well by them? What makes you think that? 9. Thinking of the students with whom you have worked, have you observed a difference or an evolution in their behavior, attitude or way of thinking? In what way or what difference/evolution have you noticed? 10. Have you received any follow-up support (one-on-one coaching, refresher training, peer coaching, etc.) as you started to train your students? 11. To what extent has this contributed to improvements in your delivery of soft skill training?\ Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 149 ANNEX 10: EVALUATION TEAM, CONFLICT OF INTEREST DISCLOSURE FORMS Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 150 Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 151 Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 152 Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 153 Mid-term Performance Evaluation of USAID Career Center, Final Evaluation Report 154