SCHOLARSHIPS AND TRAININGS FOR EGYPTIAN PROFESSIONALS ACTIVITY (STEP) END OF PROJECT PERFORMANCE EVALUATION Final Report July 14, 2019 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Hayley Alexander, Amal Nasr-Allah, Doaa Abdelaal, Soheir El Sherif, Youmna Khalil, Ehab Sakr and Mohamed Salah with support from Ahmed Gabr, Aliaa Abdella and Adriana Abreu of The QED Group. SCHOLARSHIPS AND TRAINING FOR EGYPTIAN PROFESSIONALS ACTIVITY (STEP) END-OF-PROJECT PERFORMANCE EVALUATION USAID/Egypt Office of Education and Health July 14, 2019 Task Order AID-263-I-15-00001/72026319F00006 Submitted by: The QED Group, LLC 1820 N. Fort Myer Drive, Suite 700 Arlington, VA 22209, USA Tel.: +1. 703.678.4700 www.qedgroupllc.com Egypt Office: The QED Group, LLC 1A Nadi El Etisalat off Ellaselky Street, New Maadi, 11435, Cairo, Egypt Office: +20 (0) 2 25226697 PHOTO CAPTION: STEP local partners and scholars DISCLAIMER: The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ACKNOWLEDGMENTS The Services to Improve Performance Management, Enhance Learning and Evaluation (SIMPLE) fielded a team tasked with conducting the end-of-program performance evaluation for USAID/Egypt’s Scholarships and Training for Egyptian Professionals Activity (STEP). The evaluation team thanks the organizations and individuals who contributed to our understanding of this project. In particular, we wish to express our appreciation to the following institutions and people for giving their cooperation and time to the data collection team: The Ministry of Education (MOE); the Ministry of Health and Population (MOHP); the Ministry of Higher Education and Scientific Research (MOHESR); the Career Development Consultancy (CDC); CID Consulting; Nahdet El Mahroussa; Keys to Effective Learning; AFS Egypt; Education for Employment (EFE); The American University in Cairo (AUC); STEP’s former Chief of Party (CoP); the Master of Business Administration (MBA) and Science, Technology, Engineering and Mathematics (STEM) scholars who participated in this research; and the United States Agency for International Development (USAID) Egypt Contracts Officer Representative (COR) for STEP and the USAID/Egypt program and technical offices. Each of these organizations and individuals provided the evaluation team with valuable feedback and support. III | END-OF-PROJECT PERFORMANCE EVALUATION OF STEP USAID.GOV CONTENTS ACKNOWLEDGMENTS II ACRONYMS & ABBREVIATIONS VIII BIOGRAPHIES IX PREFACE XI EXECUTIVE SUMMARY XII INTRODUCTION XII EVALUATION PURPOSE AND QUESTIONS XII DATA COLLECTION METHODOLOGY XIII DATA COLLECTION LIMITATIONS AND CHALLENGES XIII KEY CONCLUSIONS AND RECOMMENDATIONS XIV LESSONS LEARNED XIX INTRODUCTION 1 EVALUATION PURPOSE AND QUESTIONS 1 COUNTRY CONTEXT AND PROJECT BACKGROUND 2 RESEARCH METHODOLOGY 3 DATA COLLECTION METHODOLOGY 3 SAMPLING, DATA SOURCES, AND RESPONSE RATES 5 DATA ANALYSIS AND QUALITY ASSURANCE 5 DATA COLLECTION LIMITATIONS AND CHALLENGES 5 FINDINGS, CONCLUSIONS & RECOMMENDATIONS 7 QUESTION 1A. TO WHAT EXTENT DOES STEP THROUGH THE SCHOLARSHIPS COMPONENT CONTRIBUTE TO THE ACHIEVEMENT OF USAID/EGYPT’S STRATEGIC DOS, WHAT ARE THE SUCCESS FACTORS? 7 QUESTION 1B. TO WHAT EXTENT DOES STEP—THROUGH THE SCHOLARSHIPS COMPONENT— CONTRIBUTE TO THE PRIORITY NEEDS OF EGYPT’S KEY GOVERNMENT, ACADEMIC, AND PRIVATE SECTOR ORGANIZATIONS? WHAT ARE THE SUCCESS FACTORS? 11 QUESTION 2A. TO WHAT EXTENT DOES THE BENEFICIARY SELECTION PROCESS MEET THE REQUIREMENTS AGREED WITH USAID? 13 QUESTION 2B. TO WHAT EXTENT DO APPLICANTS MEET THE CRITERIA FOR SCHOLARSHIP AWARDS? 17 QUESTION 3A. WHAT IS THE RETURN ON INVESTMENT OF STEP? 18 QUESTION 3B. WHAT ARE THE REASONS FOR NON-RETURNEES? 19 QUESTION 4A. ARE STEP ALUMNI PRODUCTIVELY EMPLOYED IN FIELDS RELATED TO THEIR ACADEMIC PREPARATION? 20 QUESTION 4B. TO WHAT EXTENT ARE STEP ALUMNI POSITIONED TO TAKE POSITIONS OF LEADERSHIP IN ACADEMIA, GOVERNMENT, INDUSTRY, AND BUSINESS? 24 QUESTION 5. TO WHAT EXTENT DOES THE TECHNICAL ASSISTANCE (TA) COMPONENT SUPPLEMENT THE ACTIVITIES AND PROJECTS OF OTHER TECHNICAL OFFICES PLUS HELP ACHIEVE USAID/EGYPT’S STRATEGIC DOS AND CONTRIBUTE TO EGYPT’S GOVERNMENT, ACADEMIC, AND PRIVATE-SECTOR PRIORITIES? 26 SUMMARY CONCLUSION 30 USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | IV ANNEXES 32 ANNEX I: EVALUATION SCOPE OF WORK 33 ANNEX II: TABLES AND GRAPHS 52 ANNEX III: EVALUATION METHODS, LIMITATIONS, ISSUES 81 ANNEX IV: WORK PLAN TIMELINE 85 ANNEX V: REPORT REFERENCES 86 ANNEX VI: RETURN ON INVESTMENT ROI METHODOLOGY 88 ANNEX VII-A: DATA COLLECTION TOOLS (ENGLISH) 93 ANNEX VII-B: DATA COLLECTION TOOLS (ARABIC) 119 ANNEX VIII: PROJECT DOCUMENTS REVIEWED 144 ANNEX IX: FINDINGS, CONCLUSIONS, RECOMMENDATIONS 150 LIST OF TABLES Table 1: Number of MBA Scholars Disaggregated by Cohort Number and Return Status .................................... 53 Table 2: Distribution of MBA Scholars Disaggregated by their Current Residence Governorates....................... 53 Table 3: Distribution of MBA Scholars Disaggregated by their Previous Employers’ Sector.................................. 54 Table 4: Distribution of MBA Scholars Disaggregated by their Age............................................................................. 55 Table 5: Distribution of MBA Scholars Disaggregated by their Scholarship Location in United States................ 56 Table 6: Distribution of MBA Scholars Disaggregated by their Area of Study/Specialization ................................. 56 Table 7: Average GPA for MBA Scholars Disaggregated by Residence Governorates, and Cohort Numbers.. 57 Table 8: Number of Non-Returnees from MBA Scholars Disaggregated by Scholarship Location in United States.......................................................................................................................................................................................... 57 Table 9: Number of Non-Returnees from MBA Scholars Disaggregated by their Age............................................ 58 Table 10: Number of Non-Returnees from MBA Scholars Disaggregated by their Residence Governorates ... 58 Table 11: Number of Non-Returnees from MBA Scholars Disaggregated by their Area of Specialization ........ 58 Table 12: Number of Non-Returnees from MBA Scholars Disaggregated by their Previous Employer Sector. 58 Table 13: Number of Non-Returnees from MBA Scholars Disaggregated by their College in United States..... 59 Table 14: Number of STEM Scholars Disaggregated by Cohort Number................................................................... 60 Table 15: Distribution of STEM Scholars Disaggregated by their Current Residence Governorates................... 60 Table 16: Distribution of STEM Scholars Disaggregated by their School Types........................................................ 61 V | END-OF-PROJECT PERFORMANCE EVALUATION OF STEP USAID.GOV Table 17: Distribution of STEM Scholars Disaggregated by their Age ......................................................................... 62 Table 18: Distribution of STEM Scholars Disaggregated by their College in United States.................................... 62 Table 19: Distribution of STEM Scholars Disaggregated by their Area of Study/Specialization.............................. 63 Table 20: Average GPA for STEM Scholars Disaggregated by Residence Governorates, and Cohort Numbers63 Table 21: Distribution of MBA Scholars Disaggregated by Cohort, Age, Birth Governorate, Current Governorate, Marital Status, Employment Status, and Average Cumulative GPA.................................................... 64 Table 22: Distribution of STEM Scholars Disaggregated by Cohort, Age, Birth Governorate, Current Governorate, Marital Status, and Average Cumulative GPA ......................................................................................... 65 Table 23: Distribution of STEP Scholars (MBA and STEM) by Main Source of Learning about the Program...... 67 Table 24: Average Rate* for MBA Scholars Opinion about the Program ................................................................... 67 Table 25: Average Rate* for STEM Scholars Opinion about the Program................................................................. 67 Table 26: Average Rate* of Usefulness for MBA Scholars about STEP Activities, Training and Services............. 68 Table 27:. Average Rate* of Usefulness for STEM Scholars about STEP Activities, Training and Services.......... 68 Table 28: Average Rate* for MBA Scholars about STEP Activities, Training and Services..................................... 69 Table 29: Average Rate* for STEM Scholars about STEP Activities, Training and Services..................................... 69 Table 30: Average Rank* of Usefulness for MBA Scholars’ Personal Growth ........................................................... 69 Table 31: Average Rank* of Usefulness for STEM Scholars’ Personal Growth......................................................... 70 Table 32: Average Rank* of Usefulness for MBA Scholars’ Professional Development.......................................... 70 Table 33: Average Rank* of Usefulness for STEM Scholars’ Professional Development........................................ 70 Table 34: Average Rank* of Usefulness for MBA Scholars by Program Support in Finding or Attempting to Find a Job ................................................................................................................................................................................... 70 Table 35: Average Rank* of Usefulness for STEM Scholars by Program Support in Finding or Attempting to Find a Job ................................................................................................................................................................................... 71 Table 36: Distribution of STEP Scholars (MBA and STEM) by Their Receive the Stipend from STEP.................. 71 Table 37: Mean Amount of Stipend per Month (in USD) that STEP Scholars (MBA and STEM) Receive............ 71 Table 38: Distribution of STEP Scholars (MBA and STEM) by Additional Costs Paid (in USD)............................. 71 Table 39: Average Rate* of Usefulness MBA Scholars’ Academic Program in the US ............................................. 71 Table 40: Distribution of STEP Scholars’ (MBA and STEM) by their University Similar Activities......................... 72 USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | VI Table 41: Distribution of STEP Scholars’ (MBA and STEM) by Activities Provided by both STEP and their University................................................................................................................................................................................... 72 Table 42: Distribution of STEP Scholars’ (MBA and STEM) by Activities Not Provided by their University and Needed STEP’s Support ......................................................................................................................................................... 72 Table 43: Distribution of MBA Scholars’ Sources that Consider the Most Important Support System............... 72 Table 44: Distribution of STEM Scholars’ Sources that Consider the Most Important Support System ............. 73 Table 45: Statistics for Number of Internships MBA Scholars Completed During and After STEP Program ..... 73 Table 46: Statistics for Number of Internships that STEM Scholars Completed During and After STEP Program ..................................................................................................................................................................................................... 73 Table 47: Distribution of Internships that MBA Scholars’ by their Field of Specialization....................................... 73 Table 48: Distribution of Internships that STEM Scholars’ by their Field of Specialization ..................................... 74 Table 49: Statistics for the Duration of Internships (in Months) MBA Scholars Completed During and After STEP Program........................................................................................................................................................................... 74 Table 50: Statistics for the Duration of Internships (in Months) STEM Scholars Completed During and After STEP Program........................................................................................................................................................................... 74 Table 51: Distribution of Usefulness for MBA Scholars’ Internships Participated in their Professional Development ............................................................................................................................................................................ 74 Table 52: Distribution of Usefulness for STEM Scholars’ Internships Participated in their Professional Development ............................................................................................................................................................................ 75 Table 53: Distribution of STEP Scholars (MBA and STEM) by their Support to Get the STEP Internship .......... 75 Table 54: Average Rank* for MBA Scholars’ for their Future Career Development ............................................... 75 Table 55: Average Rank* for STEM Scholars’ for their Future Career Development.............................................. 76 Table 56: Distribution of MBA Scholars (MBA and STEM) according to their Challenges to Find a Job Related to their Field of Work aggregated by Current and Birth Governorates..................................................................... 76 Table 57: Distribution of STEM Scholars (MBA and STEM) according to their Challenges to Find a Job Related to their Field of Work aggregated by Current and Birth Governorates..................................................................... 77 Table 58: Distribution of Currently Employed MBA Scholars, according to their Job relevance to their Area of Study........................................................................................................................................................................................... 77 Table 59: Distribution of Currently Employed MBA Scholars, who have the job with Same Organization Prior to Traveling ............................................................................................................................................................................... 77 Table 60: Distribution of Currently Employed MBA Scholars, who have the job with Same Organization Prior to Traveling in Same Position................................................................................................................................................ 77 VII | END-OF-PROJECT PERFORMANCE EVALUATION OF STEP USAID.GOV Table 61: Distribution of Currently Employed MBA Scholars, according to Reasons behind Joining their Current Organization ............................................................................................................................................................. 78 Table 62: Average Duration of Currently Employed MBA Scholars, between Graduation and Job Placement (in completed months) ................................................................................................................................................................. 78 Table 63: Distribution of Currently Employed MBA Scholars, According to Number of Job Applications They Submitted................................................................................................................................................................................... 78 Table 64: Distribution of Currently Employed MBA Scholars, According to Number of Interviews They Completed................................................................................................................................................................................. 79 Table 65: Distribution of Currently Employed MBA Scholars, According to Number of Job Offers They Refused....................................................................................................................................................................................... 79 Table 66: Distribution of Currently Employed MBA Scholars, According to Reasons Behind Refusing Job Offers Aggregated by Number of Job Offers Refused.................................................................................................................. 79 Table 67: Distribution of Currently Employed MBA Scholars, According to Gross Monthly Salary (in EGP) They are Currently Receiving ............................................................................................................................................... 80 Table 68: Distribution of Currently Employed MBA Scholars, According to Gross Monthly Salary (in EGP) They Used to Receive Prior to Traveling to the US ........................................................................................................ 80 Table 69: Distribution of Currently Employed MBA Scholars, according to their Believe about Their Salary Comparison with Other MBA Holders.............................................................................................................................. 80 Table 70: Distribution of Currently Employed MBA Scholars, According Difference in Salary Scale Between Males and Females at Their Present Company.................................................................................................................. 80 Table 71: Target stakeholders and data collection tools................................................................................................. 81 Table 72: Data sources, collection methods, and response rates achieved against .................................................. 83 Table 73: MBA Scholarships Program - STEM Scholarships Progra …………………………………………...89 Table 74: STEP/STEM Scholarships ProgramCost-Benefit Analysis/ROI Calculations…………………………111 Table 75: STEP/MBA Scholarships Program Cost-Benefit Analysis/ROI Calculations…………………………91 LIST OF FIGURES Figure 1: Percent Distribution of MBA Scholars Disaggregated by their Current Residence Governorates..... 54 Figure 2: Distribution of MBA Scholars Disaggregated by their Previous Employers’ Sector............................... 55 Figure 3: Percent Distribution of STEM Scholars Disaggregated by their Current Residence Governorates.. 61 Figure 4: Percent Distribution of STEM Scholars Disaggregated by their School Types.......................................... 61 USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | VIII ACRONYMS & ABBREVIATIONS AUC The American University in Cairo CDC Career Development Consultancy CoP Chief of Party COR Contracts Officer Representative CSO Civil Society Organization CSR Corporate Social Responsibility DO Development Objective DQA Data Quality Assessment EFE Education for Employment EGP Egyptian Pounds EQ Evaluation Question EU European Union F2F Face to face GMAT Graduate Management Admission Test GPA Grade Point Average HEI Higher Education Initiative IIE Institute of International Education IP Implementing Partner KII Key Informant Interview LSP Local Scholarships Program MBA Master of Business Administration MBAs Holders of Master of Business Administration degree MOE Ministry of Education MOHESR Ministry of Higher Education and Scientific Research MOHP Ministry of Health and Population M&E Monitoring and Evaluation NGO Non-Governmental Organizations OEH USAID Office of Education and Health PIRS Performance Indicator Reference Sheets PMP Performance Management Plan ROI Return on Investment SIMPLE Services to Improve Performance Management, Enhance Learning and Evaluation STEM Science, technology, engineering, and mathematics STEMs Holders of science, technology, engineering, or mathematics degrees STEP Scholarships and Training for Egyptian Professionals SoW Scope of work TA Technical Assistance TOEFL Test of English as a Foreign Language ToT Training of Trainers UN United Nations UPP University Preparation Program USA United States of America USAID United States Agency for International Development IX | END-OF-PROJECT PERFORMANCE EVALUATION OF STEP USAID.GOV BIOGRAPHIES Team Leader Hayley Alexander is a private sector development specialist who holds an MBA and has 30 years of experience, 18 of them outside the United States managing economic development programs for Deloitte Touche Emerging Markets (later Cardno Emerging Markets), often in a Chief of Party or Team Leader capacity. He has engaged in long-term project postings/lived in seven countries, with work experience in 15 throughout Eastern Europe, Africa, The Middle East and Southeast Asia. He became an independent consultant in 2013, performing short-term technical assistance assignments in private sector development, institutional strengthening and improved competitiveness programs. Recent experience has been centered on SME competitiveness and value chains, enabling environment assessments, program evaluations, BDS & institutional capacity development, and regulatory reform analyses. Amal NasraAllah is a rigorous academic and professional with expertise in results-based management and evaluation, indicators development, performance change measurement, outcome and impact mapping and evaluation, and utilization of evaluation findings. She has had extensive work experience with donors’ programs and multi-stakeholder projects in different sectors; namely Education, Employability, Integrated Development, Small and Medium Enterprises, Entrepreneurship, Youth Empowerment, Child Protection, Civil Society, Media Development, Trade Facilitation and Sustainable Development. She has conducted consultancy services for a number of notable donor programs for clients such as USAID, UNDP, UNICEF, GIZ, IDRC, GAC, and the Italian Cooperation. Doaa Abdelaal is specialized in development projects evaluation, research and gender analysis and training. Since 2010, she worked as a gender consultant with several organizations in Egypt and regionally in the Middle East and Africa as UNDP, UN Women, JICA, Urgent Action Fund-Africa, Oxfam, HIVOS, and the International Water Management Institute. As an evaluator she was a team member in the midterm evaluation of the LOTUS scholarship program, the end of term evaluation of the LEAD scholarship program and the end of term evaluation for the STEM schools’ program; all funded by USAID. She also led the final evaluation of the livelihood program of the Catholic Relief Services in Egypt, the baseline assessment of the Gender Sensitive Citizen Charter project implemented by CARE International in Egypt. In 2018, she worked with the Association for Women Rights in Development (AWID) to develop their Theory of Change. Doaa has an MA in political science from Cairo University. Soheir El Sherif is an Economist with 35-year experience in the fields of research, lecturing and consulting related to project evaluation. She holds a Ph.D in Project Evaluation based on macro-level cost-benefit analysis, and has been certified by UNIDO as an advisor and trainer in project evaluation, using the automated COMFAR program. She has participated in the evaluation of projects funded and sponsored by bilateral and multilateral development organizations, including: USAID, CIDA, DFID, KfW, EU, UNDP, UNIDO and the World Bank. Sectors of the projects evaluated demonstrate a broad spectrum, including: education, health, manufacturing, urban development, tourism and infrastructure (water, wastewater, energy, and airports). Youmna Khalil is a development practitioner and specialist. She has more than 15 years of experience in working for regional and international development organizations in the MENA region. For the last 10 years she has been working as M&E specialist and consultant within different social￾economic projects, such as rural development, education, youth, economic empowerment for women and girls, rural communities, community development, financial education, microfinance, health, housing rehabilitation/cultural heritage, vocational training and crafts development. USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | X Youmna demonstrates solid background in managing, monitoring, assessing and evaluating projects activities by using and applying different quantitative and qualitative research methods and high analytical and writing skills especially in examining policies and practices in local and global contexts. She is currently teaching Monitoring and Evaluation at the American University in Cairo. She conducted multiple regional and national baseline studies and midterm and end line projects’ evaluations in different development areas and she has wide experience with international organizations such as USAID, CIDA, GIZ, Embassy of Finland, Plan International, IFAD and the US State Department. She also participated in writing different donors’ proposals and reports where she acquired strong knowledge on donor reporting requirement. She has worked with both governmental and international organizations, CSOs and local communities where she Led and managed teams in different projects and contexts. Ehab Sakr, Statistician, is an Assistant Lecturer in the Department of Demography and Bio-statistics, Institute of Statistical Studies and Research, Cairo University in Egypt, and holds a Masters degree in Statistics from the Faculty of Economics and Political Science. He is pursuing his Ph.D. at the Institute of Statistical Studies and Research. He has more than15 years of experience in research, working at both the national level (e.g., National Population Council and the Industrial Modernization Centre) and the international level (e.g., UNFPA, DHS, and the ILO). He contributes to the design of questionnaires and provided guidance on the quality of data collection, data entry, and data analysis using statistical packages including STATA and SPSS. He has experience in software and web development, as well as in the digital marketing field. Mohamed Salah is a manager in marketing and product development, despite his education in Law, he pursued his passion in PR and worked in major events organizing like Cairo Film Festival 2010. Then he followed his passion that led to finishing his Professional Marketing Diploma, seeking a new career path upon his adventurous nature. He is always seeking to gain new experiences and cherish them with knowledge. XI | END-OF-PROJECT PERFORMANCE EVALUATION OF STEP USAID.GOV PREFACE The STEP scope of work does not assign high-level objectives specifically to the scholarships component or the technical assistance (TA) component; the evaluation team therefore evaluated high-level objectives across both components. For clarity, the TA component analysis is moved to section EQ5, leaving sections EQ1 through EQ4 largely dedicated to the scholarships component. All recommendations are followed with the words “design” or “implementation,” which indicate whether the recommendation addresses a project design issue or an implementation issue. Some findings and conclusions are repeated if they pertain to multiple evaluation questions. Also, related findings are grouped together (without spaces between them). Three of the five EQs (EQ1, EQ2, and EQ3) were further split into two questions, because addressing them required different data sources and distinct analyses. USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | XII EXECUTIVE SUMMARY INTRODUCTION In 2014, the $250-million Higher Education Initiative (HEI) Bilateral Agreement was signed to strengthen institutions of higher learning and develop lasting partnerships among the private sector, U.S. universities, and Egyptian institutions. The ultimate aim of the initiative was to strengthen Egypt’s tertiary institutions and prepare its youth, particularly women, to effectively contribute to the country’s economic growth and development. The project targets strategic fields such as agriculture, the environment, climate change, and water management; business; science, technology, engineering and math (STEM); as well as particular fields such as nursing, where Egypt has a large human-capacity shortage. Since Egyptian women are underrepresented in study-abroad programs, recruitment for the initiative sought to correct this imbalance and focused on reaching women from disadvantaged communities. To expand Egypt’s cadre of professionals trained in critical fields at respected universities, HEI would support a robust program of local and U.S. scholarships. The $43,912,257 STEP activity ($35,385,000 obligated) was designed to support these objectives and operated from January 2014 through December 2018. STEP was designed with two primary components: 1) Scholarships for economically disadvantaged women to attend US universities, either for four-year STEM degrees or two-year MBA degrees, to advance cultural understanding and strengthen technical, managerial, and leadership capabilities and 2) Technical assistance for capacity building among the Egyptian public, private and academic sectors in support of other USAID/Egypt technical office objectives. INTENDED AUDIENCE AND USE OF EVALUATION RESULTS The primary audience for the evaluation report will be USAID/Egypt, specifically the office of Education and Health and USAID/Egypt senior management. The secondary audience includes the Government of Egypt's Ministry of Higher Education and Scientific Research (MOHESR). The purpose of the Scholarships and Training for Egyptian Professionals (STEP) evaluation is to help USAID/Egypt improve the performance of similar current and future activities, and their contribution to development objectives. The evaluation will examine the effectiveness of the interventions under STEP for both the scholarships component (how it impacted the scholars' lives) and the technical assistance component (how it supported USAID technical offices to achieve their goals). EVALUATION PURPOSE AND QUESTIONS The objective of the performance evaluation is to help USAID/Egypt improve the performance of similar current and future activities and support USAID/Egypt’s development objectives by examining the effectiveness of the activity interventions for two primary components: (1) scholarships and how they impacted the scholars’ lives and (2) technical assistance and how it supported USAID technical offices to achieve their goals. The evaluation specifically addresses five evaluation questions. EQ1A – To what extent does STEP—through the scholarships component as well as the technical assistance component—contribute to the achievement of USAID/Egypt's strategic DOs? What are the critical success factors of STEP? EQ1B – To what extent does STEP—through the scholarships component as well as the technical assistance component—contribute to the priority needs of Egypt’s key government, academic, and private sector organizations? What are the critical success factors of STEP? EQ2A – To what extent does the beneficiary selection process meet the requirements agreed with USAID? EQ2B – To what extent do applicants meet the criteria for scholarship awards? EQ3A – What is the return on investment of STEP? EQ3B – What are the reasons for non-returnees? EQ4A – Are STEP alumni productively employed in fields related to their academic preparation? XIII | END-OF-PROJECT PERFORMANCE EVALUATION OF STEP USAID.GOV EQ4B – To what extent are STEP alumni positioned to take positions of leadership in academia, government, industry, and business? EQ5 – To what extent does the technical assistance component supplement the activities and projects of other technical offices? DATA COLLECTION METHODOLOGY The end-of-project STEP performance evaluation was implemented as a descriptive study using a mixed-methods data collection approach, including both qualitative and quantitative research and statistical analysis of online quantitative questionnaires addressing each evaluation question. Eight distinct categories of stakeholders and beneficiaries were identified; individual protocols and survey instruments were created for each. Forty-two key STEP stakeholders and 158 scholar beneficiaries, including six non-returnees, were included in the population of respondents sought. Two teams working in parallel targeted 82 percent of MBA scholars and 87 percent of STEM scholars identified through purposive sampling, eventually shifting to 100 percent attempted contacts. Cooperating ministries (the MOE, MOHP, and MOHESR), USAID technical offices, and STEP subcontracted local partners were all targeted at a 100-percent response rate from the outset. Three of six non￾returnee scholars were sought, and 21 of an unknown number of employers were also targeted. Respondent Type Contact Attempts by Phone and Email KIIs Completed F2F & Telephone Online Completed Target % Completed MBAs 207 (t) 383(e) 46 26 (1 par /4 inc) 74 (of 90) 97% STEMs 253 (e) 11 21 (4 par / 7 inc) 54 (of 62) 59% Ministries 4 4 4 (of 4) 100% Local Partners 6 6 6 (of 6) 100% Employers 15 4 21 (of unknown) 19% Non- Returnees 6 0 1 3 (of 6) 33% USAID 4 5 80% TOTALs 874 75 48 (t) = Telephone Calls (e) = Email (inc) = Incomplete (par) = partially complete DATA COLLECTION LIMITATIONS AND CHALLENGES 1. STEM scholars’ response rates were 59 percent of the targeted level. This is likely due to: 1) STEMs not knowing or trusting the nature of our intent, as there was no Implementing Partner (IP) introduction made, 2) STEMs not receiving our emails due to spam filtering, and 3) STEMs being too occupied with final exams to respond to the evaluation request. Mitigation efforts included: four email contact attempts per scholar by the evaluation team, two email contact attempts by USAID, appeals to STEMs who had already completed Key Informant Interviews (KIIs) to notify other STEM scholars of the importance of their feedback, and giving them the option of filling out an online questionnaire if they did not have time to speak with the team by telephone. While the combined telephone and online response rates did not achieve their targets, these efforts brought the numbers to a level sufficient for substantive analysis. 2. Just four MBA employers were reached. This is likely the result of scholars’ reticence to allow us to speak with their employers, as most were relatively new in their jobs and had not yet developed strong relationships. This category of respondent was added as a source of contextual information TABLE 1 - DATA SOURCES, COLLECTION METHODS, AND RESPONSE RATES ACHIEVED AGAINST TARGETS USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | XIV about MBA-profile attractiveness and was never required for data validity. The few responses therefore remain useful. 3. As of the date of this evaluation, none of the STEM scholars had yet completed their four-year degrees and, therefore, had not returned to Egypt. This makes the evaluation premature to gauge program outcomes for STEM students pertaining to leadership, employment, or the likelihood they will remain in Egypt and contribute to high-level development objectives. 4. Return on Investment (ROI) calculations are only possible at a high level (total scholar component ROI), because detailed cost data per category (MBA or STEM), per university, or according to other parameters were not provided. 5. The team had no access to a comprehensive STEP-scholar database, which exists with numerous sortable fields, as referenced in the April 2017 Semi-Annual Report, “The database is tailored to project requirements and contains detailed participant data covering all phases, including participant application, selection, University Preparation Program (UPP), extracurricular workshops, academic standing, and internships.” The evaluation team requested access to detailed information files on each scholar from program inception during the Team Planning Meeting, which we later learned existed in database form but was never fully released. The information it contained could have enhanced our understanding of scholar specifics, such as employment against various isolated factors, e.g., GPA or the type of degree attained. The former CoP was very helpful, yet the team was only able to speak with her near the end of the fifth week of the evaluation. While access to the CoP or the IP were never promised, the information they possessed would have been extremely useful, especially at the desk-review stage, and would have enhanced the assessment team’s efficiency. 6. The assessment team requested randomly selected scholar application packets—15 MBAs rejected, and 15 MBAs selected, 15 STEMs rejected, and 15 STEMs accepted—as the best possible source of information to determine how individual scholars were evaluated and chosen. Random selection of these packets was important to ensure a representative cross-section of applicants was obtained; however, there is no clear indication this was done. KEY CONCLUSIONS AND RECOMMENDATIONS QUESTION 1A. TO WHAT EXTENT DOES STEP, THROUGH THE SCHOLARSHIPS COMPONENT, CONTRIBUTE TO THE ACHIEVEMENT OF USAID/EGYPT’S STRATEGIC DEVELOPMENT OBJECTIVES? WHAT ARE THE SUCCESS FACTORS? One-hundred and fifty-eight Egyptian women underwent a life-changing cultural and academic experience with a clear impact on their personal and professional development. The experience of studying in multicultural settings was as important as the degree itself and led to increased self￾confidence, a sense of independence, and the ability to take initiative. Also, the five major scholarship component outcomes have been largely achieved: 90 culturally aware MBA graduates with enhanced managerial and leadership skills returned to Egypt, and 62 skilled and culturally aware STEMs expected to return by the end of May 2020. A significant missed opportunity involved a lack of cohesive network building among returning alumni (a separate issue from the lack of networking with employers). MBA alumni represent 90 individuals, each attempting to influence a small sphere of friends, family, and colleagues. Other than organizing networking events and creating a Facebook page (an action the alumni themselves also took credit for), STEP undertook no clear process or actions to actually establish a sustainable network— something significantly greater than conducting simple events or establishing a webpage. The result is an alumni group with virtually no leverage as a group to raise their profile and share their perspective with policymakers or civil-society and private-sector organizations. Recommendation – USAID should design and IPs implement future scholarship programs from the outset to include planning and activities that systematically link scholars into a cohesive network, with support continued at least six months after their return. Activities should include partially funded networking events, topical private-sector symposia, and meetings organized with XV | END-OF-PROJECT PERFORMANCE EVALUATION OF STEP USAID.GOV policymakers and influential NGOs. The six-month time frame would also be used to help fully integrate the alumni group into a previously agreed on host NGO or other suitable institution interested and capable of taking the alumni network under its wing. For STEP, the National Council for Women would have been a natural fit. Other parallel options, such as more systematically tying STEP alumni to Institute of International Education’s (IIE) alumni network would also be appropriate. Such activities need to start from day one of new scholarship programs, not wait until scholars are ready to return. We recommend USAID employ these design features into future scholarship activities (medium-term) and consider mitigation options for the existing MBA scholars (near-term). QUESTION 1B. TO WHAT EXTENT DOES STEP—THROUGH THE SCHOLARSHIPS COMPONENT—CONTRIBUTE TO THE PRIORITY NEEDS OF EGYPT’S KEY GOVERNMENT, ACADEMIC, AND PRIVATE-SECTOR ORGANIZATIONS? WHAT ARE THE SUCCESS FACTORS? STEP scholarships support the Knowledge and Innovation Pillar and the Education and Training Pillar mainly with regard to higher education, which aspires to invest in human capacity and innovation toward developing complete value chains from research to commercialization. The STEM scholars’ science, information technology and engineering skills are particularly noteworthy in terms of their potential to contribute to Egypt’s enhanced scientific research and innovation aims, while both STEMs and MBAs have undergone experiences that strongly developed human character (see supporting findings for questions 2A, 4A and 4B with positive scholar views concerning their US￾based academic experiences). STEP scholarships do not measurably support Egypt’s other Vision 2030 objectives in the short to medium term. Any institutional capacity building is within the realm of the TA component (see also question 5, which addresses STEP’s considerable TA component contributions to the Education and Training Pillar). Recommendation – STEP scholarship activities effectively supported Egyptian government priorities under the Vision 2030 Knowledge and Innovation Pillar, no further recommendations are proposed. QUESTION 2A. TO WHAT EXTENT DOES THE BENEFICIARY SELECTION PROCESS MEET THE REQUIREMENTS AGREED WITH USAID? Financial eligibility: The term “disadvantaged” is not applied according to a clearly defined rationale that ensures the most qualified applicants will be selected. It is evident some middle, even upper￾middle income applicants were able to gain acceptance, which indicates a problem with the consistency of the application process. Also, the negative connotation of “disadvantaged” needs consideration in actual practice because it may have served as a barrier to people with a well-developed sense of pride. Finally, while STEP did rely on a set of standard financial rubrics, some of them appear arbitrary for the purpose at hand and/or not easily verifiable. The selection process: The selection criteria and process rely on identifiable rubrics and cut-off scores but are not always consistently applied or, if exceptions were required, the reasons were not documented. The use of cut-off scores for two different language tests to select candidates within the same cohort, but only in some instances, is not clearly explained or documented. Acceptance of candidates from STEM and experimental schools may be perfectly rational, though these too raise questions about how needy these applicants were versus those without the resources and exposure to such educational opportunities. These inconsistencies appear to result in the acceptance of applicants from middle- to upper-middle-income families. Recommendation – A clearer definition of “disadvantaged” applicants should be agreed upon by USAID and its Implementing Partners, given that some potential leaders may be less needy but in certain cases more deserving of scholarships than those given extra points for living in rented flats or earning less than LE 1,000/month. We further propose that financial status be used as a screening USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | XVI measure after leadership skills are evaluated, not as a criterion for eligibility pass/fail. QUESTION 2B. TO WHAT EXTENT DO APPLICANTS MEET THE CRITERIA FOR SCHOLARSHIP AWARDS? Potential leaders: STEP scholars performed very strongly in their US programs, evidenced by high academic achievements, social integration and determination in seeking internships. This indicates strong candidates were selected. There is also some evidence of leadership and becoming involved in social/community affairs; however, not enough time has elapsed to properly gauge leadership accomplishments more broadly in practice. Career guidance: While activities such as career self-assessment and career counseling were provided to MBAs and STEMs—crucial to help ensure accepted applicants met later employment objectives— satisfaction rates with these services were low to moderate. Efforts to better respond to scholars’ career counseling needs could have been intensively applied from the moment of program acceptance, with more thought given to linking career counseling to field of study, university selection, and relevant job placement. Recommendation – We recommend future IPs strengthen and extend the career counseling function to ensure adequate time is allotted to aligning scholar interests, work experience, university selection, and necessary internships (and likely availability) to a realistic assessment of employment market demand. Then ensure scholar action plans have specific milestones for each step in their career development process. We also suggest encouraging alumni to help advise new program entrants. QUESTION 3A. WHAT IS THE RETURN ON INVESTMENT OF STEP? ROI based on per-scholar averages provided by USAID are calculated as 11 percent for MBA scholars and 7 percent for STEM scholars. Significant non-monetary scholarship benefits occurred: self-confidence, leadership skills, stronger self-initiative, managerial skills, technical skills, and cultural awareness. Recommendation – For detailed ROI calculations on future scholarship evaluations, we recommend USAID provide cost data classified by field of study and educational institution to allow for a more robust analysis. QUESTION 3B. WHAT ARE THE REASONS FOR NON-RETURNEES? STEP measures were sufficient to limit the number of non-returnees to approximately 6 percent, which is not excessive. MBA scholars interviewed believe the non-returnees had already made their decision before traveling, and there was nothing STEP could have done once the scholars were overseas. Focusing on the very needy, from conservative families, may augment cultural problems of separation. Recommendation – While the evaluation team has not identified any standard measures or benchmarks to identify applicants who might be susceptible to becoming non-returnees, further research is warranted. We recommend USAID direct a future IP to look into the possibility of designing a screening mechanism using demographic and psychographic (attitudes, aspirations and interests) markers common to non-returnees, then track the predictive success rate longitudinally. Naturally, any use of psychographics for screening would need to be carefully considered to prevent potential misuse or bias against applicants. QUESTION 4A. ARE STEP ALUMNI PRODUCTIVELY EMPLOYED IN FIELDS RELATED TO THEIR ACADEMIC PREPARATION? XVII | END-OF-PROJECT PERFORMANCE EVALUATION OF STEP USAID.GOV In the absence of a continuing record of MBA alumni employment, the most recent employment information received combined with responses to questionnaires led us to estimate over 50 percent of MBA scholars are employed after an average job search duration of six months, with approximately 75 percent employed in their field of study. Once back in Egypt, MBA scholars faced several challenges to find suitable employment. Being out of the job market for two and a half years, returning scholars needed more opportunities to increase work experience in their fields, as internships were often brief or not directly relevant to their career. They also needed more coaching from the earliest days aimed at managing expectations on their return. It does appear, however, that scholars returned with strong soft skills, which were positively recognized by the employers interviewed. Recommendation – For future scholarship programs of similar design, USAID should consider a sustainable, holistic cycle approach wherein IPs do the following: 1. At program start, review available labor market assessments to zero-in on specific labor demand as it relates to fields of study; undertake a new study if warranted. 2. Begin career counseling weeks before scholars select fields of study and universities, factoring in work experience, career-shift interests, and labor-market demand. 3. Starting from day one, have a program component focused on demand-side linkage building to strengthen internship opportunities and practical employment linkages, well before the scholars return. QUESTION 4B. TO WHAT EXTENT ARE STEP ALUMNI POSITIONED TO TAKE POSITIONS OF LEADERSHIP IN ACADEMIA, GOVERNMENT, INDUSTRY, AND BUSINESS? Scholars’ leadership and soft skills were further developed through their U.S. academic studies; these are the very factors that provide them with an edge over those who have not undergone such an experience. However, despite some strong success stories, insufficient time has elapsed to assess whether leadership is being demonstrated more broadly among MBA alumni during employment or community service in Egypt. Nor has enough time passed to determine the likelihood that STEM graduates will assume positions of responsibility within Egypt, as the evaluation is being conducted prior to the completion of their studies and return home. Recommendation – We recommend critical terms such as “leadership” be clearly defined by future scholarship program IPs, shared with each scholar at the application phase, then tracked against clear competency indicators throughout their studies for at least six months after their return. This would also help refine future selection rubrics. QUESTION 5. TO WHAT EXTENT DOES THE TECHNICAL ASSISTANCE (TA) COMPONENT SUPPLEMENT THE ACTIVITIES AND PROJECTS OF OTHER TECHNICAL OFFICES PLUS HELP ACHIEVE USAID/EGYPT'S STRATEGIC DOs AND CONTRIBUTE TO EGYPT’S GOVERNMENT, ACADEMIC, AND PRIVATE SECTOR PRIORITIES? The inability to achieve multiple high-level STEP objectives—most notably, institutional capacity building in academia, the private sector, and the government (the MOE being the exception)—raises a project design issue. Each of these are arguably stand-alone projects in their own right. Such multiple and overlapping objectives, especially for a project primarily tasked with educating scholars, is counterproductive and obscures programmatic focus. Technical assistance with the MOE did successfully address two capacity building tracks: enabling the ministry to effectively manage the adoption and monitoring of new K-3 Early Grade Learning curricula and providing Training of Trainers (ToT) and all curricula for the teacher training itself. This was a highly leveraged capacity-building program with strong chances for sustainability, and it USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | XVIII was well appreciated by the MOE. Key success factors involved STEP’s start-to-finish integrated approach, which addressed the complete cycle for sustainability. Three of six local partners implemented STEP’s TA component as a stand-alone activity with no relation to the scholarships component. Without any cross-fertilization among local partners, which may have otherwise enhanced support of the scholarship component, practical means to improve job-placement activities, such as developing relationships with progressive Egyptian employers, were never undertaken. This represents a significant missed opportunity for STEP and its beneficiaries. A similar process is often undertaken in private-sector value-chain programs, which coalesce large buyers and small subcontractors by working both sides of the equation simultaneously to sensitize them to each other’s needs. Recommendation – We recommend USAID not apply multiple, overlapping, high-level objectives to future scholarship programs, which require the addition of a TA component and a distinct set of objectives largely unrelated to those of the scholarship component itself. Instead, align the scholarship objectives with attainable and measurable outcomes based on what the scholarships are designed to accomplish: enhanced scholar technical skills, strengthened leadership qualities, and strong linkages to employment opportunities, all of which will contribute to development objectives in the long term. Recommendation – Building further on the previous recommendation, we advise USAID to imbue future scholarship programs with a strong employment objective and require intensive private￾sector engagement to this end. The IP should be tasked to develop relationships with large employers from the earliest days of the activity. This should start with an awareness campaign directed to progressive employers in sectors that align with anticipated scholar fields of study followed by individual meetings between project staff and targeted employers and finally, at least six months before their return, matching individual scholar profiles with potential employers and facilitating introductions. While some design and implementation improvements would have benefitted STEP, its core activity, the scholarships component, greatly impacted 158 Egyptian women by strengthening their knowledge and skills through international study (see Figure 1). FIGURE 1 – THE STEP SCHOLARSHIPS COMPONENT IN BRIEF XIX | END-OF-PROJECT PERFORMANCE EVALUATION OF STEP USAID.GOV LESSONS LEARNED1 1. Scholarships program designs require clearly articulated and actionable objectives, which may be directly achieved due to the scholarship activities and those of the beneficiaries. 2. In cases where projects are designed with multiple components, one of which involves scholarships, the components should, to the extent possible, interlock and support each other’s objectives. 3. Cohesive alumni networks are difficult to achieve and require mentoring. Efforts and linkage mechanisms to create networks should begin early, and sufficient time should be allotted for post-graduation support, including integration into existing organizations/NGOs. 4. Outreach efforts carefully targeted to pre-defined, specific demographic profiles may streamline the process to reach applicants, reduce the number of unqualified submissions, and improve screening. 5. “Financially disadvantaged” may be negatively perceived and is too open to interpretation for effective targeting of candidates. Conversely, overly restrictive “disadvantaged” criteria may eliminate desirable candidates. Well-defined, transparent criteria, as verifiable as possible, need to be applied. 6. Quotas and/or criteria weighting systems may need to be applied to secure enough participants across predetermined demographic profile strata, e.g., to solicit applicants from different types of schools (STEM vs. non-STEM) or for geographic diversity. 7. Career counseling is best applied from the earliest stages of applicant acceptance and should take place over a matter of weeks, not days, to ensure decisions about majors and universities are given adequate consideration. Also, such decisions should ultimately tie back to labor market demand. 8. Multi-year degree programs have the advantage of starting relationships with foreign universities, which if nurtured may morph into forms of continuing education for current and future alumni. 9. A scholarship program ought to be measured according to its ability to place graduates in productive employment. This means a parallel effort underway, from program inception that develops the demand side of the employment equation with direct private sector engagement. 10. MBA degrees do not assure employment beyond the entry-level, as directly relevant experience is also needed. This means multi-month, highly relevant internships (aligned with sector and function) are an absolute requirement to ensure MBA alumni are attractive as potential hires. 11. Leadership skills may expect to be strengthened during foreign study experiences. However, demonstrating leadership in practice, as a program accomplishment, requires monitoring for several years upon the scholars’ return. There is also no universally accepted definition of leadership, which makes it incumbent on a project to specify what it means for their beneficiaries. 12. In cases of capacity building through international training, clearly defined plans are needed to delineate exactly how the training knowledge/skills will be institutionalized on the participants’ return. Establishing programmatic relationships with public and private counterparts at the program outset is crucial to this end. 1 The listing of lessons learned is included in the executive summary and not repeated in the main body of the report. 1 | END-OF-PROJECT PERFORMANCE EVALUATION OF STEPUSAID.GOV INTRODUCTION In 2014, the $250-million Higher Education Initiative (HEI) Bilateral Agreement was signed to strengthen institutions of higher education and develop lasting partnerships between the private sector, U.S. universities, and Egyptian institutions. The ultimate aim of the initiative was to strengthen Egypt’s tertiary institutions and prepare its youth, particularly women, to effectively contribute to the country’s economic growth and development. The project targets strategic fields such as agriculture, the environment, climate change, and water management; business; science, technology, engineering and math (STEM); as well as particular fields such as nursing where Egypt has a large human capacity shortage. Since Egyptian women are underrepresented in study-abroad programs, recruitment for the initiative sought to correct this imbalance and focused on reaching women from disadvantaged communities. To expand Egypt’s cadre of professionals trained in critical fields at respected universities, HEI would support a robust program of local and U.S. scholarships. The $43,912,257 ($35,385,000 obligated) STEP activity (USAID contract number AID-OAA-I-12- 00009) was designed to bolster these objectives and operated from January 2014 through December 2018. INTENDED AUDIENCE AND USE OF EVALUATION RESULTS The primary audience for the evaluation report will be USAID/Egypt, specifically the office of Education and Health and USAID/Egypt senior management. The secondary audience includes the Government of Egypt's Ministry of Higher Education and Scientific Research (MOHESR). The purpose of the Scholarships and Training for Egyptian Professionals (STEP) evaluation is to help USAID/Egypt improve the performance of similar current and future activities, and their contribution to development objectives. The evaluation will examine the effectiveness of the interventions under STEP for both the scholarships component (how it impacted the scholars' lives) and the technical assistance component (how it supported USAID technical offices to achieve their goals). EVALUATION PURPOSE AND QUESTIONS The objective of the performance evaluation is to help USAID/Egypt improve the performance of similar current and future activities and support USAID/Egypt’s development objectives by examining the effectiveness of the activity interventions for two primary components: (1) scholarships and how they impacted the scholars’ lives and (2) technical assistance and how it helped USAID technical offices to achieve their goals. The evaluation specifically addresses five evaluation questions. Questions 1, 2, and 3 were further divided into two questions each, as their sub-questions required distinct analyses (question 4 was already divided): EQ1A – To what extent does STEP—through the scholarships component as well as the technical assistance component—contribute to the achievement of USAID/Egypt’s strategic DOs? What are the critical success factors of STEP? EQ1B – To what extent does STEP—through the scholarships component as well as the technical assistance component—contribute to the priority needs of Egypt’s key government, academic, and private sector organizations? What are the critical success factors of STEP? EQ2A – To what extent does the beneficiary selection process meet the requirements agreed with USAID? EQ2B – To what extent do applicants meet the criteria for scholarship awards? EQ3A – What is the return on investment of STEP? EQ3B – What are the reasons for non-returnees? EQ4A – Are STEP alumni productively employed in fields related to their academic preparation? EQ4B – To what extent are STEP alumni positioned to take positions of leadership in academia, government, industry, and business? EQ5 – To what extent does the technical assistance component supplement the activities and USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 2 projects of other technical offices? COUNTRY CONTEXT AND PROJECT BACKGROUND STEP was premised on the hypothesis that by supporting the development of a highly trained core of professionals across Egypt, the program will contribute to a more dynamic, innovative, and productive private sector and more effective public institutions. The STEP activity will lead to individuals and institutions better equipped to: (i) implement and support policies important to Egypt's Human and Institutional Capacity development in sectors of interest for USAID/Egypt; (ii) carry out research and development that will lead to new, innovative discoveries, solutions and products; (iii) fill new opportunities offered by private industry requiring specialized skills; (iv) participate more effectively in and contribute more broadly to the country's economic and social development; (v) be accountable and transparent in managing public and private sector resources; (vi) provide better public services; and (vii) support ongoing efforts to secure the transition to a democratic Egypt. Moreover, in addition to directly addressing the HEI Bilateral, the STEP activity was designed to directly contribute to the Foreign Assistance Strategic Framework functional objective, Investing in People, and the USAID Forward focus on building the capacity of local institutions. STEP was also intended to respond to Egypt’s political and socioeconomic transition with an eye toward supporting the country’s long-term development activities. Specifically, STEP was premised on the hypothesis that by supporting the development of a highly trained core of professionals across Egypt, the program will contribute to a more dynamic, innovative, and productive private sector and more effective public institutions. STEP had two primary objectives: 1. Improve the capacity of key government, academic, and private-sector organizations through the provision of scholarships, training, study tours, and technical assistance. 2. Create a cadre of highly-trained female leaders in the fields of science, math, technology, and business. STEP aimed to improve the performance and leadership skills of Egyptian professionals, particularly women, in order to promote economic and social development in Egypt. STEP was expected to enable key Egyptian organizations and individuals to better: • Design and implement policies important to Egypt’s development and economic growth (i.e., in agriculture, private sector and workforce development, trade, science, engineering, health, education, democracy and governance, technology management, and the environment). • Manage public sector resources in an accountable and transparent manner. • Support women’s leadership and professional advancement in organizations where women have been underrepresented at decision-making levels. • Provide better public services. • Participate in and contribute more broadly to the country’s economic and social development. • Support ongoing efforts to secure the transition to a democratic Egypt. FIGURE 2 – THE FOUR PHASES OF STEP SCHOLARSHIPS AND THEIR RELATION TO EVALUATION QUESTIONS 3 | END-OF-PROJECT PERFORMANCE EVALUATION OF STEPUSAID.GOV In support of these overall efforts, the following major outcomes were expected from the implementation of the STEP activity: • A cadre of skilled professionals who can apply and transfer new knowledge and skills in priority sectors. • Increased women’s leadership and professional advancement. • Strengthened technical, managerial, and leadership capabilities in targeted public and private institutions and organizations. • Increased good will and cultural understanding through training opportunities in the United States. • Up to 100 women with MBA degrees from American universities. • Up to 60 women with bachelor’s degrees in science, technology, engineering, and mathematics from American universities. RESEARCH METHODOLOGY DATA COLLECTION METHODOLOGY The end-of-project STEP performance evaluation was implemented between March 4, 2019 and October 25, 2019 (projected) as a descriptive study using a mixed-methods data-collection approach that included both qualitative and quantitative research and statistical analysis of online quantitative questionnaires addressing each evaluation question. Eight distinct categories of stakeholders and beneficiaries were identified; individual protocols and survey instruments were created for each. Forty-two key STEP stakeholders and 158 scholar beneficiaries, including six non-returnees, were included in the population of respondents sought. Two teams working in parallel targeted 82 percent of MBA scholars and 87 percent of STEM scholars identified through purposive sampling, eventually shifting to 100 percent attempted contacts. Cooperating ministries (the MOE, MOHP, and MOHSR), USAID technical offices, and STEP subcontracted local partners were all targeted at a 100-percent response rate from the outset. Three of six non-returnee scholars were sought, and 21 of an unknown number of employers were also targeted. Two teams, each consisting of two people, one sub-team leader and one Monitoring and Evaluation (M&E) specialist, collected all primary data. The team leaders led the question and discussion sessions, while both team members took digitized notes for accuracy and later comparison. M&E team members asked clarifying questions, as warranted. The first data collection week focused more TABLE 2 - TARGET STAKEHOLDERS AND DATA COLLECTION TOOLS USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 4 heavily on face-to-face (F2F) Key Informant Interviews (KIIs), in part as a contingency measure to allow increased telephone contacts in the second and third weeks, should the F2F meetings prove difficult to arrange. This in fact proved a useful strategy, as most of those willing to meet face-to-face were exhausted by the middle of the second week of data collection. Data collection was accomplished using the following methods: 1. A desk review of project-related documents provided by USAID’s activity COR, using thematic content analysis. The desk review included five days of document analysis followed by continued reviews as documents became available. These included the 2013 STEP Scope of Work (SoW), the Bilateral Agreement, the 2014 Performance Management Plan (PMP) Narrative, all semi￾annual reports from 2014 through 2018, PMP results, Performance Indicator Reference Sheets (PIRs) for 2014 through 2018, two Data Quality Assessment reports (DQAs), MBA and STEM application packets and selection criteria, candidate status reports, STEP Project Achievements, the 2018 USAID Education Policy, the IFP Alumni Tracking Study by the Institute of International Education (IIE), and Egypt’s Vision 2030 development strategy, among others. The team learned at the outset of data collection activities about the existence of a comprehensive database of scholar information, which the former CoP confirmed toward the end of the evaluation. Access to this information would have been useful from the outset to cross-tabulate a number of scholar variables, such as scholar performance by various profile characteristics. However, only narrow segments of the database were made available to us, some of them quite late in the evaluation process. 2. Key informant interviews (KIIs) were conducted F2F whenever possible to glean nuanced respondent answers, which were sought via telephone when F2F interviews were not feasible. Virtually all of the MBA scholars willing to meet F2F were interviewed within the first 10 days of the data collection phase; all others were interviewed by telephone. All STEM scholars were interviewed by phone, with the exception of one who happened to be in Cairo during data collection. The online questionnaire tool was piloted for use as a dual-purpose instrument—also for use in telephone KIIs—in order to increase the number of responses for statistical analysis. However, its multiple￾choice structure was somewhat complicated for telephone use and the team reverted back to the original KII (also designed for both F2F and phone interviews), which proved more effective. The KII for MBA and STEM scholars was identical except for a few skip-logic questions, mainly pertaining to the additional employment information needed from the MBAs. 3. Online questionnaires became an invaluable source of data for the STEP evaluation. While this is not always the case, as response rates to online questionnaires tend to be low, the online responses filled a critical gap that developed due to a slower than expected scholar response rate. Without the Implementing Partner (IP) to support the evaluation team, our attempts to reach out to scholars typically went unanswered, particularly by the U.S.-based STEMs, even with precursor emails from USAID announcing our task. After multiple attempts to contact scholars by email, followed by attempts via WhatsApp for phone interviews, telephone KIIs and completion of online response rates gradually increased. In order to achieve a critical mass of online responses for statistical evaluation, all KII respondents, whether MBA or STEM, were asked to complete an online questionnaire as well. This tactic proved successful, as did engendering the support of KII respondents to inform STEP colleagues that their cooperation and feedback were vital to the program. 4. Group discussions preceded by paper and pencil surveys were originally envisioned as a useful, additional means of obtaining qualitative and quantitative data efficiently from the same respondents who, in a group setting, might elevate and more fully articulate key findings. However, the difficulties encountered securing F2F scholar interviews made assembling small groups virtually impossible, and only one discussion group of three participants was held. Another group was scheduled, but just one participant appeared and was interviewed as a normal F2F KII. 5 | END-OF-PROJECT PERFORMANCE EVALUATION OF STEPUSAID.GOV SAMPLING, DATA SOURCES, AND RESPONSE RATES Sampling was structured to achieve a 95-percent confidence interval among MBA and STEM scholars. One hundred percent of all other direct stakeholders were targeted, except employers whose numbers were not known (see Table 3). Respondent Type Contact Attempts by Phone and Email KIIs Completed F2F & Telephone Online Completed Target % Completed MBAs 207 (t) 383(e) 46 26 (1 par /4 inc) 74 (of 90) 97% STEMs 253 (e) 11 21 (4 par / 7 inc) 54 (of 62) 59% Ministries 4 4 4 (of 4) 100% Local Partners 6 6 6 (of 6) 100% Employers 15 4 21 (of unknown) 19% Non- Returnees 6 0 1 3 (of 6) 33% USAID 4 5 80% TOTALs 874 75 48 (t) = Telephone Calls (e) = Email (inc) = Incomplete (par) = partially complete DATA ANALYSIS AND QUALITY ASSURANCE Descriptive statistical analyses were conducted on the quantitative data—whereas a thematic approach was employed on the qualitative data—to identify explicit and implicit patterns in respondents’ answers relating to the evaluation questions. Data analysis actually began during the data collection phase with three half-day sessions involving the members of both data collection teams. The sessions were used to compare findings, seek data patterns, agree on questions for emphasis (during continuing KIIs), and begin identifying priority findings. As all data-collection tools were designed to address necessary themes arising from each evaluation question, the resulting analyses were comparable across all tools. Data collection tools and protocols were piloted to ensure relevance and clarity, and modifications subsequently made to two of them (the scholars’ key informant interview protocol and the scholars’ online questionnaire). Both quantitative and qualitative findings were sought, ensuring cross-validation between each type was possible when drawing conclusions and developing recommendations. This was most often done by comparing the responses from KIIs with those of online questionnaires, which made the observed findings stronger and more reliable. However, cross-validation was also accomplished through the examination of secondary data. One important example was derived from the scholar application packets (31 MBAs and 31 STEMs), which were invaluable to validating primary data about the application process in greater detail, including: the application scoring and rubric, the interview scoring and rubric, the workshop/camp scoring and rubric, and the English test scoring and process. Two further measures were undertaken to mitigate the fact that no Implementing Partner was available to validate interim findings. An informal meeting was held with STEP’s former CoP, which was very helpful to fill in data gaps and validate findings. In addition, a more formal validation workshop was held with the activity COR that was very useful to discuss and validate what the team perceived were its core preliminary findings. DATA COLLECTION LIMITATIONS AND CHALLENGES TABLE 3 - DATA SOURCES AND RESPONSE RATES ACHIEVED AGAINST TARGETS USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 6 1. Response rates from STEM scholars were 59 percent of the targeted level. The relatively low number is likely due to 1) STEMs not knowing or trusting the nature of our intent, as no IP introduction was made; 2) STEMs not receiving our emails due to spam filtering; and 3) STEMs being too occupied with final exams to respond to the evaluation request. Mitigation efforts included four email contact attempts per scholar by the evaluation team, two email contact attempts by USAID, appeals to STEMs who had already completed KIIs to notify other STEM scholars of the importance of their feedback, and giving them the option of simply filling out an online questionnaire if they did not have time to speak with the team by telephone. While the combined telephone and online response rates did not achieve their targets, these efforts brought the numbers to a level sufficient for substantive analysis. 2. Just four MBA employers were reached. This is likely the result of scholars’ reticence to allow us to speak with their employers, as most were relatively new in their jobs and had not yet developed strong relationships. This category of respondent was added as a source of contextual information about MBA profile attractiveness and was never required for data validity. The few responses garnered therefore remain useful. 3. As of the date of this evaluation, none of the STEM scholars had yet completed their four-year degrees and, therefore, had not returned to Egypt. This makes the evaluation premature to gauge program outcomes for STEM students pertaining to leadership, employment, or the likelihood they will remain in Egypt and contribute to high-level development objectives. 4. ROI calculations are only possible at a high level (total scholar component ROI) because detailed cost data per category (MBA or STEM), per university, or other parameters were not provided. 5. The team had no access to a comprehensive STEP scholar database, which features numerous sortable fields, as referenced in the April 2017 Semi-Annual Report. “The database is tailored to project requirements and contains detailed participant data covering all phases including participant application, selection, University Preparation Program (UPP), extracurricular workshops, academic standing, and internships.” The evaluation team requested access to detailed information files on each scholar from program inception during the team planning meeting, which we later learned existed in database form but was never fully released. The information it contained could have greatly enhanced our understanding of scholar specifics, such as employment against various isolated factors, e.g., GPA or degree attained. The former CoP was very helpful; however, the team was only able to speak with her near the end of the fifth week of the evaluation. While access to the CoP or the IP were never promised, the information they possessed would have been extremely useful, especially at the desk review stage to enhance the assessment team’s efficiency throughout the process. 6. The assessment team requested randomly selected scholar application packets—15 MBAs rejected and 15 selected; 15 STEMs rejected and 15 accepted—as the best possible source of information to determine how individual scholars were evaluated and chosen. Random selection of these packets was important to ensuring a representative cross-section of applicants; however, there is no clear indication this was done. 7 | END-OF-PROJECT PERFORMANCE EVALUATION OF STEPUSAID.GOV FINDINGS, CONCLUSIONS & RECOMMENDATIONS The findings, conclusions and recommendations section contains the core of the STEP activity assessment. Each evaluation question is addressed, beginning with specific findings that pertain to it followed by conclusions resulting from the findings, all culminating in one or more recommendations targeted to the Implementing Partner and/or USAID. QUESTION 1A. TO WHAT EXTENT DOES STEP THROUGH THE SCHOLARSHIPS COMPONENT CONTRIBUTE TO THE ACHIEVEMENT OF USAID/EGYPT’S STRATEGIC DOs, WHAT ARE THE SUCCESS FACTORS? FINDINGS MBA and STEM scholars (MBAs and STEMs) rated their U.S. academic experience as very useful, with MBAs giving it a weighted average score of 4.3 out of 5 (n=27) and STEMs giving it a weighted average of 4.4 out of 5 (n=21). The multicultural setting instilled self-confidence, independence, the ability to take initiative, and a stronger sense of cultural perspective that was described as “life changing.” Moreover, 72 percent of all scholars graduated with, or had achieved as of project final reporting, a 3.5 GPA or higher. [MBA KIIs, STEM KIIs, Online Survey, Achievement Presentation, MBA and STEM Scholar Databases] STEP’s objectives are not specifically aligned with either of its components (scholarships or technical assistance); instead, they may be applied across both components. The first of two STEP primary development objectives, aims to: Improve the capacity of key government, academic, and private-sector organizations through the provision of scholarships, training, study tours, and technical assistance (see section 5 for findings related to the TA component): • There are no indications of STEP scholarship component activities directly supporting capacity building for government, academic, or private-sector organizations/institutions other than 18 public-sector and eight academic-sector employees being selected for MBA scholarships. [Semi-Annual Reports 1-9, PMP 2014-2018] • Six MBA alumni in the public sector were interviewed. Three of them returned to government positions with no substantial change in position or salary. The other three joined the public sector for the first time (the MOHP), and they all indicated plans to leave for a higher salary after several years of networking with international organizations. [MBA KIIs] • The scholarships component was not intended to support STEP’s high-level organizational capacity-building objectives; this was the responsibility of the TA component. [CoP Discussion] The second of two STEP primary objectives is to: create a cadre of highly trained female leaders in the fields of science, mathematics, technology, and business. Sixty-two STEM degrees are being pursued: 32 engineering, 23 science, 7 technology, 0 mathematics, plus 90 scholars graduated with MBAs and returned to Egypt. [STEM Scholar Database, MBA Scholar Database and Achievements Presentation] There are five planned outcomes for the scholarships component, achievements for which are highlighted in Table 3 below. [SoW, MBA, STEM Scholars’ Databases, MBA KIIs, STEM KIIs, and Online Survey] USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 8 While interacting with MBAs, STEP undertook a number of activities that scholars indicated hindered relationship-building and/or their interest in cooperation. The contractor contends that challenges related to bullets one and two (below) were substantially improved during the second cohorts for both STEMs and MBAs: [Job Placement Challenges 2018, MBA KIIs, STEM KIIs, Local Partner KIIs, Online Survey, Contractor] • STEP was perceived by 36 percent of MBA scholars (n=45)—more heavily weighted toward Cohort 1—as having poor planning, long periods without communication and/or providing short notice for final-stage applicants, e.g., “your TOEFL is tomorrow,” “you need to be here Monday.” • Policies were not consistently communicated to scholars from the outset, as quoted in a 2018 subcontractor’s project document: “Since communicating program policy has made things more clear, we recommend to share IIE/EFE|EGYPT Policy with HEI MBA Cohort 2 from the outset in order to eliminate resistance, miscommunication, or misunderstanding…” • Four MBA scholars—without a formal question being asked—indicated that newly accepted applicants were required to sign a contract with STEP but were refused a copy of their own. • Seventy-one percent of MBAs (n=45) and 45 percent of STEMs (n=11) indicated that core program objectives were communicated as commitments or promises and not consistently fulfilled, e.g., relevant internships will be provided, you will be employed on your return, on￾campus housing will be provided. The contractor indicates on-campus housing was arranged for all students during their first year and US-based internships were never promised. • The local partner responsible for providing job search assistance acknowledged that MBAs were led to believe they had returned to Egypt with unrealistic employment expectations. “Scholars came back with attitude.” This was supported by MBA KIIs. • Returning MBA scholars were asked to sign job rejection letters and told they would not be supported if they rejected two offered positions deemed relevant by IIE and/or its local partner, EFE, who reviewed offers against applicant qualifications and objectives. This was intended to prevent scholars from rejecting reasonable offers. TABLE 4 - PLANNED OUTCOME ACHIEVEMENT RESULTS 9 | END-OF-PROJECT PERFORMANCE EVALUATION OF STEPUSAID.GOV • Four MBA scholars—without a formal question being asked—indicated that an outplacement consultant worked intensively with individual returning MBAs and committed to provide follow-up support but did not contact them again, despite assurances. • MBA scholars were unsatisfied with the employment support process upon their return to Egypt. Career counseling and job placement received the lowest weighted average ratings for STEP activities at 1.9 and 1.8, respectively (5 highest) (n=27). See also section 4A. • Twenty-nine percent of MBAs (n=28) and 20 percent of STEMs (n=25) rated STEP staff as first or second (out of five) in terms of importance as a source of support. There were no clearly identified actions undertaken by STEP to establish a sustainable scholar network, with the possible exception of creating a scholar Facebook page, an action both the scholars and STEP took credit for. Forty-seven percent of MBA scholars (n=45) referenced the alumni Facebook page as their primary networking tool, while no STEMs or MBAs reported actively using IIE’s alumni network or their partners’ networks, aside from receiving the IIE newsletter. One scholar succinctly expressed that she felt less strong without a cohesive network: “On my return to Egypt, I tried to contact multiple members of government to share development ideas but, as one person, I didn’t receive a single reply.” [MBA KIIs, STEM KIIs, CoP Discussion, Semi-Annual Reports 1-9] IIE recognizes the benefits of effectively assisting alumni to become networked, as quoted in the IFP Alumni Tracking Study: [IFP-Alumni Tracking Study 2018] In cases where IFP alumni have been able to collaborate on activities, programs, or in organizations, they have noted a shared sense of purpose and identity that helps them work together successfully and pursue joint activities. Support of alumni networks can be very successful in bringing together catalysts of change who have a shared identity, and who have common characteristics and aspirations. Promoting these alumni and their collective action can speak directly to programs’ sustainable impacts on a greater scale. Six of the STEM scholar KII respondents (n=11) indicated plans to pursue post-graduate degrees in the U.S. or Europe after a brief repatriation in Egypt. The STEM degrees include fields of concentration without strong Egyptian employment prospects, such as aerospace and biomedical engineering, prompting STEP management to voice concern with USAID about the unsuitability of certain scholar degree choices for the Egyptian job market. However, STEP was told scholars should not be limited in their selection. [STEM KIIs, STEM Scholar Database, CoP Discussion] Four-year STEM students required more time to be identified, selected, and returned than was available within STEP’s five-year project design. The origins of the STEP program design are in Washington, D.C. [Semi-Annual Reports 1-9, STEM scholar database, USAID KIIs] A costed extension (January 2019 through September 2020) for continuing support to STEM reintegration and employment is ongoing under the USAID Local Scholarships Program (LSP) with four to five staff members managed by a former STEP employee. One local partner is preparing to work with the first cohort of STEM scholars on their return in May 2019. [CoP Discussion, Local Partner KIIs] CONCLUSIONS One-hundred and fifty-eight Egyptian women underwent a life-changing cultural and academic experience that had a clear impact on their personal and professional development. The experience of studying in multicultural settings was as important as the degree itself and led to increased self￾confidence, a sense of independence, and the ability to take initiative. That 72 percent of all scholars registered GPAs of above 3.5 (as of project final reporting) is impressive and indicates strong intellectual capacity among selected program participants. USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 10 The first STEP primary objective—to improve the capacity of key government, academic and private sector organizations through scholarships, training, study tours and technical assistance—has not been measurably achieved by the scholarships component for organizations among government, academic, or the private-sector (business development services). There were no clear activities undertaken through the scholarship component that were directly linked to the achievement of STEP’s organizational capacity-building objectives, other than the selection of 18 scholars from the public sector and eight from the academic sector. However, of the six public-sector MBAs reached, none appear likely to assume leadership roles. The scholarships component was not realistically designed to achieve STEP’s high-level objectives, though the STEP scope of work is unclear in this regard and applies all objectives to the project more broadly. The second STEP primary objective—to create a cadre of highly trained female leaders in the fields of science, mathematics, technology, and business—has been numerically achieved, with 62 STEM scholars pursuing engineering, science, and technology degrees (only mathematics is not represented), and 90 scholars who have graduated with MBA degrees. However, not enough time has elapsed to conclude to what extent the graduates will assume positions of leadership. The five major scholarship component outcomes have been largely achieved: 90 culturally aware MBA graduates with enhanced managerial and leadership skills having returned to Egypt, and 62 skilled and culturally aware STEMs expected to return by the end of May 2020. STEP undertook a number of activities that inhibited relationship-building while interacting with MBA scholars and contributed to a less cooperative mindset during the critical job-seeking phase. Inconsistent communication with applicants, insufficient academic counseling, mismatched university selection, short travel deadlines, failure to provide scholars with copies of contracts, unfulfilled promises of directly relevant internships, unrealistic expectations on their return, inadequate job search assistance “as though for fresh graduates,” and threats to stop all assistance after two job rejections all played a role in MBAs appearing less cooperative during their job searches. The contractor asserts on-campus housing was arranged in all cases for the first year and US-based internships were supported but never promised to scholars. A significant missed opportunity involved a lack of cohesive network building among returning alumni (a separate issue from the lack of networking with employers). MBA alumni represent 90 individuals, each attempting to influence a small sphere of friends, family, and colleagues. Other than organizing networking events and creating a Facebook page (an action the alumni themselves also took credit for), STEP undertook no clear process or actions to actually establish a sustainable network— something more significant than holding simple events or establishing a webpage. The result is an alumni group with virtually no leverage, as a group, to raise their profile or share their perspective with policymakers, civil society, and private-sector organizations. This represents one of STEP’s most notable disappointments and diminishes the scholars’ potential influence to more significantly contribute to USAID’s and Egypt’s development objectives. A questionable design issue exists with respect to sending 17- to 19-year-olds from disadvantaged backgrounds to a country with a radically different culture for four full years. The issue is whether there is an acceptable likelihood of their permanent return to Egypt in support of the country’s development objectives, particularly when their degrees include biomedical engineering, aerospace engineering, and interests in applied research, fields they perceive offer limited employment opportunities in Egypt. It may be they do return for the long term, but this is yet to be determined. Moreover, it is also not possible to locate, select, and return students pursuing four-year degrees to their home country, all within a five-year project horizon. This requires finding alternative mechanisms to assist with their reintegration, as has been done with the costed extension to support STEM scholars on their return. 11 | END-OF-PROJECT PERFORMANCE EVALUATION OF STEP USAID.GOV RECOMMENDATIONS R1A.1 – We recommend USAID exclude higher-level objectives—such as building the capacity of government, academic, and private-sector institutions—from scholarship programs unless mechanisms and interventions are clearly defined to ensure they occur (under the scholarship activities), e.g., the selection of career public-sector employees devoted to government reforms. If the intention is for the TA component to be solely responsible for institutional capacity building objectives, this should be clearly articulated in the design and related components aligned with distinct sets of objectives. (Design) R1A.2 – We recommend USAID evaluate Implementing Partners (IPs) at the proposal stage for strategies and tactics to build strong relationships with scholars based on support and trust; then ensure IPs follow-through with such activities. A brief plan should be required (up to two pages) identifying relationship-building strategies, frequency, and type of contact, problem resolution mechanisms, and lessons learned addressing previous scholar concerns. (Design and Implementation) R1A.3 – We recommend USAID design and IPs implement future scholarship programs from the outset to include planning and activities that systematically link scholars into a cohesive network with support continued for at least six months after their return. Activities should include partially funded networking events, topical private-sector symposia, and meetings organized with policymakers and influential NGOs. The six-month time frame would also be used to help fully integrate the alumni group into a previously agreed host NGO or other suitable institution that’s interested and capable of taking the alumni network under its wing. For STEP, the National Council for Women would have been a natural fit. Other parallel options, such as more systematically tying STEP alumni to IIE’s alumni network, would also be appropriate. Such activities need to start from day one of new scholarship programs, not wait until scholars are ready to return. We recommend USAID employ these design features into future scholarship activities (medium-term) and consider mitigation options for the existing MBA scholars (near-term). Perhaps the planned near-term support for STEM scholars, under a costed extension for reintegration and employment assistance, could apply to MBAs as well through this or another contract vehicle. (Design and Implementation) R1A.4 – While it may be premature for the evaluation team to determine the likelihood of whether STEM scholars are long-term contributors to program objectives, we recommend USAID conduct a longitudinal study of STEM scholars in particular to better understand their future impact as leaders and contributors to Egypt’s development. Upon completion of the study, consideration may be given to the desirability of program designs wherein just one or two terms of study are done in the U.S., with the remainder completed in Egypt. We also suggest USAID consider scholarship program designs in which more attention is paid to specific scholar fields of study and how they correlate with labor market demand in Egypt. (Design) QUESTION 1B. TO WHAT EXTENT DOES STEP—THROUGH THE SCHOLARSHIPS COMPONENT—CONTRIBUTE TO THE PRIORITY NEEDS OF EGYPT’S KEY GOVERNMENT, ACADEMIC, AND PRIVATE SECTOR ORGANIZATIONS? WHAT ARE THE SUCCESS FACTORS? FINDINGS Egypt’s comprehensive Vision 2030 strategy is based on 12 strategic pillars, four of which are related to STEP: Education and Training, Knowledge and Innovation (and Scientific Research), Health, and Economy. Strategic pillars under the purview of the MOE, MOHP, and MOHESR each have multiple objectives aligned with STEP’s TA component outcomes (see also section 5), whereas the scholarships component supports objectives under two pillars—Education and Training and Knowledge and Innovation. [Egypt Vision 2030, Semi-Annual Reports 1-9] USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 12 The Vision 2030 Education and Training Pillar components relevant to STEP (applicable to MOE) include: • Empowering the learner with the requirements and skills of the 21st century; • Providing comprehensive and sustainable professional development for teachers; • Developing curricula to cope with global developments and information updates that take into consideration the age of the learner; • Developing the organizational structure of the ministry, educational directorates, departments, and schools to improve educational services; • Using effective technological techniques to present and share knowledge among students and teachers. The Vision 2030 Health Pillar components relevant to STEP (applicable to the MOHP) indicate: Egypt must reduce its rate of Hepatitis C; address the health care system crisis due to a lack of separation between roles (planning, executive leadership, financing, and regulation); improve governance and accuracy of data; and strengthen efficiency, accountability, transparency, and resource management. The Vision 2030 Higher Education Pillar components relevant to STEP (applicable to the MOHESR) include: Education; Scientific Research and Innovation; Building Human Character; Health; and Society Service. STEM education is also a priority; the MOHESR plans 22 new universities by 2030, eight specializing in technology. Twenty-two MBA scholars (n=45) assessed themselves as generally able to influence their immediate sphere of colleagues, family, and social contacts but unable to affect Egypt’s broader development in a measurable way. One succinct response: “The mindset in Egypt will need to change over the next decade for us to truly have influence beyond our immediate friends and colleagues.” Moreover, it is the opinion of the former CoP that the scholarships component was not designed to address STEP’s high-level objectives, which were more in the domain of the TA component. [MBA KIIs, CoP Discussion] CONCLUSIONS STEP scholarships support the Vision 2030 Knowledge and Innovation Pillar mainly with regard to higher education, which aspires to invest in human capacity and innovation toward developing complete value chains from research to commercialization. The STEM scholars’ science, information technology and engineering skills are particularly noteworthy in terms of their potential to contribute to Egypt’s enhanced scientific research and innovation aims, while both STEMs and MBAs have undergone experiences that strongly developed human character (see supporting findings under questions 2A, 4 A and 4B with positive scholar views concerning their US-based academic experiences). STEP scholarships do not measurably support Egypt’s other Vision 2030 objectives in the short to medium term. Any institutional capacity building is within the realm of the TA component (see also question5, which addresses STEP’s considerable TA component contributions to the Education and Training Pillar). RECOMMENDATIONS STEP scholarship activities effectively supported Egyptian government priorities under the Vision 2030 Knowledge and Innovation Pillar, no further recommendations are proposed. 13 | END-OF-PROJECT PERFORMANCE EVALUATION OF STEP USAID.GOV QUESTION 2A. TO WHAT EXTENT DOES THE BENEFICIARY SELECTION PROCESS MEET THE REQUIREMENTS AGREED WITH USAID? FINDINGS Outreach: Outreach efforts to solicit applications for academic studies in the U.S. aimed mainly at underserved and rural areas. Programmatic efforts planned to reach 14 governorates for regional diversity; 16 were achieved. [HEI Bilateral Agreement, PMP Narrative] The selection process: [Semi-Annual Reports 2-3, Selection Criteria, Selection Process, Cutoff points from CoP, MBA KIIs, Contractor] • A structured candidate selection process and selection criteria were shared with USAID and revised. The selection process has three main filtration points: • Stage 1: achieving a cutoff score for financial eligibility at the application stage. • Stage 2: achieving a cutoff score on a language test. o For MBA/STEM Cohort 1: EPE score (an IIE language proficiency test) with a cutoff score of 80 for applicants in Greater Cairo and Alexandria and 74 for the governorates. o For MBA/STEM Cohort 2: TOEFL/IELTS score (cutoff score 450/39/4.0). • Stage 3: achieving a cutoff score of 2.5 on an individual interview. • Group activities are used to screen personality attributes such as group work, initiation, leadership, and creativity; however, it is not clear if or how group activities scores are weighted. • The selection process was viewed by 11 MBAs (n=45), especially in Cohort 1, as disorganized and poorly communicated, with lengthy periods of no information followed by tight deadlines. Examples from MBA Cohort 1 KIIs: told to report to English TOEFL in five days; had no idea for six months if they were wasting their time; given two days to show up for an interview; given one week notice for international travel, given three days’ notice over weekend for travel from Sohag to Cairo; short notice for applicants to organize themselves for six months of training at AUC. The contractor indicates the selection process and applicant deadlines were less rushed during Cohort 2, which is validated by the fact the majority of complaints emanated from Cohort 1 scholars. Academic eligibility: The academic performance eligibility criteria for STEMs included a minimum score of 90 percent on the Thanaweya Amma exam with minimum average grades of 90 percent in science and math. For MBAs, the eligibility criteria for academic performance ranged from moderate (“good”) to low (“pass”) with a diploma. The broad eligibility requirements resulted in a large number of applications (over 9,000). [MBA & STEM Selection Criteria, Achievements Presentation] Financial eligibility: The “disadvantaged” financial criterion was used for pass/fail eligibility rather than as late-stage screening and awarded extra points to the very needy. [Application Packets, Financial Status Rubrics, Semi-Annual Reports 1-4, Local Partner KIIs, COP Discussion, USAID KIIs] • STEP developed a financial rubric on the application form to assess applicants’ “disadvantaged” financial status. Highest scores were awarded to applicants with monthly family incomes of EGP 750 or less, candidate salaries of EGP 1000 or less, a rented house/apartment, monthly expenses of 75 to 100 percent of monthly income, and young applicants working. • Home visits were conducted for STEM candidates but not for MBAs. STEP did not perceive home visits as appropriate for the MBA age group. Home visits to STEM candidates were not perceived by STEP and its local partners as the best way to verify financial status, e.g., some addresses for home visits were not the candidates’ actual residences. Home visits were considered important to verify parental consent. USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 14 • Two local partners viewed the selection of scholars with very low financial status as inappropriate for U.S.-based study: “When you focus on the very needy, the culture shock is too much, and they often break out from their communities. It is hard for them to be reintegrated…focus on scholarships inside Egypt …. the pressure from the parents and community is huge on reentry… this can contribute to non-returnees.” • “Disadvantaged” was a standard USAID term being used at the time of STEP’s design. There are other educational programs in the USAID Office of Education and Health (OEH) that do not rely on such criteria as heavily; STEP fills a niche. Scholar diversity challenges: [Achievements Presentation, Scholar Database, Semi-Annual Report 4, CoP Discussion, Contractor] • Large numbers of applications were received versus those selected: 1 percent of the applications submitted were selected for MBA Cohort 1, 1.8 percent of the applications were selected for MBA Cohort 2, 7.5 percent of the applications submitted were selected for STEM Cohort 1, and 8.3 percent of applications submitted were selected for STEM Cohort 2. • With regard to geographical diversity, 57 percent of MBAs (n=56) and 52 percent of STEMs (n=32) live in Greater Cairo and Alexandria, while the rest emanate from 14 other governorates. Among the MBAs, five governorates are represented by just two scholars and six governorates by just one each. A similar situation exists among the STEMs (see Figure 3). • Fifty-three percent of STEM scholars (n=33) were selected from Maadi STEM schools while all remaining schools in economically disadvantaged regions became a source for less than half of the accepted applicants. The MOE recommended recruitment should be open to all high school students or graduates with the required English proficiency, not strictly STEM school graduates (see Figure 3). • The contractor perceives the lack of diversity issues were partially the result of difficulty finding candidates from governorates with the willingness and adequate language proficiency to meet requirements. • With the exception of considerable program outreach supported by the MOE, there is little evidence of STEP attempting to leverage capable government entities or universities to attract candidates for MBA scholarships from the public or academic domains. An estimated 62 percent of MBA scholars are working in the private sector, 19 percent in the public sector and 8 percent at academic institutions (the remainder are unemployed, self-employed, work with NGOs, or are undefined). The former CoP said government employees were not targeted, as there was another large program offering MBA scholarships to them. Selection of potential leaders: Leadership skills for MBA and STEM applicants were assessed against the number of volunteer community services they performed together with their answers to essay questions on the application form. Candidates were also screened against a set of skills-based criteria FIGURE 3 – STEM VERSUS NON-STEM BREAKDOWN AND GEOGRAPHIC DIVERSITY 15 | END-OF-PROJECT PERFORMANCE EVALUATION OF STEP USAID.GOV through group activities and individual interviews. [MBA and STEM Application Packets, Application Scoring Rubrics, Semi-Annual Reports 1-4] The University Preparation Program (UPP) process: [Semi-Annual Reports 2-6, GMAT Tracker, CoP Discussion] • All MBA scholars attended UPP for six months at AUC and prepared for TOEFL and GMAT tests to meet admission requirements by U.S.-based universities. • MBAs with non-scientific backgrounds in some cases retook the GMAT as many as three times to improve their scores; then a waiver was negotiated by the IIE with some schools. • Applicants selected from STEM schools didn’t attend a bridge year like those from experimental and public schools, since STEM students will generally achieve higher scores on TOEFL and GMAT given the more advanced education they receive. The compressed time frame to send Cohort 2A to the U.S. required handpicking MBAs from the STEM Maadi School, which potentially eliminated opportunities for students from non-STEM schools. The STEP DQA states: Documentation describes the operational procedures of the scholarship programs rather than documenting the end-to-end data management process… it is recommended STEP document the data collection, transcription, and analysis procedures to minimize potential data misinterpretation and transcription errors. The contractor states the issues raised were corrected after the DQA report was reviewed. Selection process inconsistencies – or reasons for selection/non-selection that were not identified in IIE’s documentation (see Annex XII for additional information): [MBA and STEM Application Packets, Selection Criteria, Selection Process, Cut-off Points Document] • Twelve of 15 rejected MBA applicants achieved cutoff scores on the language tests. The reason given for rejection was they did not achieve the cutoff score on another language test (the SEPT). There was no documentation to indicate why the additional SEPT test was required if applicants met TOEFL/IELTS cutoff scores. • Four MBA applicants (n=15) who met the language cutoff score and the individual interview cutoff score were still rejected for their language scores. • One accepted STEM candidate did not meet the academic application cutoff score; she obtained a 30 while the cutoff score was 60. • The financial eligibility cutoff scores for both MBAs and STEMs lacked clarity. At least seven selected STEMs (n=16) achieved scores on their financial applications that should have disqualified them. • Six accepted MBA applications of 16 reviewed included daughters of middle- to upper￾middle-income families (parents in the judiciary or petroleum, retired air force members, business owners). CONCLUSIONS The outreach process to inform potential applicants of the STEP scholarship opportunity reached a wide array of people via online vehicles combined with newspapers and regional visits for governorates. The large number of applications received indicates project outreach efforts certainly generated interest. However, the small percentages of STEP scholarships actually accepted are disproportionate to the outreach results (9,000+ MBA applications with 117 selected as finalists and 800+ STEM applications with 69 selected as finalists), which suggests the possibility of targeting for increased efficiency. The selection process needed more effective planning, communication, and transparency for candidates to plan for the life changing activity before them. • Financial eligibility: The term “disadvantaged” is not applied according to a clearly defined rationale that ensures the most qualified applicants will be selected. It is evident some USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 16 middle-, even upper-middle-income applicants were able to gain acceptance, which indicates a problem with the consistency of the application process. Also, the negative connotation of “disadvantaged” needs consideration in practice because it may have served as a barrier to people with a well-developed sense of pride. Finally, while STEP did rely on a set of standard financial rubrics, some of them appear arbitrary for the purpose at hand and/or not easily verifiable. • Diversity: The overweight percentage of scholars from the Maadi STEM School and the high number of MBAs/STEMs more generally from Greater Cairo or Alexandria are at cross purposes with diversity objectives. Scholars from 16 governorates has the appearance of diversity, yet the very small numbers from each indicate otherwise. Moreover, the 90- percent academic achievement requirement for STEMs may be too stringent to strike a balance between study, leadership, and extracurricular activities. • The selection process: The selection criteria and process rely on identifiable rubrics and cutoff scores but are not always consistently applied or, if exceptions are required, the reasons are not documented. The rationale for using cutoff scores for two different language tests to select candidates within the same cohort, but only in some instances, is not clearly explained or documented. Acceptance of candidates from STEM schools and experimental schools may be perfectly rational, though these too raise questions about how needy these applicants were versus those who lack the resources for and exposure to such educational opportunities. These inconsistencies appear to result in the acceptance of applicants from middle- to upper-middle- income families. • Selecting STEM scholars from STEM schools alongside government, experimental, and Azhar schools is not seen as providing equitable competition, since screening was done based on academic achievement across educational systems of different strengths (Thanaweya Amma versus STEM Assessment System). RECOMMENDATIONS R2A.1 – We recommend IPs use a more targeted approach to outreach and selection with intended recipients more narrowly predefined according to demographic and other characteristics, such as: years of work experience, age range and limits, GMAT score, language proficiency, defined leadership requirements for community service, and relative weighting for each. Applicant performance during the AUC program (or similar) should also be considered as an opportunity for late-stage screening. A quota system may be considered to ensure underserved governorates are adequately represented. (Implementation) R2A.2 – MBA scholarships should be announced by IPs through government entities and universities to attract eligible candidates working within them; this will contribute to the objective of strengthening the performance of government and academic institutions. (Implementation) R2A.3 – A clearer definition of which applicants qualify as “disadvantaged” should be agreed between USAID and its Implementing Partners, with the notion that some potential leaders may be less needy but, in certain cases, more deserving of scholarships than those given extra points for living in rented flats or earning less than 1,000 EGP/month. We further propose financial status be used as a screening measure, after leadership skills are evaluated, not as a criterion for eligibility pass/fail. (Design and Implementation) R2A.4 – We recommend USAID and its Implementing Partners handle STEM and non-STEM school applicants as two separate tracks under STEM scholarship programs with a quota for the number of scholarships available for each track. Selection process and criteria should be developed for each track to ensure applicants are competing against each other and not with those in the other track. This will help increase the number of non-STEM school applicants selected for STEM scholarships. (Design and Implementation) 17 | END-OF-PROJECT PERFORMANCE EVALUATION OF STEP USAID.GOV QUESTION 2B. TO WHAT EXTENT DO APPLICANTS MEET THE CRITERIA FOR SCHOLARSHIP AWARDS? FINDINGS USAID’s priority to empower youth: One of the six principles of USAID’s education policy is to focus investments on measurably and sustainably improving learning and educational outcomes. This principle is served by USAID’s priority of empowering youth to gain the skills they need to lead productive lives, gain employment, and positively contribute to society. STEP scholarship activities contributed to this priority. [USAID Education Policy 2018] A core STEP objective is to develop potential leaders: [MBA KIIs, STEM KIIs, Achievement Presentation, GPA Tracker, Semi-Annual Report 5] • Seventy-seven percent of MBA scholars (n=96) achieved a GPA of 3.50 to 4.0, whereas STEM scholars (in-progress grades) indicate 64 percent (n= 62) achieved a GPA of 3.5 to 4.0. • Eighty-two percent of MBAs (n=45) and 96 percent of STEMs (n=11) believed the cultural integration at their U.S.-based universities made for a life-changing experience. • Despite challenges finding unpaid internships in the U.S., 100 percent of scholars who were not enrolled in summer classes succeeded in finding and securing an internship at a U.S. organization in their city of residence; however, 66 percent of MBAs (n=45) believe internships should have been longer and more directly relevant. • Twenty-four percent of MBA scholars (n=45) stated they took the initiative to find a job themselves, and 7 percent (n=45) started new ventures upon their return. One MBA scholar commented: “After I came back, I started my academy. This was due to the knowledge, skills, and courage gained during my MBA experience. My way of thinking is now different.” Career guidance to ensure those meeting basic selection criteria also support program objectives: [Semi-Annual Reports 4-9, MBA KIIs, Online Survey] • The STEP program provided career self-assessment sessions to STEM cohorts that identified fields of study related to interests, career goals, and areas of strength by applying psychometric tests that were later discussed during one-on-one career counseling sessions with a certified advisor. Forty-five percent of STEMs (n=22) found these useful. • One-week career counseling workshops for MBA graduates were held on their return to Egypt to augment their skills for reintegration into the workforce. Thirty-seven percent (n=26) of them considered the workshops useful. • Career counseling provided by STEP was perceived by 93 percent of MBA scholars (n=45) as not long enough to enable adequate planning for academic choices, universities, and internships. As noted in the section 4 findings, there is no evidence tying career counseling to labor market analysis. • Career counseling and job placement received the lowest weighted average ratings for STEP activities at 1.9 and 1.8, respectively (5 was the highest) (n=27). See also sections 1A & 4A. • Sixty-six percent of MBAs (n=25) and 67 percent of STEMs (n=21) found the STEP action planning process useful to organize themselves toward career objectives. CONCLUSIONS Potential leaders: STEP scholars performed very strongly in their U.S. programs, as evidenced by their high academic achievements, social integration, and determination in seeking internships. This indicates strong candidates were selected. There is also some evidence of leadership and becoming involved in social/community affairs; however, not enough time has elapsed to properly gauge USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 18 leadership accomplishments more broadly in practice. Career guidance: While activities such as career self-assessment and career counseling were provided to MBAs and STEMs (crucial activities to help ensure accepted applicants met later employment objectives), satisfaction rates with these services were low to moderate. Efforts to better respond to scholars’ career counseling needs could have been intensively applied from the moment of program acceptance, with more thought given to linking career counseling to a scholar’s field of study, university selection, and relevant job placement. RECOMMENDATIONS R2B.1 – We recommend future IPs strengthen and extend the career counseling function to ensure adequate time is allotted to aligning scholars’ interests, work experience, university selection, and relevant internships (and likely availability), all against a realistic assessment of employment market demand. Then they should ensure scholar action plans have specific milestones for each step in their career development process. We also suggest encouraging alumni to help advise new program entrants. (Implementation). QUESTION 3A. WHAT IS THE RETURN ON INVESTMENT OF STEP? FINDINGS The average cost per MBA is $143,598 for the two-year program and $267,096 per STEM for the four-year program, totaling $30,345,360. The average costs are inclusive of indirect and administrative costs but not classified by field of study. [USAID] Despite the plethora of research on ROI for higher education, no empirical studies indicate an ROI benchmark for Egypt. An possible exception is the End-of-Term Evaluation for USAID’s LEAD scholarship program (2016), which estimated a program-specific ROI of 11 percent to 19 percent depending on the field of study. A corresponding ROI calculation was made for higher education initiatives in Egyptian private universities estimated at 18 percent to 21 percent. [LEAD End-of-Term Evaluation, 2016] CONCLUSIONS ROI based on per-scholar averages provided by USAID are calculated as 11 percent for MBA scholars and 7 percent for STEM scholars. Significant non-monetary scholarship benefits occurred: self-confidence, leadership skills, stronger self-initiative, managerial skills, technical skills, and cultural awareness (see also sections 2 and 4). RECOMMENDATIONS R3A.1 – For detailed ROI calculations on future scholarship evaluations, we recommend USAID provide cost data classified by field of study and educational institution to allow a more robust analysis. (Design) R3A.2 – To benchmark future program results, we recommend USAID sponsor an impact analysis of STEP after two to three years have elapsed following its completion. (Design) 19 | END-OF-PROJECT PERFORMANCE EVALUATION OF STEP USAID.GOV QUESTION 3B. WHAT ARE THE REASONS FOR NON-RETURNEES? FINDINGS As of May 2018, six MBA scholars (four from Cohort 1 and two from Cohort 2) had not returned to Egypt. Six non-returnees out of 96 MBA scholars represents a non-return rate of slightly over 6 percent. Four of the non-returnees disclosed that they had filed applications for asylum to the United States. [Semi-Annual Reports 7 and 9, Non-returnee Status Notes] STEP, through its Implementing Partner, enforced mitigation mechanisms to address the status of announced non-returnees through intensive communication. An airport escort process was also enforced. [Semi-Annual Reports 7 and 9, Non-returnee Status Notes] Fifty-eight percent of MBAs (n=45) and 27 percent of STEMs (n=11) offered opinions as to why some scholars chose not to return to Egypt, which centered on two factors: limited work opportunities and expected problems with family acceptance upon their return. None of the respondents believed there were any actions STEP could have taken to prevent the non-returnees. One STEP local partner agreed: “When you focus on needy people, the culture shock is too much, causing them to break from their communities. This happened in a previous program, and we stopped focusing on the needy. Scholarships within Egypt may be more effective for such communities. The very needy can contribute to non-returnee numbers. Relaxing the disadvantaged/needy selection criteria may be the most likely solution to prevent non-returnees.” A non-returnee confirmed that STEP could not have done anything to prevent her from not returning to Egypt. [MBA KIIs, STEM KIIs, Local Partner KIIs, Non-returnee Questionnaire] CONCLUSIONS STEP measures were sufficient to limit the number of non-returnees to approximately 6 percent, which is not excessive. MBA scholars who were interviewed believe the non-returnees had already made their decisions before traveling, and there was nothing STEP could have done once the scholars were overseas. Focusing on very needy students from conservative families may augment the cultural separation problems. RECOMMENDATIONS R3B.1 – While the evaluation team has not identified any standard measures or benchmarks to identify applicants who might be susceptible to becoming non-returnees, further research is warranted. We recommend USAID direct a future IP to look into the possibility of designing a screening mechanism using demographic and psychographic (attitudes, aspirations and interests) markers common to non￾returnees then track the predictive success rate longitudinally. Any use of psychographics for screening would need to be carefully evaluated to prevent potential misuse or bias against applicants. (Design and Implementation) R3B.2 – We recommend USAID and future Implementing Partners make the disadvantaged selection criteria less stringent by awarding extra points to the very needy rather than using such criteria as pass/fail. (Design and Implementation) USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 20 QUESTION 4A. ARE STEP ALUMNI PRODUCTIVELY EMPLOYED IN FIELDS RELATED TO THEIR ACADEMIC PREPARATION? FINDINGS Pre-departure support The university selection process was not consistently understood by new scholars, as 31 percent of MBAs (n=45) and 36 percent of STEMs (n=11) did not know how they were selected. Scholars reported being enrolled in universities not listed in their applications. Two scholars reported not getting the major they requested (in some cases this was due to not being accepted at schools with their selected majors), one said she was advised to “take it or leave it,” one discovered her university hadn’t required the GMAT due to her work experience, one needed extra courses to finish within two-years, another had six courses waived due to previous credits and returned six months early. Moreover, universities were not carefully screened for basic logistics such as access to low-cost, off-campus housing and availability of public transportation. Six MBA students (n=45) mentioned these factors as a source of stress for them, especially upon arrival. The contractor indicates some universities provided information to IIE on housing, transportation, etc. that turned out to be inaccurate once the scholars arrived on campus. In those instances, IIE worked with scholars to find housing and transportation options that worked for them. IIE also did not use those schools again for Cohort 2. [MBA KIIs, STEM KIIs, Semi-Annual Reports 1-9, Contractor] There is no evidence of systematic efforts to advise scholars on selection of majors pertaining to specific labor market needs, and six MBAs (n=45) plus two STEMs (n=11) reported choosing majors without any support from STEP. Two STEM scholars (n=11) also cited limited internship opportunities for aerospace engineering majors in Egypt. STEP raised the unsuitability of the Egyptian job market for certain degree choices (biomedical and aerospace), but were told scholars should not be limited in their selection (see also section 2). [MBA KIIs, STEM KIIs, Semi-Annual Reports 1-9, CoP Discussion, Egypt Vision 2030] US academic/cultural (internships) Eighteen percent of MBAs (n=45) and 45 percent of STEMs (n=11) said U.S. internships were limited by the need to wait until junior year. Also, few unpaid internships relevant to their fields of study were available due to U.S. labor laws: “IIE should have helped us find internships with multinationals, and as it was not paid, I could not use it as job experience.” Initially, there was no mitigation strategy to counteract these obstacles, as evidenced by 33.3 percent of MBAs indicating they were satisfied or very satisfied (n=27) with the usefulness of internships to their professional development, and 27.3 percent of STEM scholars (n=22) believing the same. The contractor states the unpaid internship requirement was eliminated following the Cohort 1 experience, along with the need to wait until junior year for internship participation. [MBA KIIs, STEM KIIs, Semi-Annual Report 8, Online Survey, Contractor] There is no evidence of activities to contend with the lack of an internship culture in Egypt. Twenty percent of MBAs (n=45) expressed outright dissatisfaction with their internships. One MBA described reading a book because she was given nothing to do; one mentioned performing secretarial duties such as making photocopies; and another stated: “The program found me an internship with the Saudi-German Hospital, which was a waste of my time. The culture of internships in Egypt is not good, people are not supportive, and I didn’t write this junior-level experience in my CV.” The contractor mitigated the lack of an internship culture in Egypt after the first year by requiring job descriptions and interviews before internship acceptance. [MBA KIIs, STEM KIIs, Contractor] 21 | END-OF-PROJECT PERFORMANCE EVALUATION OF STEP USAID.GOV Reintegration The STEP SoW identifies a project objective to assist scholars to seek employment. [SoW] • For STEMs: The contractor will assist scholarship recipients in identifying internships for the summer period with organizations and companies that will enable them to obtain valuable hands-on experience. Upon completion of their scholarships, the contractor will assist the graduates in finding appropriate employment back in Egypt. • For MBAs: During the last semester of the scholarship recipients’ program, the contractor will assist the student in identifying employment opportunities in Egypt. The contractor will continue to work with the student to identify appropriate employment opportunities after she returns to Egypt. Eighty percent of MBA scholars (n=45) came back with a strong sense of self-confidence. Seventy￾three percent of STEM scholars (n=11) also expressed developing self-confidence by living alone in a foreign country. [MBA KIIs, STEM KIIs] IIE and USAID online networks are not actively used (see also section 1A). Two MBA scholars reported occasionally accessing the IIE network, and three others noted being USAID alumni (n=45). However, 47% of MBA scholars interviewed (n=45) said they were current members of the Facebook alumni network. . Sixty percent of MBA scholars interviewed (n=45) indicated they secured jobs by themselves, often helping each other through their informal networks and connections. One recommended an alumni to the MOHP; one assisted four alumni to obtain jobs at a tourism company; another helped a colleague apply to a contractor for the EU; another passed a UN consultancy position to a scholar. Two MBA graduates are currently working at incubators through their own networks. [MBA KIIs] Thirty-one percent of MBA scholars (n=45) reported that without practical experience, an MBA alone was insufficient to find employment in Egypt, and many accepted entry-level positions. The job placement local partner concurred, indicating lack of experience and being out of the job market for three years worked against the scholars – a structural design issue for postgraduate study programs outside one’s country. Four MBA scholars said employers do not take an MBA into consideration: “There are many with master’s degrees now in Egypt, what makes you special?” “Go back to what you were doing or accept an entry-level job.” “Work experience is the most valued criterion by employers in Egypt.” [MBA KIIs, Local Partner KIIs, Semi-Annual Report 8] Seventy-one percent of MBA scholars (n=45) emphasized the need for an introduction/linkage to relevant employers as a high priority to increase their employability. “Most of us changed careers, and our networks were no longer useful.” “IIE should have promoted us to secure jobs with the private sector even before returning, they had two years to promote us.” “No one looked in-depth into the employment part, if you fix that, the program would have been very successful.” [MBA KIIs] MBAs and STEMs were asked to rank their satisfaction with job placement services and internships, vis-à-vis finding employment: [Online Survey] • Twenty-five percent of MBAs (n=24) indicated satisfied or very satisfied to the statement: STEP Internships helped me find work. • Twenty-one percent of MBAs (n=24) indicated satisfied or very satisfied to the statement: The job placement and career counseling activities conducted by STEP helped me find work. • Thirty-five percent of STEMs (n=25) indicated satisfied or very satisfied to the statement: I am confident I will find work, as my specialization is in demand in the Egyptian labor market. • Twenty-eight percent of STEMs (n=18) indicated satisfied or very satisfied to the statement: I am confident the job placement and career counseling activities conducted by STEP will help me find work. USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 22 This is further supported by KIIs: 29 percent of MBAs (n=45) viewed STEP career counseling and placement services such as career fairs, lists of online job postings, and attempts to set up interviews as adding little value, things they could have largely done themselves. [MBA KIIs] MBAs, in particular, do not express satisfaction with internships and job placement support (N=25). See Figure 4 below. [Online Survey] Of six employers interviewed, four stressed the importance of soft skills and positive attitudes in the workplace in addition to specific sector knowledge. One employer indicated waiving an experience requirement due to the MBA’s personality and communication skills. Another employer hired an MBA for her adaptability, people skills, and the international perspective gained by living abroad. While the sample is very small, it offers contextual insights about the importance of soft skills. [Employer KIIs] At least 54 percent of MBA scholars are believed to be employed (N=90). Seventy-five percent of MBAs answering an employment question online (n=16) indicated an average time between graduation and employment of six months, and 75% of them indicated the position was relevant to their field of study. [Online survey, STEP and MBA Scholar Database] MBA and STEM scholars were also asked about challenges to find work. Fifty-seven percent of MBAs (n=28) cited organization work environments as a main challenge, followed by 54 percent who cited a lack of adequate salaries. Seventy-one percent of STEMs (n=21) felt a lack of jobs to suit their qualifications was a challenge, followed by 62 percent who said the primary challenge was finding an adequate salary. One-hundred percent of employed MBA scholars interviewed (n=13), answering a question about why they refused job offers, cited inadequate salaries and benefits, while 77 percent said the position didn’t match their qualifications/field of study. [Online Survey, MBA KIIs] USAID allocated $5 Million USD for a 21-month Higher Education Initiative extension to allow STEM scholars to complete their education in the US then facilitate career guidance and reintegration on their return. This was confirmed by a local partner who was signing a SoW with IIE to work with returning STEM Cohort 1 scholars. Moreover, two STEM scholars indicated IIE had contacted them to clarify their needs after graduation. [CoP Discussion, Local Partner KII, STEM KIIs, Contractor] FIGURE 4 – PERCENTAGE OF SCHOLARS SATISFIED OR VERY SATISFIED WITH SELECTED SERVICES 23 | END-OF-PROJECT PERFORMANCE EVALUATION OF STEP USAID.GOV Enabling scholars to secure employment that is relevant to their study is a vital precursor to any expected involvement in civic and social objectives. A tracer study of scholarship graduates concludes: [Macatangay, Tracer Study of BSCS Graduates] The [scholars’] commitment to serve their country occurs when they are in suitable employment positions with suitable salaries… As also noted in section 1A, some actions by STEP affected their relationship with returning alumni and contributed to a lack of cooperation on the part of the scholars during the job-search process. Two mini case studies: Finally, while no contractual requirement existed, there is no evidence of attempts to foster a continuing relationship between participating U.S./Egyptian universities and STEP alumni once scholars graduated, and therefore, no means of tapping into potential low-cost continuing education opportunities. [Semi-Annual Reports 5-9] CONCLUSIONS In the absence of a continuing record of MBA alumni employment, based on the most recent employment information received combined with responses to questionnaires, led us to estimate over 50 percent of MBA scholars are employed after an average job search duration of six months, with approximately 75 percent working in their field of study. Career counseling and job placement assistance were not well planned from the project’s start and were less effective than they could have been at linking MBAs to employment. Practical relationship building with large employers—not just job search assistance—was not undertaken, and we did not see any evidence of consistent attention being given to job market demand. No networks of linked employers were informed of project objectives or offered profiles of returning MBAs as potential USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 24 new hires. Internships did not fully satisfy their original purpose: to provide practical work experience and networking for potential employment. Scholars expressed difficulty finding suitable unpaid internships as a main challenge and the Egypt-based internships were not consistently relevant to fields of study. Once back in Egypt, MBA scholars faced several challenges finding suitable employment. Being out of the job market for two-and-a-half years, as occurs during such programs, returning scholars needed more opportunities to build work experience in their fields, since internships were often brief or not directly relevant to their careers. They also needed more coaching from the earliest days that was aimed at managing expectations on their return. It does appear, however, that scholars returned with strong soft skills, which were positively recognized by the employers interviewed. Without a concerted effort toward continuing a relationship between STEP alumni and their universities that extended beyond graduation, any expected continuing education benefits for alumni may be lost, e.g., webinars, conference announcements, or discounted educational offers. RECOMMENDATIONS R4A.1 – For future scholarship programs of a similar design, USAID should consider a sustainable, holistic cycle approach wherein IPs do the following: 1. At the program start, review available labor-market assessments to zero-in on specific job demand as it relates to fields of study; undertake a new study if warranted. (Implementation) 2. Begin career counseling weeks before scholars select fields of study and universities, factoring in work experience, career-shift interests, and labor-market demand. (Implementation) 3. Starting from day one, have a program component focused on demand-side linkage building to strengthen internship opportunities and practical employment linkages well before the scholars return. (Design) QUESTION 4B. TO WHAT EXTENT ARE STEP ALUMNI POSITIONED TO TAKE POSITIONS OF LEADERSHIP IN ACADEMIA, GOVERNMENT, INDUSTRY, AND BUSINESS? FINDINGS There is no evidence of a consistent leadership definition or a framework for its measurement throughout the project. However, there are reported success stories and achievements for MBA and STEM scholars during their stay in the U.S., such as club participation, volunteer activities, awards, and published academic papers. Six MBA scholars received a grant from the U.S. Embassy for The Dream Project, which seeks to improve the lives of 100 orphans and youth as well as social workers and educators. In October 2017, the alumni started implementing their project, working with a local NGO. [MBA and STEM Scholar Success Stories, Semi-Annual Report 8] Eighty-five percent of MBA scholars (n=26) rated themselves as satisfied or highly satisfied with the program’s contribution to their personal development and 69 percent with their professional development. Sixty-four percent of STEMs (n=21) said they were satisfied or highly satisfied with the program’s contribution to their personal development and 82 percent with their professional development. See figure 5. [Online Survey] 25 | END-OF-PROJECT PERFORMANCE EVALUATION OF STEP USAID.GOV Pre-departure support and US academic/cultural Leadership training occurred prior to departure from Egypt. The four-day agenda of the pre￾departure orientation lists the following sessions: Leadership and Culture, Emotional Intelligence and Leadership, Style of Management, Team leadership and Eight Characteristics of Team Excellence, Leadership Models, Challenges with Intercultural Teams, and Psychological Wellbeing and its Relation to Building Social Capital. Each session lasted between 30 and 45 minutes. Also, at least two separate workshops were held on Self-Leaderships Skills in 2016, one for 34 STEM Cohort 2 scholars and one for 46 MBA Cohort 2 scholars. [Pre-departure Orientation Agenda for STEM Cohort 2, July 2016, Semi-Annual Report 4] In the online questionnaire, scholars reflected on the usefulness of leadership activities provided by STEP: Thirty-six percent of MBAs (n=25) rated leadership activities as useful or very useful for their personal development, and 35 percent (n=26) rated the leadership activities as useful or very useful for their professional development. Thirty-six percent of STEMs (n=22) also rated the leadership activities as useful or very useful for their personal development, and 36 percent (n=22) ranked the leadership activities as useful or very useful for their professional development. [Online Survey] Ninety-six percent of MBAs (n=45) offered definitions of leadership qualities and believe STEP helped further develop them. Most common definitions were: having vision, inspiring others, acting as role models, being an integral part of/supporting a team, and understanding people’s strengths and weaknesses. Also, 96 percent of MBAs (n=45) and 91 percent of STEMs (n=11) reported gaining self￾confidence, becoming more tolerant and/or feeling empowered, all strengthened by dealing with people from different cultures while living abroad. [MBA KIIs, STEM KIIs] Reintegration Three of six employers reached predict their MBA employees are headed into leadership positions within their respective organizations, due to their soft skills, communication abilities, and personalities. One MBA scholar was promoted to manager within eight months of joining her organization and given a 30-percent salary increase; another received a 20-percent salary increase within a year of employment, while another mentioned she will be promoted soon. [Employers KIIs, MBA KIIs] FIGURE 5 – SCHOLAR SATISFACTION RATINGS WITH STEP SERVICES USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 26 Twenty-three percent of MBA scholars (n=26) agree or strongly agree leadership training helped with employment; 15 percent of STEM scholars (n=18) also agree or strongly agree that it will. [Online Survey] Six of 11 STEM scholars are planning to continue their post-graduate studies: Four said they were already accepted into U.S. programs, while two others plan to study in Europe. [STEM KIIs] CONCLUSIONS Scholars’ leadership and soft skills were further developed by their U.S. academic studies; these are the very factors that offer them an edge over those who have not undergone such an experience. However, despite some strong success stories, insufficient time has elapsed to assess whether leadership is being demonstrated more broadly among MBA alumni during employment or community service within Egypt. There has also been insufficient time to determine the likelihood that STEM graduates will assume positions of responsibility within Egypt, as this evaluation is being conducted prior to the completion of their studies and return home. Scholars were very positive concerning STEP’s contribution to their personal and professional development. RECOMMENDATIONS R4B.1 – We recommend critical terms such as “leadership” be clearly defined by future scholarship program IPs, shared with each scholar at the application phase, then tracked against clear competency indicators throughout their studies and for at least six months after their return. This will also help refine future selection rubrics. (Implementation) QUESTION 5. TO WHAT EXTENT DOES THE TECHNICAL ASSISTANCE (TA) COMPONENT SUPPLEMENT THE ACTIVITIES AND PROJECTS OF OTHER TECHNICAL OFFICES PLUS HELP ACHIEVE USAID/EGYPT’S STRATEGIC DOS AND CONTRIBUTE TO EGYPT’S GOVERNMENT, ACADEMIC, AND PRIVATE-SECTOR PRIORITIES? FINDINGS The STEP scope of work delineates objectives to build the capacity of key government, academic, and private-sector organizations through the provision of scholarships, training, study tours, and technical assistance. These are enumerated below. [SoW, Achievements Presentation, FY18 Results PMP, Semi-Annual Reports 1-9, Local Partner KIIs, MOE KII, Egypt Vision 2030, MOHP KIIs, USAID KIIs] • Technical assistance support to government master’s degrees (including facilitating master’s degrees for government and health) – zero completed. • Technical assistance support to government and civil society (including training for local government, elections officials, judiciary, members of parliament, local councils, civil society organizations) – zero completed for local government, judiciary, members of parliament, or local councils; one civil society organization (CSO) assessment on combatting violence against women. • Technical assistance support to government education (including training for ministries and authorities, training with MOE, MOHESR, tertiary institutions and NGOs) – Multi-year capacity-building interventions occurred under the Early Grade Learning (EGL) initiative including: 22 ministry officials and 271 regional supervisors trained, 17,000 CDs with training 27 | END-OF-PROJECT PERFORMANCE EVALUATION OF STEP USAID.GOV materials sent to regions, 1,800 trainers manuals developed, 72,807 teacher guides, 2,312 training kits, and 15 MOE officials sent to the U.S. for participant training. o MOE capacity building including full cycle support: work plan, performance management plan, assessment/gap analysis, curricula integration, teaching methodology and training, student assessments, program monitoring, and teacher certification. Training was performed in a participatory manner with beneficiaries learning alongside trainers in a ToT setting. o The MOE is very satisfied with the EGL capacity building and training, which supported Egypt’s priority vision (Education 2.0). They believe they are well positioned to sustainably administer and monitor the newly adopted curricula and ensure teachers are trained in their use. A ministry committee has been formed to continue a cascaded EGL training methodology. • Technical assistance support to government health (including training for ministry staff, resident physicians at university and government hospitals, university students, nurses, and other hospital and health-center staff) The MOHP was supported with 127 participants in hepatitis workshops, and 34 MOHP officials (including nurses and physicians) were sent abroad for training/workshops in infection prevention and control. Both types of support were perceived as technically strong by the ministry. The activity addresses an MOHP priority, the National Hepatitis C Campaign. o The participant training group for infection control found the subject matter very useful. The group faced difficulties in their ability to effect policy changes on their return. This is believed due to inadequate forethought about the eventual dispersion of knowledge gained and, though selection of operational level participants was seen as a strength (including nurses), the lack of one or two senior-level participants meant impactful changes in the near term were unlikely. There is ongoing work on a new health and insurance law that will achieve sustainable impact if it moves forward. o The returning hepatitis conference participants also faced challenges to achieve change as hospitals have traditionally prioritized treatment over prevention. Ministry staff turnover was also cited as a challenge due to three of five ministry participants having left their jobs. While participants were encouraged to disseminate information on their return to achieve institutional change, specific mechanisms to do so were not always clearly articulated. One suggestion to help enhance the ongoing development of the training group involved a mechanism for sustained communication between the host institution and the Egyptian training group. o The MOHP indicates it requires a longer learning cycle that doesn’t end with the completion of the study. • Unspecified technical assistance support to academic institutions – One assessment of university Career Development Centers completed; one regional adult and youth literacy assessment for the MOE Adult Education Authority completed, though unpublished. • Technical-assistance support to the private sector (including capacity building in economic growth issues for policymakers, technical experts at ministries/authorities, advocacy groups, private-sector business associations, councils, and chambers of commerce) – zero completed. STEP TA was designed to be demand-driven, according to the needs of USAID technical offices. The USAID Office of Education and Health was the only technical office actively engaged in using available STEP TA as a resource. Every technical office developed its own plan and did not necessarily factor in STEP TA. The Office of Education naturally accessed available STEP TA, which is believed to be the reason the conjoined Office of Health also participated. STEP made three or four presentations concerning the nature and use of its TA component to all USAID/Egypt technical offices to inform USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 28 them of STEP TA availability. Following the political upheaval of 2011, USAID/Egypt underwent budgetary and internal planning changes, which scaled back and created uncertainty about technical activities and programs. [USAID KIIs, CoP Discussion] The scholarships and technical assistance components were two projects that were “stuck together,” neither one designed to support the other. Initially it was envisioned that all USAID technical offices would approach STEP with specific needs, and STEP signed subcontractors at the proposal phase in anticipation of addressing them. However, as noted above, this concept never fully materialized from the standpoint of demand for STEP services. [USAID KIIs, CoP Discussion, Semi￾Annual Reports 1-9] Three of six STEP locally-subcontracted partners provided services in support of the scholarships component, including outreach, regional information sessions, initial training for selected scholars, and/or developing selection rubrics. One of these local partners provided career counseling and outplacement services directly to scholars and alumni. Two of the local partners completed two or fewer assignments during the entire STEP program; STEP did not attempt to advocate for their use in support of scholar employment objectives. [Semi-Annual Report 2, CoP Discussion, Local Partner KIIs] The STEP local partner tasked with assisting returning MBAs with their job searches acknowledged the following intention: “EFE/Egypt will draw upon its network of private-sector employers and link graduates to employment opportunities.” However, there is no indication direct linkages to any network of employers actually occurred. Instead, job placement services were offered through three job-search firms who identified companies that had posted positions online (see also section 4A). [MBA KIIs, Last Weekly Report Candidate Status 2018, Semi-Annual Reports 6-9, MBA KIIs] Initially, STEP was designed with a 100-percent alumni employment mandate, which was later reduced to placement assistance. Successful educational programs ensure graduates are gainfully employed. [CoP Discussion, Buenviaje1 et. al, Employability and Skills of MBA Graduates] Employability of graduates is one of the factors that determine the effectiveness of the delivery of services. Educational institutions are expected to prepare students for employment or practice of profession. Other than an initial kickoff meeting, there is no indication of actions by STEP to bring its local partners together to inform them of each other’s activities or cross-fertilize using their collective knowledge. Local partners were not informed of project events or outcomes outside their immediate scope or offered regular opportunities to interact with one another. [Semi-Annual Reports 1-9, Local Partner KIIs] CONCLUSIONS STEP effectively supported the requests for technical assistance it received, namely from the OEH. However, the lack of more extensive STEP TA utilization was largely due to its demand-driven design, as demand never materialized from technical offices other than the OEH. The inability to achieve multiple high-level STEP objectives—most notably institutional capacity building with academia, the private-sector, and government (MOE being the exception)—raises a project design issue. Each of these arguably comprise standalone projects in their own right. Such multiple and overlapping objectives, especially for a project primarily tasked with educating scholars, is counterproductive and obscures programmatic focus. Technical assistance with the MOE supported the objectives of the Egypt Vision 2030 Education and Training Pillar and successfully addressed two capacity building tracks: enabling the MOE to effectively manage the adoption and monitoring of new K-3 Early Grade Learning curricula and 29 | END-OF-PROJECT PERFORMANCE EVALUATION OF STEP USAID.GOV providing Training of Trainers (ToT) and all curricula for the teacher training itself. This was a highly leveraged capacity building program with strong chances for sustainability and was well appreciated by the MOE. Key success factors involved STEP’s start-to-finish integrated approach, which addressed the complete cycle for sustainability. Despite MOHP beneficiaries greatly valuing the technical content of health-related training, the measurable post-event knowledge transfer and policy-level outcomes remain unclear. This is largely due to inadequate attention being paid to practical implementation strategies and mechanisms on their return. Three of six local partners implemented STEP’s TA component as a standalone activity with no relation to the scholarships component. Without any cross-fertilization between local partners, which may have otherwise enhanced support of the scholarships component, practical methods of improving job placement activities, such as developing relationships with progressive Egyptian employers, were never undertaken. This represents a significant missed opportunity for STEP and its beneficiaries. A similar process is often undertaken in private-sector value-chain programs that coalesce large buyers and small subcontractors by working both sides of the equation simultaneously to sensitize the needs of one to the other. STEP did not conceive of a methodological connection between their TA component resources and improving the chances of relevant employment for returning scholars, nor did they advocate for it. RECOMMENDATIONS R5.1 – We recommend USAID not apply multiple, overlapping, high-level objectives to future scholarship programs, which require the addition of a TA component and a distinct set of objectives largely unrelated to those of the scholarship component itself. Instead, align the scholarship objectives with attainable and measurable outcomes resulting from what the scholarships are designed to accomplish: enhanced scholar technical skills, strengthened leadership qualities, and strong linkages to employment opportunities, all of which will contribute to development objectives in the longer term. (Design) R5.2 – Building further on recommendation R5.1, we advise USAID to imbue future scholarship programs with a strong employment objective and, in so doing, require intensive private-sector engagement by its IPs to this end. The IP should be tasked with developing relationships with large employers from the earliest days of the activity. This should start with an awareness campaign directed to large, progressive employers in sectors that align with anticipated scholar fields of study. The awareness campaign should be followed by individual meetings between project staff and the targeted employers (plus those expressing an interest in the program following the awareness campaign) to accomplish several goals: 1. Explain the objectives of the scholarship program and why they are vital to [Egypt’s] development, while appealing to their need for effective Corporate Social Responsibility (CSR) programs to give something back—not to infer neediness on the part of graduates because they are among the most qualified, rather to help educate employers to be less class conscious and hire the best people for the job. 2. Impress upon the employers the rigorous selection process and why the scholars are among the most intellectually gifted in the country. 3. Differentiate the value of these students versus their domestic counterparts, e.g., their international perspective, strong ability to take initiative, increased self-confidence, enhanced leadership qualities, and heightened cross-cultural sensitivity. The final stage will involve matching individual scholar profiles, at least six months before their return, with potential employers, then facilitating introductions. We recommend these impactful activities be designed into future scholarship programs by USAID and implemented by IPs as a practical means to more precisely tie scholarship initiatives to economic growth. (Design and Implementation) USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 30 R5.3 – In cases where technical assistance is used to send people for participant training or foreign conferences, it is crucial to have clear strategies and mechanisms that have been agreed with the beneficiaries prior to their travel and lead to the adoption of the knowledge gained upon their return. We recommend IPs base the selection of participant training candidates on a well-articulated mini-proposal that describes how the candidate proposes to disburse the acquired knowledge within their institution/organization upon their return, precisely how this will be implemented, how it will be sustained, and why institutional change will occur. This should be a precursor for participation. (Implementation) SUMMARY CONCLUSION The overriding intent of the USAID STEP activity was to provide U.S.-based MBA and STEM scholarship opportunities to academically gifted Egyptian women with leadership potential who would not otherwise have been able to afford a degree from an American university. STEP was designed to support up to 160 scholars to secure degrees and return to Egypt with a respected academic credential, increased self-confidence, enhanced leadership, and heightened cultural awareness as a powerful cadre of women poised to help shape Egypt’s developing economic, cultural, and political landscapes. There is no question the STEP activity succeeded in forever altering the personal and professional framework of 158 Egyptian women, and the phrase “life-changing” was repeatedly heard from the MBA and STEM scholars interviewed. Ninety MBA alumni returned with a respected degree—and the skills and knowledge that accompany it—plus a stronger sense of initiative as independent women in the 21st century. Sixty-two STEM students are in various stages of completion to the same end, all of which will have completed their studies by May of 2020. Perhaps the most commonly referred to and appreciated experience among scholars was precisely what USAID had aspired to: a stronger cultural understanding of the USA and other nationalities. In these respects, the STEP scholarships component achieved its short-term objectives. Less certain is whether or not the STEP scholarships component achieved sustainable impact beyond educating 158 Egyptian women and potentially empowering them as leaders in their fields. Did or will they measurably improve the capacity of key government, academic, and private-sector organizations (read institutions)? Not likely in the near to medium term. Did or will the scholars become leaders in science, technology, engineering, mathematics, and business? Possibly, but there is no evidence to substantiate this; it is simply too early to determine how many will succeed in that regard. And there are myriad other objectives ascribed to STEP including: STEP is expected to enable key Egyptian organizations and individuals to better: design and implement policies important to Egypt’s development and economic growth (in agriculture, the private sector and workforce development, trade, science, engineering, health, education, democracy and governance, technology management, and the environment); manage public-sector resources in an accountable and transparent manner; provide better public services; participate in and contribute more broadly to the country’s economic and social development; and support ongoing efforts to secure the transition to a democratic Egypt. Did STEP measurably achieve any of these? One of them, certainly, through its TA component: the provision of highly regarded capacity building to the Ministry of Education (MOE) for the installation of upgraded teaching curricula (grades K-3), which by all accounts are in widespread use among governorates. This activity was sustainable and impactful. While the work with the MOE was viewed as very successful, and foreign participant training support to the MOHP was also technically strong, many other aims of the STEP activity remained unfulfilled. Despite the inherent demand-driven design of the TA component, the evaluation team’s contention is the broad array of objectives were simply too numerous and far-reaching for any 31 | END-OF-PROJECT PERFORMANCE EVALUATION OF STEP USAID.GOV individual project to achieve them—even if greater technical office demand had transpired. Moreover, the overlapping nature of the objectives and the fact that many involve structural changes to the public sector, raise questions about how realistic they were from the outset and what damage they may have done to programmatic focus. Such concerns are referred to as design issues under the recommendation sections of this report. A number of implementation issues also arose during the five-year STEP activity, which in hindsight could have been handled more effectively. For instance, during the crucial applicant-selection process, selection rubrics were numerous and explicit yet applied in a manner that did not consistently make clear why one candidate was selected over another. Or why 17-year-old applicants—forced to work because their families were very needy—were necessarily given higher points than 17 year-olds simply attending school with their peers. STEP designed its scholar selection process and criteria to target potential leaders from among the most financially disadvantaged groups in Egypt under the rationale that they could never afford an education at a U.S. university. However, given the high cost associated with obtaining a U.S.-based degree, such an education is in fact out of reach of large segments of Egyptian women, from low- to middle- and even upper-middle-income families. The STEP financial criteria were aligned to screen out all but the low-income candidates. There are two potential problems with this: 1) The STEP selection criteria were designed to target low-income individuals but did not necessarily prevent middle-income candidates from gaining acceptance, i.e., those who could easily avoid disclosure of multiple sources of income or bank accounts or even misrepresent their primary residence. 2) The very low target set by the financially disadvantaged selection criteria likely deprived other individuals with low incomes—but greater than 1,000 EGP—who were equally deserving of STEP scholarships. For MBA scholarships, these included women in middle management, women employed in the public sector, and women in the early stages of entrepreneurship. For STEM scholarships, these included middle-income public-school students who excelled in science and math. In most cases, scholars indicated universities and fields of study were chosen in hours or days rather than weeks, though the contractor does not agree with this, and important terms such as disadvantaged and leadership were not clearly defined. Nonetheless, the selected scholars achieved great academic success and, as of the project’s end, well over half had sustained GPAs in excess of 3.5. Two particularly notable conclusions of this evaluation, which are both design and implementation oriented, involve activities that transpired upon the scholars’ return: 1) The inattention to building a cohesive scholars’ network to empower them as a group, not simply as individuals; and 2) the lack of private-sector engagement to productively employ STEP alumni in their field of study. Various activities were applied toward both ends, but none were particularly creative, nor were they undertaken with the necessary intensity. As such, more robust and deliberate approaches are recommended to engage private-sector companies for employment and Non-Governmental Organizations (NGOs) for network building, with both activities starting from the very outset of future scholarship programs. USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 32 ANNEXES 33 | END-OF-PROJECT PERFORMANCE EVALUATION OF STEP USAID.GOV ANNEX I: EVALUATION SCOPE OF WORK STATEMENT OF WORK End of Project Performance Evaluation of the Scholarships and Training for Egyptian Professionals (STEP) Activity I. PURPOSE OF THE EVALUATION A. Purpose The purpose of the final evaluation of the Scholarships and Training for Egyptian Professionals (STEP) is to help USAID/Egypt improve the performance of similar current and future activities, and its contribution to USAID/Egypt’s development objectives. The evaluation will examine the effectiveness of the interventions under STEP for both the scholarships component and how it impacted the scholars' lives and the technical assistance component and how it supported USAID technical offices to achieve their goals. STEP is funded by USAID/Egypt and implemented by the Institute of International Education (IIE). The evaluation will collect and analyze evidence to assess the following: • The degree to which STEP contributes to the achievement of USAID/Egypt's development objectives such as: - Improve the capacity of key government, academic and private sector organizations through scholarships, training, study tours and technical assistance. - Create a cadre of highly trained female leaders in the fields of STEM and business administration. • How STEP supported Mission projects and activities through providing administrative, logistical, and programmatic supportservices. • The extent to which the activity hasrealized its planned outcomes and outputs. • The process employed to select scholarship recipients. B. Audience and Intended participants The primary audience of the evaluation report will be USAID/Egypt, specifically the office of Education and Health and USAID/Egypt senior management. Secondary audiences include the Government of Egypt's Ministry of Higher Education and Scientific Research (MOHESR). USAID/Egypt will review and share the executive summary, final report and recommendations with the MOHESR and the general public via the Development Experience Clearinghouse (DEC). USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 34 II. SUMMARV INFORMATION Activity Name Scholarships and Training for Egyptian Professional (STEP) Implementer Institute of International Education Contract # AID-OAA-I-12-00009 Total Estimated Ceiling $43,912,257 Life of Activity January 2014 - December 2018 Active Geographic Regions Nation Wide Development Objective(s) (DOs) DO 22: Quality Education Enhanced USAID Office Office of Education and Health, USAID/Egypt III. BACKGROUND A. Development Hypothesis STEP was premised on the hypothesis that by supporting the development of a highly trained core of professionals across Egypt, the program will contribute to a more dynamic, innovative, and productive private sector and more effective public institutions. The STEP activity will lead to individuals and institutions better equipped to: (i) implement and support policies important to Egypt's Human and Institutional Capacity development in sectors of interest for USAID/Egypt; (ii) carry out research and development that will lead to new, innovative discoveries, solutions and products; (iii) fill new opportunities offered by private industry requiring specialized skills; (iv) participate more effectively in and contribute more broadly to the country's economic and social development; (v) be accountable and transparent in managing public and private sector resources; (vi) provide better public services; and (vii) support ongoing efforts to secure the transition to a democratic Egypt. B. Activity Summary Several reports highlight that Egypt lags in labor market efficiency, education quality, management quality and staff training. To tackle this challenge, Egypt needs to better develop the skills and knowledge of Egyptian students and professionals. In 2014, a $250 million Higher Education Initiative (HEI) bilateral was signed to strengthen institutions of higher education and develop lasting partnerships between the private sector, U.S. universities and Egyptian institutions. The ultimate aim of the Initiative was to strengthen Egypt’s tertiary institutions and prepare Egypt’s youth, particularly women, to effectively contribute tothe country's economic growth and development. The Initiative targets strategic fields such as agriculture, environment, climate change, business, water management, science, technology, engineering and math (STEM) as well as particular fields such as nursing where Egypt has a large human capacity shortage. As 35 | END-OF-PROJECT PERFORMANCE EVALUATION OF STEP USAID.GOV Egyptian women are underrepresented in study abroad programs, recruitment for the Initiative sought to correct this imbalance and focus on reaching women from disadvantaged communities. The HEI is aligned with: (i) Goal II 2 in the USAID Education Strategy]; (ii) Education and Training priority paper (part of the USAID/Egypt interim strategic plan for 2012-2013); (iii) Workforce Development Crisp paper2; and (iv) the USAID/Egypt Results Framework for Education and Training. The USAID Human and Institutional Capacity Development (HICD) Handbook] states that it is an Agency policy to integrate a human and institutional capacity development approach when conducting regional, country, or sector strategic planning and when designing new development activities. In order to expand Egypt's cadre of professionals trained in critical fields at respected universities, HEI would support a robust program of local and U.S. scholarships. I.1 Project Description The Scholarships and Training for Egyptian Professionals (STEP) activity directly contributes to the Foreign Assistance Strategic Framework Functional Objective “Investing in People" and to the USAID forward focus on building the capacity of local institutions. At the same time, STEP responds to Egypt's political and socioeconomic transition with an eye towards supporting the country's longer-term development activities. The activity results and indicators contributing to USAID indicators are as follows: • Result: Individuals attending tertiary education institutions with USG scholarship or financial assistance; • Result: Individuals from underserved and/or disadvantaged groups who accessed tertiary education programs; • Result: A cadre of skilled professionals who can apply and transfer new knowledge and skills in priority sectors; • Result: Increased women's leadership and professional advancement; • Result: Strengthened technical, managerial, and leadership capabilities in targeted public and private institutions and organizations; • Result: Increased good will and cultural understanding through training 2 Goal II in the USAID Education Strategy: proved ability of tertiary and workforce development programs to produce a workforce with relevant skills to support country development goals by 2015; http://www.usaid.gov /our work/education and universities/documents/USALD ED Strategy feb2011.pdf USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 36 opportunities in the U.S. and; • Result: Up to 100 women with Masters of Business Administration degrees from US universities; and up to 60 women with bachelor's degrees in science, technology engineering and mathematics from American universities. 2 lntermediate Result 1: Tertiary & Workforce Development Programs able to produce graduates with relevantskills; and Intermediate Result 2: Improved quality and relevance of Technical Education 3USAID HCID Policy paper (http://www.usaid.gov /poLicy / ads /2 00 /20 I maf.pdO 37 | STEP END OF TERM EVALUATION REPORT USAID.GOV I.2 Activity objectives STEP has two primary objectives: • Improve the capacity of key government, academic private sector organizations through the provision of scholarships, training, study tours and technical assistance. • Create a Cadre of highly trained female leaders in the fields of science, math, technology and business. STEP would improve the performance and leadership skills of Egyptian professionals, particularly women, in order to promote economic and social development in Egypt. STEP is expected to enable key Egyptian organizations and individuals to better: - Design and implement policies important to Egypt's development and economic growth (i.e.: in agriculture, private sector and workforce development, trade, science, engineering, health, education, democracy and governance, technology management and the environment.) - Manage public sector resourcesin an accountable and transparent manner; - Support women'sleadership and professional advancement in organizations where women have been underrepresented at decision making levels; - Provide better public services; - Participate in and contribute more broadly to the country's economic and social development; and - Support ongoing efforts to secure the transition to a democratic Egypt. In support of these overall efforts, the following major outcomes are expected from the Implementation of the STEP activity: - A cadre ofskilled professionals who can apply and transfer new knowledge and skills in priority sectors; - Increased women's leadership and professional advancement; - Strengthened technical, managerial, and leadership capabilities in targeted public and Private institutions and organizations; - Increased good will and cultural understanding through training opportunities in the U.S. and; - Up to 100 women with Master of Business Administrationdegrees from U.S. universities; and up to 60 women with bachelor's degrees in science, technology engineering and mathematics from American universities. I.3 Activity’s key achievements STEP awarded 98 U.S. master’s degree scholarships in business administration to women in order to provide them with the management and leadership skills needed to rise to leadership positions in both the private and public sectors. This activity included support to identify internship opportunities while they are in the U.S. as well as job placement support after they return to Egypt. Also, STEP awarded approximately 62 Undergraduate Scholarships at U.S. Universities to Egyptian women to pursue undergraduate degrees in science, technology, engineering and math (STEM). STEP engaged undergraduate women from the broader scientific community and encouraged them to take an active role in changing the underrepresentation of women in middle and high-level positions in the public and private sectors in STEM fields. Such scholarships enabled women to benefit academically from an extended period in the U.S. while developing an understanding of American systems, institutions, and culture. Supplemental English language training, leadership skills' development activities and acculturation opportunities were also provided. The scholarships were awarded to women in at least two separate cohorts and the selection criteria focused on economically disadvantaged young women, USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 38 with high academic achievements and leadership potential. 96 MBA scholars have already graduated with a total of 27 employed until now, however; the cohort I STEM undergraduate scholars will graduate in 2019 and cohort II will graduate in 2020. As STEP ended in December 2018, the STEM undergraduate scholarships will complete their program through the Local Scholarships program for private universities (LSP/Private). Lastly, STEP provided technical expertise and services for the design and implementation of training and professional development activities in Egypt and the U.S. STEP assisted technical offices in designing activities in order to meet their development objectives. Training and professional development opportunities included: classroom training, technical workshops, study tours, peer exchanges, conferences, and assessments. I.4 Critical Assumptions - Allscholars will return to Egypt after they finish their study in the U.S. - All scholars will find suitable jobs when they come back to Egypt where they can benefit from what they have studied. - The GOE maintains trained employees and do not lose them to the private sector. C. Relevant Documentation The Contracting Officer’s Representative, through USAID/Egypt’s Education and Health Office, will provide relevant activity documents that are available within 5 working days of award of contract. These documents include: 1. Higher Education Initiative Bilateral Assistance Agreement 2. STEP contract and amendments 3. STEP Proposed Annual Work Plan 4. STEP Progress Reports 5. STEP Fact Sheet 6. STEP Semiannual reports 7. STEP Activity work plan 8. Beneficiary list and contact information. 9. List of universities. 10. The complete database of the STEP alumni and STEM current beneficiary students In addition to the above list, the evaluator document review will consider other secondary literature determined relevant by the evaluation team. 39 | STEP END OF TERM EVALUATION REPORT USAID.GOV IV. EVALUATION QUESTIONS This section presents the fundamental questions that the evaluation will answer. The evaluation will produce a comprehensive analysis of the outcomes and the causal contributors to those outcomes. Evaluation questions fall into four basic evaluation criteria - relevance, effectiveness, efficiency and impact. The specific evaluation questions are: Questions Suggested Data Sources (*) Suggested Data Collection Methods Data Analysis Methods 1. To what extent does STEP through the scholarships component as well as the technical assistance component contribute to the achievement of the USAID/Egypt’s strategic development objectives as well as to the priority needs of Egypt key government, academic and private sector organizations? What are the critical success factors of STEP? - Performance monitoring data - Activity progress/semi annual reports - Activity beneficiaries -Desk review - Key informant interviews and surveys -Analyze results of survey and key informant interviews USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 40 Questions Suggested Data Sources(*) Suggested Data Collection Methods Data Analysis Methods - Explanation: The U.S. government is providing assistance to the people of Egypt in order to achieve specific strategic objectives. It also seeks close cooperation with the Government of Egypt to address the country's development needs. The activities supported by USAID resources are intended to support these objectives. Investigation of this question will determine if the fields in which awards are made are sufficiently aligned and balanced to achieve USAID' s development objectives (* Improve the capacity of key government, academic and private sector organizations through scholarships, training, study tours and technical assistance. *Create a cadre of highly trained female leaders in the fields of STEM and business) as well asthe priority needs of MOHESR and key government, academic and private sector organizations. It will also provide information to determine the ideal balance of awards among current and other possible disciplines - STEP achieved many successes either in the scholarships component or the technical assistance component. It also faced many challenges. As the purpose of this evaluation is to help the mission make informative future decision, the evaluation 41 | STEP END OF TERM EVALUATION REPORT USAID.GOV Questions Suggested Data Sources(*) Suggested Data Collection Methods Data Analysis Methods should assess the critical success factors of the activity, best practices, and challenges and how the risks were mitigated. 2. To what extent does the beneficiary selection process meet the requirements agreed with USAID and to what extent do applicants meet the criteria for scholarship awards? Explanation: USAID agreed with STEP on the selection criteria for the selected students. The Evaluation should evaluate to which extent applicants meet this criteria. - - Selection criteria memo - Students applications and scoring sheets - Reports - Students surveys - Desk review - Surveys - Key informant interviews. - Compare selection criteria to the applications - Analyze results of survey and key informant interviews. 3. What is the Return on Investment of STEP? What are the reasons for non-returnees? Explanation: Calculating the ROI of STEP will be extremely challenging. The "investment" per student is simply USAID's fully-loaded costs, including overhead, of attaining the degree. However, it is much more difficult to reliably and credibly monetize the return to education programs. For whatever efficiency measure the evaluation team deems feasible, it will calculate the measure separately for students in bachelor's vis-a-vis master’s degrees to enable USAID to make informed decisions about its future strategy. As part of measuring the return on investment of STEP, the mission needs to know the reasons behind non-returnees. The - Reports - Interviews - Desk review - Focus groups - Key informant interviews - Analyze the data in Reports - Interviews USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 42 Questions Suggested Data Sources(*) Suggested Data Collection Methods Data Analysis Methods non-returnees ratio for the HEI/MBA scholarships program is considerable, it is about 5 percent. Knowing the reasons behind that will help USAID in future planning for similar activities. 4. A. Are STEP alumni productively employed in fields related to their academic preparation? Explanation: For various reasons, alumni may be unemployed, under￾employed, or work in fields unrelated to their degree. The data collected in response to this question will document employment rates and will characterize the nature of the employment. The information gained in response to this question will also help USAID ensure that students are selected who are committed to work in fields aligned with their academic preparation. It will also help to align fields of study with workforce requirements. Reported data, subject to feasible methods of collection, will include, but may not be limited to, the following: • Employment status (i.e. is the graduate employed?, percentage employed after graduation, time between graduation and first job, percentage going for future studies); • Sector of employment, cross￾tabbed with degree discipline (i.e. is the sector of employment related to the discipline in which the degree was attained); • Indicators of productivity (e.g. salary, promotions, publications, - STEP Alumni - STEP Beneficiaries / students - Focus groups - Key informant interviews - Desk review - Analyze data from of the interviews and focus groups - STEP Alumni / beneficiarias - Prívate sector employers - Focus groups - Surveys - Interviews - 43 | STEP END OF TERM EVALUATION REPORT USAID.GOV Questions Suggested Data Sources(*) Suggested Data Collection Methods Data Analysis Methods academic rank, civil service level); and • All data should be disaggregated by degree type, discipline, gender, and other available and relevant demographic variables. B. To what extent are the alumni of the STEP positioned to assume positions of leadership in academia, government, industry, business? Explanation: Information on alumni's employment profile will help to determine whether the current position can be extrapolated to positions of leadership in the future. Some operational indicators include: ranking of private sector employer, salary promotions since graduation and Government's ranking methods. 5. To what extent does the technical assistance component supplemented the activities and projects of other technical offices? Explanation: STEP’s technical assistance component is working with different technical offices through providing a broad variety of services. Satisfaction of different technical offices about these services and the level of support it gave to their activities is important to be measured. - Activity report (progress and semiannual) - USAID staff - Beneficiaries/stud ents - Desk review - Focus groups - Interviews - Surveys - Analyze data of the reports and interviews USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 44 V. EVALUATION DESIGN ANDMETHODOLOGY The STEP evaluation is envisioned as a final performance evaluation that also provides summative information. It will collect and evaluate the data and evidence necessary to address the fundamental evaluation questions. As the evaluation will rely heavily on data that have been collected by program management, the evaluation will assess secondary data and verify its reliability. USAID recognizes that data limitations and feasible methods of collecting data may significantly impact the analysis. An essential element of the evaluation is to collect data from students, alumni and their employers to determine how STEP recipients are benefitting from their educational experience and how they are using their education to contribute to economic growth and social stability. The Evaluation Team must provide an update to the above design matrix that includes the most appropriate data collection and analyses methods. A mixed method approach to the evaluation that complies to the greatest extent possible with the USAID evaluation policy is recommended. The evaluation will rely on quantitative and qualitative evidence to answer the evaluation questions. Proposed data collection methods include document review, online survey of alumni, on-campus survey of current beneficiaries, group discussions of alumni and current beneficiaries. To facilitate analysis, the data will, to the extent feasible, be collected and reported in a way that enables disaggregation across multiple dimensions. These dimensions include, but are not limited to, sex, academic discipline, degree type, institution, and sector of employment. Data Collection Methods: Desk review of relevant documentation - STEP COR will provide all available background documents. Relevant reports include the USAID Education Strategy, the USAID HCID Policy paper, Egypt’s Vision 2030, annual program reports, program agreement documents and Performance Management Plan. USAID/Egypt will provide the Evaluation Team with soft copies of the activity related documents before arrival to Cairo. All team members will review these documents in preparation for the initial team planning meeting. Key informant interviews The evaluation team will conduct semi-structured interviews with the officials of USAID, MOHESR and recipient universities' administration to collect information about how and why STEP works, the extent to which it contributes to institutional and development objectives, and its relevance to institutional and government priorities. Key Informant and Group Interviews will include, but may not be limited to: • Ministry of Higher Education and Scientific Research • USAID/Egypt Office of Education and Training Activity Manager (COR) and Alternate Activity Manager (ACOR). • USAID activity managers for activities implemented under STEP. Beneficiaries: MBA/STEM graduates, STEM students (online interviews orsurveys), MOE beneficiaries, MOHP beneficiaries. Online and On-campus surveys- The quantitative portion of the evaluation is expected to be on a large-scale online survey of STEP's alumni and on-campus current recipients. USAID maintains a database of STEP alumni and current students. If the evaluation team can access these lists, it will include all alumni in an on-line survey, and a random sample of students in an on-campus survey. The evaluation team will use its local IT unit to implement the online survey also translated into Arabic as necessary with using standard online survey applications. Group discussions - The evaluation team will conduct group discussions with samples of STEP alumni and current beneficiary students. If possible, the team will recruit participants randomly from lists of alumni. The qualitative interviews will provide evidence of "how" and "why" the STEP program works and will support the quantitative data collected through the proposed online and on￾campus surveys. 45 | STEP END OF TERM EVALUATION REPORT USAID.GOV Secondary data - The evaluation will rely heavily on secondary data collected by program management - USAID, MOHESR and universities. The evaluation team will review and assess the quality of these data, address quality issues to the extent possible, and note any remaining deficiencies in the evaluation report. The evaluation team will also identify and validate other sources of secondary data such as labor market surveys that may contribute to the evaluation. The final evaluation report will present evidence to determine what the program has achieved; how its achievements relate to expected results; how STEP is implemented; perceptions of the program among alumni, employers, and the GOE. The report will explicitly state any methodological or data limitations encountered. The report will include at least the following data - disaggregated by degree type, discipline, sex, and other available demographic variables as relevant. • Number of STEP applicants. • Number of STEP awards. • Number of degrees attained. • For alumni who are no longer in Egypt, country of destination. • Employment status (i.e. is the graduate employed? reasons). • Sector of employment, cross-tabbed with degree discipline (i.e. is the sector of employment related to the discipline in which the degree was attained). Potential Limitations of the Data and Analysis One of the key limitations to conducting this evaluation is the close out of the STEP activity. The activity ended December 2018 and therefore the implementing partner (IIE) will not be involved during the implementation of the evaluation. As such, the Evaluation team will not include IIE in the plan for data collection through Key Informant Interviews. Additionally, the Evaluation team will not hold a validation meeting with IIE after data collection. The STEP COR will be providing to the best of her best knowledge all background documents needed, she will also be facilitating the interview schedules and the meetings with the Ministry of Education, Ministry of Higher Education and Scientific Research and Ministry of Health staff as well as the scholarship recipients. The Evaluation Team will work closely with OEH STEP COR in identifying and getting to the population of stakeholders and beneficiaries to be sampled. The evaluation team will work closely with the Office of Education and Health COR and the Program Office Evaluation COR. The OEH will provide relevant background documents and contact lists. The Program Office and the OEH Assistance Officer’s Representative (AOR) will be made available to the team for consultations regarding sources and technical challenges during the evaluation process. In the Limitations section of the draft final report, the Evaluation Team must disclose how findings for each question were affected due to the unavailability of the Implementing Partner. Also, the team should describe how those limitations were mitigated. The methods proposed for collecting and analyzing evaluation information are potentially subject to selection bias. Selection bias occurs when the subjects of surveys or interviews are not representative of the population of interest. In this case, selection bias will likely result from online survey respondents self-selecting themselves. Similar self-selection bias may affect employment numbers as those who are employed are more likely to respond than those who are not. Lack of current postal addresses or email address will also favor urban relative to rural recipients. USAID recognizes that a limitation of the study is the relatively short period of time it has operated. It takes several years after graduation and entry into the workforce to achieve the ultimate objective of producing citizens who contribute to Egypt's economic and social development. Therefore, the evaluation will collect the best available data and make evidence based inferences while recognizing the limits of the analysis. USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 46 USAID/Egypt will review and approve a data analysis plan that details how focus groups and key informant interviews will be transcribed and analyzed; what procedures will be used to analyze qualitative and quantitative data from key informant and other stakeholder interviews; and how the evaluation will weigh and integrate qualitative data from these sources with quantitative data from performance indicators and the activity performance monitoring records to reach conclusions about the effectiveness and efficiency of STEP. Data will be disaggregated by governorates and academic backgrounds when applicable, to identify how the activity benefitted women from different governorates and different academic backgrounds especially for the MBA scholarships. VI. DELIVERABLES AND REPORTINGREQUIREMENTS A. Deliverables o Team Planning Meeting (TPM): A team planning meeting must be held in Egypt before the start of the field work. This meeting will allow USAID/Egypt to discuss the purpose, expectations, and agenda of the assignment with the Evaluation Team. In addition, the team will: • Finalize team members' roles and responsibilities; • Review and make recommendations for improving the precision of evaluation questions; • Review and finalize the assignment timeline; • Present and discuss data collection methods, instruments, tools and guidelines; and • Review and clarify any logistical and administrative procedures for the assignment. o Work Plan: A well-written, detailed methodology and data analysis plan (evaluation design, data analysis steps and detail, operational work plan) must be prepared by the team and discussed with USAID during the planning meeting. An interview schedule will be submitted as part of the work plan. USAID will provide the evaluation team with an initial list of interviewees, from which the evaluation team can work to create a more comprehensive list. The evaluation team will construct an interview schedule that includes different stakeholders, and then share with USAID the updated lists of interviewees and schedules as meetings/interviews take place and informants are added to/deleted from the schedule. A final list must be sent to USAID within five working days after the USAID Team Planning Meeting. The evaluation team should submit the Work Plan and evaluation methodology (including data collection tools) to the Evaluation Manager for review and approval. USAID reviews and approves the final work plan within 5 working days. Once USAID approves the final Work Plan, the team will request background documents and data from USAID and the implementing partners. For this evaluation, USAID will quickly provide critical contact information of STEP alumni and current recipients to begin the online survey. o Evaluation Design: The evaluation team must submit to the Evaluation Manager an evaluation design prior to the TPM by at least three working days (which will become an annex to the Evaluation report). The evaluation design will include: • (1) A detailed evaluation design matrix that links the Evaluation Questions in the SOW to data sources, methods, and the data analysis plan; • (2) Online survey: as soon as USAID approves the Work Plan and the data collection tools , the evaluation team will begin to develop and deploy the online survey of STEP alumni and current recipients. The survey will remain open for at least five weeks unless additional time is required to boost response rates. There are no deliverables associated with the online survey phase. The output of this phase is cleaned and prepared survey data; • (3) Field work: the evaluation field work will begin as the online survey is almost complete so that the survey data will be cleaned and analyzed before the evaluation team drafts the evaluation report. Field work consists of the document review, site visits, semi-structured interviews, Focus Group Discussions and secondary data collection described in the methodology section of this SOW. At the beginning of the field work the evaluation team will conduct a team planning meeting to plan the evaluation field work, refine field data collection methods and develop interview guides and other field protocols. Before starting data collection, the evaluationteam will meet with USAID to clarify 47 | STEP END OF TERM EVALUATION REPORT USAID.GOV evaluation objectives, review evaluation questions, discuss details of the field work, and obtain approval for the field work plan. The evaluation team will also meet with the various stakeholders (e.g., USAID, MOHESR and recipient universities) early in the field work phase to gain a deeper understanding of the activity prior to beginning data collection. The list of potential interviewees and sites to be visited and proposed selection criteria and/or sampling plan (must include calculations and a justification of sample size, plans as to how the sampling frame will be developed, and the sampling methodology); • (4) known limitations to the evaluation design; and • (5) A dissemination plan. Data collection instruments will be shared with Evaluation COR for review, feedback and/or discussion and approval prior to start offieldwork. o Midterm briefing to USAID OEH, Program offices on the work progress and the status of data collection. o Final OEH/PROGRAM OFFICE Exit Briefing: The evaluation team is expected to hold a final exit briefing prior to leaving the country to discuss the preliminary findings and present the skeletal report. This presentation will be scheduled as agreed upon during the Team Planning Meeting. This briefing is mainly for the OEH Office and the Program office to present preliminary findings. o Final presentation/Mission Debrief: The evaluation team must present their final findings to USAID/Egypt within 15 business days after conducting the field visits. The Mission debriefing must be presented by the Evaluation team leader and it should include a discussion of achievements and issues as well as recommendations for future activities' designs and implementation. The team must consider any USAID /Egypt comments and revise the draft report as appropriate. o Draft Report: The draft evaluation report should be consistent with the guidance provided in Section IX: Final Report Format. The report will address each of the questions identified in the SOW, and include a section on lessons learned, and any other issues the team considers to have a bearing on the objectives of the evaluation. Any such issues can be included in the report only after consultation with USAID. In the Limitations section of the draft final report, the Evaluation Team must disclose how findings for each question were affected due to the unavailability of the Implementing Partner. Also, the team should describe how those limitations were mitigated. The team leader will submit a draft report of findings as per the attached timeline, conclusions and recommendations to QED/SIMPLE for review and comment. QED will submit the draft evaluation report to the USAID Evaluation Manager within 15 business days of the team leader’s departure. The written report must clearly describe findings, conclusions and recommendationsfor future programming. The submission date for the draft evaluation report will be determined in the evaluation work plan however, it shouldn't be later than 45 days after the evaluation start date. Once the initial draft evaluation report is submitted, it must undergo a peer review and the Mission will have 10 business days in which to review and comment on the initial draft, after this point the Evaluation Manager will submit the consolidated comments to the evaluationteam. o Final Report and data sets: The final report must be submitted within seven business days from receiving USAID's comments. The final report must not exceed 30 pages in length (not including appendices, lists of contacts, etc.). The format must include an executive summary, table of contents, list of acronyms, evaluation purpose, methodology, findings, conclusions and recommendations. The report must be submitted initially in English, electronically, and later, an Arabic translation of the executive summary must be submitted within seven business days. o All project data and records will be submitted in full and should be in electronic form in easily readable format, organized and documented for use by those not fully familiar with the intervention or evaluation, and owned by USAID. At the time of submission the final English language report, the survey instruments, interviews and data sets must be submitted on a flash drive to the evaluation USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 48 program manager. USAID/Egypt intends to disseminate the report within USAID and to stakeholders. All quantitative data collected by the evaluation team must be provided in machine-readable, non￾proprietary formats as required by USAID's Open Data policy (see ADS 579). The data should be organized and fully documented for use by those not fully familiar with the activity or the evaluation. USAID will retain ownership of the survey and all datasets developed. o Infographics English and Arabic: After approval of the final report, the evaluation team must submit a maximum of two page infographic summarizing the main findings and other relevant information. USAID will have to review and approve. The infographic must be finalized within 10 working days from approving the evaluation report. B. Reporting Requirements Final Report Format The evaluation final report should include executive summary; background of the local context and the strategies/projects/activities being evaluated; the evaluation purpose and main evaluation questions; the methodology or methodologies; the limitations to the evaluation; findings, conclusions, and recommendations. For more detail, see "How-To Note: Preparing Evaluation Reports" and ADS 20l, USAD evaluation Report Requirements. An optional evaluation report template is available in the Evaluation Toolkit. The executive summary should be 2- 5 pages in length and summarize the purpose, background of the activity being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The executive summary will be translated into Arabic. The evaluation methodology will be explained in the report in detail. Limitations to the evaluation will be disclosed in the report, with particular attention to the limitations associated with the absence of the implementing partner and the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.) The annexes to the report will include: • The Evaluation SOW; • All data collection and analysistools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; • All sources of information properly identified and listed; and • Signed disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of any "statements of difference" regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team. • Summary information about evaluation team members, including qualifications, experience, and role on the team. In accordance with ADS 201, the contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within three months of the evaluation’s conclusion. 49 | STEP END OF TERM EVALUATION REPORT USAID.GOV Criteria to Ensure the Quality of the Evaluation Report Per ADS 201, Criteria to Ensure the Quality of the Evaluation Report, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report. In addition, tools need to be tested prior to deployment in the field. • Evaluation reportsshould represent a thoughtful, well-researched, and well￾organized effort to objectively evaluate the strategy, project, or activity. • Evaluation reportsshould be readily understood and should identify key points clearly, distinctly, and succinctly. • The Executive Summary of an evaluation reportshould present a concise and accurate statement of the most critical elements of the report. • Evaluation reportsshould adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. • Evaluation methodology should be explained in detail and sources of information properly identified. • Limitations to the evaluation should be adequately disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findingsshould be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people's opinions. • Findings and conclusions and lessons learned should be specific, concise, and supported by strong quantitative or qualitative evidence. • If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females. • If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, andspecific. The final evaluation report must be submitted to USAID/Egypt's Evaluation Program Manager in electronic format (Microsoft Word) as well as printed and bound copies(five copiesin English and five in Arabic for the executive summary. The Arabic translation of the executive summary must be submitted to USAID within 7 working days after USAID formal written approval of the evaluation report. The entire report must be no longer than 30 pages, single￾spaced in Times New Roman font, size 12 type font. The evaluation report is not to exceed the 30pages and will serve as the document of reference for creating an Infographics version of the evaluation report. All data and materials are to be surrendered to and will remain the property of USAID. VIII. Evaluation Composition A. Logistics USAID will provide overall direction to the evaluation team, identify key documents, and assist in facilitating a work plan. USAID will facilitate arranging meetings with key stakeholders identified by USAID prior to the initiation of field￾work. The evaluation team is responsible for arranging other meetings as identified USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 50 during the course of this evaluation and advising USAID/ Egypt prior to each of those meetings. The evaluation team is also responsible for arranging transportation as needed for site visits in Cairo. The evaluation team will be responsible for arranging its own work/office space, computers, internet access, printing, and photocopying. The evaluation team is also responsible for procuring and paying for translation services for interviews, reports and any other evaluation related task. Evaluation team members will be required to make their own lodging and travel payments. USAID personnel will be made available to the team for consultations regarding sources and technical issues, before and during the evaluation process. B. Scheduling Work is to be carried out over a period of approximately 16 weeks, o/a beginning of September. Each week will be a 6 working day week. 1. Desk Review and preparation for data collectiontools 2. Team planning meeting with USAID including submission of the draft schedule of data collection interviews, draft methodology, and data analyses plan 3. Submission of final methodology, final schedule of data collection interviews and data analysisplan 4. Midterm debrief with USAID/OEH and program Offices 5. Debriefing session with OEH and Program 6. Debrief session for theMission 7. Draft Report in English (including an Executive Summary) 8. USAID will provide written comments to the Evaluating team as the per the draft report peer review outcome. USAID reviews within 10 business daysfrom draft report submission 9. Final Report in English 10. Executive summary translated into Arabic 11. Developing Infographic Sheets in English and Arabic C. Team Composition: The team will include the following personnel and all efforts should be made for the team to be compromised of a balanced number of male and female members. The evaluation team will include the following areas of expertise: (a)Evaluation in accordance with USAID accepted principles and methods; and (b) Higher education, economic development and employment generation. Key Personnel- International Team Leader: A senior international consultant with 10 years' experience in conducting evaluations for higher education scholarships programs. S/he should be familiar with evaluating higher education programs, particularly student support programs (i.e. scholarship and fellowship programs) and be knowledgeable about the higher education sector and development needs in Egypt. S/he should also have a minimum of 5 years in leading evaluation teams, interpersonal relations and writing skills. 51 | STEP END OF TERM EVALUATION REPORT USAID.GOV The Evaluation Team leader will be responsible for designing and implementing the evaluation and for writing the evaluation report. I.1 The Team Leader will: • Finalize and negotiate with USAID/Egypt the evaluation work plan; • Design the evaluation plan. • Establish evaluation team roles, responsibilities, and tasks; • Facilitate the Team Planning Meeting • Ensure that the logistics arrangements in the field are complete; • Manage team coordination meetings in-country and ensure that team members are working to schedule; • Coordinate the process of assembling individual input/findingsforthe evaluation report and finalizing the evaluation report; • Lead the preparation and conduct presentation of key evaluation findings and recommendations to USAID/Egyptteam prior to departing Egypt. • Design the Evaluation Plan. I.2 Recommended Team Members: Senior-level Economist: A senior international Economist with at least 10 years' experience in evidence-based policy research and in conducting quantitative analyses including assessing returns on investment for higher education programs, particularly student support programs (i.e. scholarship and fellowship programs) and be knowledgeable about the higher education sector and development needs in Egypt. Senior M&E Specialist: Senior M&E specialist with minimum 5 years' experience in monitoring and evaluating and designing education projects, with strong writing skills, excellent understanding of the higher education system in Egypt, as well as familiarity with USAID programs. He / She should also have proven experience in conducting evaluations in Middle Eastern countries, in the areas of employment, vocational training, labor market and gender. Have experience in drafting high quality reports and participate in different evaluation activities and may be assigned specific tasks by the Team Leader as appropriate. He/She will be responsible for focus group interviews with STEP alumni and current students and interviews with relevant officials and recipient universities. Design the online questionnaire and all interview guides and analyze survey data. Team members will also write specific sections of the report. I.3 The Team Members will collaborate in: • Designing the evaluation plan. • Developing a data collection plan. • Conducting field visits, surveys, and interviews. • Collecting the data. • Recording and summarizing the data. • Analyzing the data collected. • Preparing reports and presentations for discussing the findings. Logistics coordinator: Three (3) to five (5) years' experience in handling travel related logistics and providing administrative support. The person should be fluent in written and spoken Arabic. USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 52 I.4 The Logistics Coordinator will: • Provide administrative support to Evaluation Teammembers. • Be responsible for setting up meetings with USAID and stakeholders. ANNEX II: TABLES AND GRAPHS Task/Deliverable International Team Leader (Specialist) Senior Economist Senior M&E Specialist Logistical support Coordinator Travel to Egypt 2 days 0 0 0 Review background documents, draft work plan, methodology and data collection tools 5 days 5 days 5 days 0 Team Planning meeting and meeting with USAID/Egypt 10 days 10 days 10 days 10 Data collection. Includes interviews with key stakeholders (stakeholders and USAID staff) and site visits 21 days 21 days 21 days 21 days Discussion, analysis,and draft evaluation report in country 14 days 14 days 14 days 0 Exit briefing with the OEH Team and Debrief meeting with USAID and key stakeholders (preliminary report and infographic due to USAID); and presentation to Mission 7 days 7 days 7 days 5 Depart Egypt/travel to US 1 day 0 0 0 USAID/Egypt provide comments on draft report and infographic 0 0 0 0 Team revises draft report and draft infographic and submits final to USAID 10 days 10 days 10 days 0 Translation of Executive Summary 0 7 days 3 days Submission of final report to the USAID DEC and DDL 1 day 1 day 1 day Total estimated LOE 71 days 75 days 71 days 36 days 53 | STEP END OF TERM EVALUATION REPORT USAID.GOV PART A: HIGHER EDUCATION INITIATIVE (HEI) MBA SCHOLARS DATABASE Table 1: Number of MBA Scholars Disaggregated by Cohort Number and Return Status COHORT # NON-RETURNEES 1 2 TOTAL NO 49 41 90 YES 4 2 6 TOTAL 53 43 96 Table 2: Distribution of MBA Scholars Disaggregated by their Current Residence Governorates GOVERNORATES N % CAIRO 33 34.4% ALEXANDRIA 10 10.4% GIZA 10 10.4% GHARBEYA 7 7.3% SOHAG 5 5.2% ISMAILIA 5 5.2% DAKAHLEYA 4 4.2% FAYOUM 3 3.1% QALYOUBEYA 3 3.1% MENOUFEYA 2 2.1% SHARKEYA 2 2.1% SUEZ 2 2.1% QENA 2 2.1% KAFR EL SHEIKH 2 2.1% BEHAIRA 1 1.0% DAMIETTA 1 1.0% MENIA 1 1.0% BANI SWAIF 1 1.0% PORT SAID 1 1.0% ASSIUT 1 1.0% TOTAL 96 100.0% USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 54 Figure 1: Percent Distribution of MBA Scholars Disaggregated by their Current Residence Governorates Table 3: Distribution of MBA Scholars Disaggregated by Previous Employer’s Sector Employer’s Sector N % Private 60 62.5% Public 18 18.8% Academic Institutions* 8 8.3% Undefined 5 5.2% NGO 2 2.1% Self-employed 2 2.1% Unemployed 1 1.0% Total 96 100% * Academic Institutions are: 1. Arabic Studies Academy 2. National Organization for Drug Control & Research 3. Kafer el Sheikh Higher Institute 4. Alquds International Language School 5. Nile Modern Language School 6. El Shams Exp. Lang School, M.O.E 7. Haram Academy 8. Helwan University 34% 10%10% 7% 5% 5% 4% 3% 3% 2% 2% 2% 2% 2% 1% 1% 1% 1% 1% 1% 0% 5% 10% 15% 20% 25% 30% 35% 40% Percent Governorates 55 | STEP END OF TERM EVALUATION REPORT USAID.GOV Figure 2: Distribution of MBA Scholars Disaggregated by Previous Employer’s Sector Table 4: Distribution of MBA Scholars Disaggregated by Age Age N % 26 7 7.3% 27 7 7.3% 28 12 12.5% 29 8 8.3% 30 11 11.5% 31 6 6.3% 32 14 14.6% 33 6 6.3% 34 3 3.1% 35 8 8.3% 36 3 3.1% 37 3 3.1% 38 2 2.1% 39 2 2.1% 40 3 3.1% 43 1 1.0% Total 96 100% Mean 31.5 Median 31 63% 19% 8% 5% 2% 2% 1% 0% 20% 40% 60% 80% USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 56 Table 5: Distribution of MBA Scholars Disaggregated by Scholarship Location in United States States N % [CA] California 16 16.7% [OH] Ohio 13 13.5% [MO] Missouri 13 13.5% [NY] New York 12 12.5% [MA] Massachusetts 7 7.3% [CT] Connecticut 6 6.3% [OR] Oregon 6 6.3% [NJ] New Jersey 6 6.3% [NH] New Hampshire 6 6.3% [PA] Pennsylvania 3 3.1% [IL] Illinois 2 2.1% [MI] Michigan 2 2.1% [NE] Nebraska 1 1.0% [FL] Florida 1 1.0% [CO] Colorado 1 1.0% [MN] Minnesota 1 1.0% Total 96 100.0% Table 6: Distribution of MBA Scholars Disaggregated by Area of Study/Specialization Major N % Business Administration 24 25.0% International Business 5 8.3% Management 4 5.2% Management and Leadership 4 5.2% Finance 3 4.2% International Business 3 4.2% Human Resource Management 3 3.1% Marketing 3 3.1% Pharmaceutical Management 2 3.1% Human Resources 2 2.1% Management Program 2 2.1% Human Resources Management 2 2.1% Marketing 2 2.1% Project Management Professional 2 2.1% Health Care management 2 2.1% Supply Chain Management 2 2.1% Human Resource Management 1 2.1% Management Strategy 1 1.0% Global Social Sustainability Entrepreneurship 1 1.0% Project Management 1 1.0% Operations and Supply Chain Management 1 1.0% Public Administration 1 1.0% Information Tech Management 1 1.0% Strategy and International 1 1.0% Global Supply Chain Management 1 1.0% 57 | STEP END OF TERM EVALUATION REPORT USAID.GOV Major N % Strategy and International Business 1 1.0% Project Management 1 1.0% Strategic Marketing 1 1.0% Health Care Management Concentration 1 1.0% Strategy and International Business 1 1.0% Human Resources Management 1 1.0% Sustainability & Environmental Compliance 1 1.0% Information Systems 1 1.0% Business Analytics 1 1.0% International Business and Management 1 1.0% Emphasis in Marketing 1 1.0% Management and Leadership 1 1.0% Enterprise Resource Planning (ERP) with SAP 1 1.0% Healthcare Management 1 25.0% Finance 1 8.3% Marketing International Business 1 5.2% General 1 5.2% Health care Management 1 4.2% Supply Chain Management 1 4.2% Total Quality Management 1 3.1% Environmental sustainability 1 3.1% Accounting 1 3.1% Total 96 100% Table 7: Average GPA for MBA Scholars Disaggregated by Residence Governorates and Cohort Numbers Current Governorates Cohort 1 Cohort 2 Total N Average GPA N Average GPA N Average GPA Greater Cairo & Alexandria 33 3.74 23 3.62 56 3.69 Rest of Governorates 20 3.72 20 3.57 40 3.64 Total 53 3.73 43 3.59 96 3.67 Table 8: Number of Non-Returnees from MBA Scholars Disaggregated by Scholarship Location in United States States N [CA] California 1 [CO] Colorado 1 [IL] Illinois 1 [MO] Missouri 1 [NH] New Hampshire 1 [NY] New York 1 Total 6 USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 58 Table 9: Number of Non-Returnees from MBA Scholars Disaggregated by Age Age N 27 1 28 1 30 1 32 1 35 1 38 1 Total 6 Mean 31.7 Table 10: Number of Non-Returnees from MBA Scholars Disaggregated by Residence Governorates Governorates N Alexandria 1 Cairo 3 Dakahleya 1 Giza 1 Total 6 Table 11: Number of Non-Returnees from MBA Scholars Disaggregated by Area of Specialization Area of Specialization N Business Administration 1 Global Social Sustainability Entrepreneurship 1 Management and Leadership 1 Operations and Supply Chain Management 1 Public Administration 1 Strategy and International 1 Total 6 Table 12: Number of Non-Returnees from MBA Scholars Disaggregated by Previous Employer’s Sector Sector N Private 2 Undefined 4 Total 6 59 | STEP END OF TERM EVALUATION REPORT USAID.GOV Table 13: Number of Non-Returnees from MBA Scholars Disaggregated by College in the United States College N California State University - East Bay 1 Colorado State University 1 Southern New Hampshire University 1 University at Buffalo 1 University of Illinois at Chicago 1 Webster University 1 Total 6 USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 60 PART B: HIGHER EDUCATION INITIATIVE (HEI) STEM SCHOLARS DATABASE Table 14: Number of STEM Scholars Disaggregated by Cohort Number Cohort N One 21 Two 41 Total 62 Table 15: Distribution of STEM Scholars Disaggregated by Current Residence Governorates Governorate N % Cairo 15 24.2% Giza 10 16.1% Dakahlia 5 8.1% Minya 4 6.5% Alex 4 6.5% Gharbia 3 4.8% Qaliyobia 3 4.8% Port Said 2 3.2% Assiut 2 3.2% Menofeya 2 3.2% Sharkeya 2 3.2% Kafr El Sheikh 2 3.2% North Sinai 1 1.6% Beheira 1 1.6% South Sinai 1 1.6% Damietta 1 1.6% Marsa Matrouh 1 1.6% Fayoum 1 1.6% Aswan 1 1.6% Ismailia 1 1.6% Total 62 100.0% 61 | STEP END OF TERM EVALUATION REPORT USAID.GOV Figure 3: Percent Distribution of STEM Scholars Disaggregated by Current Residence Governorates Table 16: Distribution of STEM Scholars Disaggregated by School Types School Type N % STEM 33 53.2% Governmental 18 29.0% Experimental 11 17.7% Total 62 100.0% Figure 4: Percent Distribution of STEM Scholars Disaggregated by School Types 24% 16% 8% 6% 6% 5% 5% 3% 3% 3% 3% 3% 2% 2% 2% 2% 2% 2% 2% 2% 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0% Percent Governorates STEM 53% Governmental 29% Experimental 18% USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 62 Table 17: Distribution of STEM Scholars Disaggregated by Age Age N % 20 8 12.0% 21 28 44.2% 22 19 31.5% 23 5 8.7% 24 2 3.6% Grand Total 62 100.0% Mean 21.4 Median 21.5 Table 18: Distribution of STEM Scholars Disaggregated by College in the United States University N % Illinois Institute of Technology 8 12.9% Rutgers, The State University of New Jersey 7 11.3% University of Arkansas 6 9.7% Montana State University 6 9.7% Arizona State University 5 8.1% University of Tennessee, Knoxville 3 4.8% Milwaukee School of Engineering 3 4.8% Western Michigan University 3 4.8% University of Wyoming 3 4.8% Florida Institute of Technology 3 4.8% Rochester Institute of Technology 3 4.8% St. Cloud State University 3 4.8% University of Missouri Columbia 2 3.2% Wayne State University 2 3.2% Missouri State University 2 3.2% Worcester Polytechnic Institute 1 1.6% University of Oklahoma 1 1.6% University of Kansas 1 1.6% Total 62 100.0% 63 | STEP END OF TERM EVALUATION REPORT USAID.GOV Table 19: Distribution of STEM Scholars Disaggregated by Area of Study/Specialization Major N % Biomedical Engineering 13 21.0% Electrical Engineering 10 16.1% Architectural Engineering 5 8.1% Chemical Engineering 3 4.8% Electrical and Computer Engineering 3 4.8% Mechanical Engineering 3 4.8% Biomedical Sciences 3 4.8% Aerospace Engineering 3 4.8% Biology 2 3.2% Biochemistry and Chemical Biology 2 3.2% Engineering Physics 2 3.2% Nuclear Engineering 2 3.2% Civil Engineering 2 3.2% Aerospace Engineering-Aeronautics 1 1.6% Materials Science and Engineering 1 1.6% Information Systems 1 1.6% Biotechnology 1 1.6% Bioinformatics 1 1.6% Biomedical Science 1 1.6% Computer Engineering 1 1.6% Computer Science 1 1.6% Cell Biology & Neuroscience 1 1.6% Total 62 100.0% Table 20: Average GPA for STEM Scholars Disaggregated by Residence Governorates and Cohort Numbers Current Governorates Cohort 1 Cohort 2 Total N Average GPA N Average GPA N Average GPA Greater Cairo & Alexandria 11 3.52 21 3.62 32 3.59 Rest of Governorates 10 3.59 20 3.39 30 3.45 Grand Total 21 3.55 41 3.51 62 3.52 USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 64 PART C: HIGHER EDUCATION INITIATIVE (HEI) STEP SCHOLARS’ SURVEY Table 21: Distribution of MBA Scholars Disaggregated by Cohort, Age, Birth Governorate, Current Governorate, Marital Status, Employment Status, and Average Cumulative GPA Cohort N % Cohort 1 16 57.1 Cohort 2 12 42.9 Age 26 4 14.3 27 1 3.6 28 4 14.3 29 2 7.1 30 4 14.3 31 3 10.7 32 4 14.3 33 2 7.1 35 1 3.6 36 2 7.1 40 1 3.6 Mean Age 30.54 Birth Governorate Alexandria 4 14.3 Beni Suef 1 3.6 Cairo 9 32.1 Dakahlia 1 3.6 Gharbia 1 3.6 Giza 4 14.3 Ismailia 1 3.6 Kafr El Sheikh 1 3.6 Port Said 1 3.6 Sharqia 1 3.6 Sohag 3 10.7 Suez 1 3.6 Birth Governorate Categories Greater Cairo and Alexandria 17 60.7 Other Governorates 11 39.3 Current Governorate 1. Alexandria 3 10.7 6. Cairo 15 53.6 7. Dakahlia 1 3.6 11. Giza 4 14.3 24. Sharqia 1 3.6 25. Sohag 2 7.1 27. Suez 1 3.6 65 | STEP END OF TERM EVALUATION REPORT USAID.GOV Cohort N % Cohort 1 16 57.1 Cohort 2 12 42.9 Age Outside Egypt 1 3.6 Current Governorate Categories Greater Cairo and Alexandria 22 78.6 Other Governorates 5 17.9 Outside Egypt 1 3.6 Marital Status Never Married 24 85.7 Married 3 10.7 Divorced 1 3.6 Employment Status Employed 16 57.1 Unemployed 10 35.7 System Missing 2 7.1 Mean Cumulative GPA 4.84 Total 28 Table 22: Distribution of STEM Scholars Disaggregated by Cohort, Age, Birth Governorate, Current Governorate, Marital Status, and Average Cumulative GPA Cohort N % Cohort 1 13 52.0 Cohort 2 12 48.0 Age 20 2 8.0 21 10 40.0 22 8 32.0 23 3 12.0 24 2 8.0 Mean Age 21.72 Birth Governorate 1. Alexandria 3 12.0 2. Aswan 1 4.0 4. Beheira 1 4.0 6. Cairo 6 24.0 7. Dakahlia 4 16.0 8. Damietta 1 4.0 10. Gharbia 1 4.0 16. Minya 2 8.0 17. Monufia 1 4.0 USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 66 Cohort N % Cohort 1 13 52.0 Cohort 2 12 48.0 Age 20. Port Said 1 4.0 21. Qalyubia 1 4.0 24. Sharqia 1 4.0 27. Suez 1 4.0 Outside Egypt (Oman) 1 4.0 Birth Governorate Categories Greater Cairo and Alexandria 10 40.0 Other Governorates 14 56.0 Outside Egypt 1 4.0 Current Governorate 1. Alexandria 3 12.0 2. Aswan 1 4.0 6. Cairo 10 40.0 7. Dakahlia 4 16.0 10. Gharbia 1 4.0 16. Minya 2 8.0 17. Monufia 1 4.0 21. Qalyubia 1 4.0 24. Sharqia 1 4.0 26. South Sinai 1 4.0 Current Governorate Categories Greater Cairo and Alexandria 14 56.0 Other Governorates 11 44.0 Outside Egypt 0 0.0 Marital Status Never Married 25 100.0 Mean Cumulative GPA 3.53 Total 25 67 | STEP END OF TERM EVALUATION REPORT USAID.GOV Table 23: Distribution of STEP Scholars (MBA and STEM) by Main Source of Learning about the Program Main Source MBA Scholars Stem Scholars N % N % 1. Family/friends 5 17.9 10 40.0 4. Newspaper 8 28.6 2 8.0 5. Social Media 12 42.9 2 8.0 6. Meeting with AFS/STEP in my home governorate 1 3.6 5 20.0 Other 2 7.1 1 4.0 Total 28 25 Table 24: Average Rating* of the Program by MBA Scholars Opinions About the Program Average Rank Rank Percentage 1 2 3 4 5 A. The STEP program is well known in my community/home governorate 1.70 51.9% 25.9% 22.2% 0.0% 0.0% B. The time between announcement and deadline was sufficient 3.78 3.7% 3.7% 22.2% 51.9% 18.5% C. The selection process for the scholarship gave everyone an equal opportunity (i.e. camps, home visits, English placement, interviews, …etc.) 3.96 0.0% 14.8% 14.8% 29.6% 40.7% D. The scholarship requirements selected those from disadvantaged communities 3.85 3.7% 7.4% 18.5% 40.7% 29.6% * Scale of 1 to 5 (one indicating the lowest and five the highest rating) Table 25: Average Rating* of the Program by STEM Scholars Opinions About the Program Average Rank Rank Percentage 1 2 3 4 5 A. The STEP program is well known in my community/home governorate 2.16 16.0% 56.0% 24.0% 4.0% 0.0% B. The time between announcement and deadline was sufficient 3.64 4.0% 16.0% 16.0% 40.0% 24.0% C. The selection process for the scholarship gave everyone an equal opportunity (i.e. camps, home visits, English placement, interviews, …etc.) 3.84 0.0% 16.0% 16.0% 36.0% 32.0% D. The scholarship requirements selected those from disadvantaged communities 3.56 4.0% 12.0% 28.0% 36.0% 20.0% * Scale of 1 to 5 (one indicating the lowest and five the highest rating) USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 68 Table 26: Average Rating* of STEP Activities, Training, and Services in Terms of Usefulness for MBA Scholars STEP Activities, Training and Services Total N Weighted Average NA Don't Know Rank Percentage 1 2 3 4 5 A. Leadership trainings and workshops 27 3.07 0 0 7.4% 25.9% 29.6% 25.9% 11.1% B. Cultural activities 27 3.26 0 0 11.1% 25.9% 3.7% 44.4% 14.8% C. Career counselling provided by STEP 26 1.85 1 0 57.7% 19.2% 7.7% 11.5% 3.8% D. Job placement support provided by STEP 26 1.77 1 0 61.5% 15.4% 7.7% 15.4% 0.0% E. Pre-departure orientation 26 3.65 0 1 3.8% 11.5% 23.1% 38.5% 23.1% F. Reentry workshops 27 2.96 0 0 22.2% 14.8% 18.5% 33.3% 11.1% G. Your action plan 25 3.28 1 1 4.0% 20.0% 44.0% 8.0% 24.0% H. Internships provided by STEP 26 2.31 1 0 30.8% 34.6% 11.5% 19.2% 3.8% I. Other internships 12 3.17 11 4 25.0% 16.7% 8.3% 16.7% 33.3% J. University placement process by STEP 27 2.96 0 0 11.1% 22.2% 37.0% 18.5% 11.1% * Scale of 1 to 5 (one indicating the lowest and five the most useful) NA: Not Applicable Table 27:. Average Rating* of STEP Activities, Training, and Services in Terms of Usefulness for STEM Scholars STEP Activities, Training, and Services Total N Weighted Average NA Don't Know Rank Percentage 1 2 3 4 5 A. Leadership trainings and workshops 21 3.19 0 1 4.8% 19.0% 42.9% 19.0% 14.3% B. Cultural activities 22 3.77 0 0 0.0% 13.6% 27.3% 27.3% 31.8% C. Career counseling provided by STEP 22 2.27 0 0 45.5% 18.2% 13.6% 9.1% 13.6% D. Job placement support provided by STEP 16 2.06 5 1 43.8% 25.0% 18.8% 6.3% 6.3% E. Pre-departure orientation 21 4.33 0 1 0.0% 4.8% 14.3% 23.8% 57.1% F. Reentry workshops 8 3.38 7 7 0.0% 37.5% 12.5% 25.0% 25.0% G. Your action plan 21 3.33 1 0 9.5% 14.3% 33.3% 19.0% 23.8% H. Internships provided by STEP 22 2.18 0 0 45.5% 22.7% 9.1% 13.6% 9.1% I. Other internships 13 3.54 7 2 15.4% 7.7% 23.1% 15.4% 38.5% J. University placement process by STEP 21 3.14 0 1 9.5% 23.8% 19.0% 38.1% 9.5% * Scale of 1 to 5 (one indicating the lowest and five the most useful) NA: Not Applicable 69 | STEP END OF TERM EVALUATION REPORT USAID.GOV Table 28: Average Rating* of STEP Activities, Training, and Services by MBA Scholars STEP Activities, Training and Services Total N Weighted Average NA Don't Know Rank Percentage 1 2 3 4 5 A. STEP program fully prepared me to study and live in the U.S. as a student 27 3.41 0 0 14.8% 3.7% 29.6% 29.6% 22.2% B. STEP contributed to my professional development 26 3.81 1 0 11.5% 15.4% 3.8% 19.2% 50.0% C. STEP contributed to my personal development 27 4.11 0 0 11.1% 3.7% 0.0% 33.3% 51.9% D. The STEP internships contributed to my professional development 27 2.59 0 0 44.4% 3.7% 18.5% 14.8% 18.5% E. STEP staff were responsive to my needs 27 3.48 0 0 11.1% 3.7% 40.7% 14.8% 29.6% * Scale of 1 to 5 (one indicating the lowest and five highest rate) NA: Not Applicable Table 29: Average Rating* of STEP Activities, Training, and Services by STEM Scholars STEP Activities, Training, and Services Total N Weighted Average NA Don't Know Rank Percentage 1 2 3 4 5 A. STEP program fully prepared me to study and live in the U.S. as a student 22 3.18 0 0 9.1% 13.6% 36.4% 31.8% 9.1% B. STEP contributed to my professional development 22 3.91 1 0 9.1% 4.5% 4.5% 50.0% 31.8% C. STEP contributed to my personal development 22 3.82 0 0 9.1% 9.1% 18.2% 18.2% 45.5% D. The STEP internships contributed to my professional development 22 2.45 0 0 40.9% 13.6% 18.2% 13.6% 13.6% E. STEP staff were responsive to my needs 22 3.64 0 0 0.0% 22.7% 13.6% 40.9% 22.7% F. After STEP has ended, my needs are still being met 8 4.38 8 6 0.0% 0.0% 12.5% 37.5% 50.0% * Scale of 1 to 5 (one indicating the lowest and five highest rate) NA: Not Applicable Table 30: Average Rating* of Usefulness in Terms of MBA Scholars’ Personal Growth STEP Activities Total N Weighted Average NA Don't Know Rank Percentage 1 2 3 4 5 A. Leadership activities and training 25 3.04 2 0 16.0% 16.0% 32.0% 20.0% 16.0% B. Pre-departure workshop 25 3.32 2 0 8.0% 20.0% 24.0% 28.0% 20.0% C. Career and job placement support 23 1.96 4 0 52.2% 13.0% 21.7% 13.0% 0.0% D. Internships 24 2.54 3 0 29.2% 29.2% 12.5% 16.7% 12.5% * Scale of 1 to 5 (one indicating the lowest and five highest rate) NA: Not Applicable USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 70 Table 31: Average Rating* of Usefulness in Terms of STEM Scholars’ Personal Growth STEP Activities Total N Weighted Average NA Don't Know Rank Percentage 1 2 3 4 5 A. Leadership activities and training 22 3.09 0 0 4.5% 27.3% 31.8% 27.3% 9.1% B. Pre-departure workshop 22 3.64 0 0 4.5% 18.2% 13.6% 36.4% 27.3% C. Career and job placement support 21 2.19 1 0 33.3% 33.3% 14.3% 19.0% 0.0% D. Internships 22 2.59 0 0 31.8% 18.2% 18.2% 22.7% 9.1% * Scale of 1 to 5 (one indicating the lowest and five highest rating) NA: Not Applicable Table 32: Average Rating* of Usefulness in Terms of MBA Scholars’ Professional Development STEP Activities Total N Weighted Average NA Don't Know Rank Percentage 1 2 3 4 5 A. Leadership activities and training 26 2.96 1 0 15.4% 19.2% 30.8% 23.1% 11.5% B. Pre-departure workshop 25 3.12 2 0 12.0% 24.0% 20.0% 28.0% 16.0% C. Career and job placement support 24 2.13 3 0 54.2% 8.3% 20.8% 4.2% 12.5% D. Internships 23 2.26 4 0 47.8% 13.0% 17.4% 8.7% 13.0% * Scale of 1 to 5 (one indicating the lowest and five highest rating) NA: Not Applicable Table 33: Average Rating* of Usefulness for Professional Development by STEM Scholars STEP Activities Total N Weighted Average NA Don't Know Rank Percentage 1 2 3 4 5 A. Leadership activities and training 22 3.18 0 0 0.0% 36.4% 27.3% 18.2% 18.2% B. Pre-departure workshop 22 3.27 0 0 13.6% 22.7% 18.2% 13.6% 31.8% C. Career and job placement support 20 2.65 2 0 25.0% 25.0% 15.0% 30.0% 5.0% D. Internships 22 2.91 0 0 27.3% 18.2% 9.1% 27.3% 18.2% * Scale of 1 to 5 (one indicating the lowest and five highest rating) NA: Not Applicable Table 34: Average Rating* of Usefulness of Program Support in Finding or Attempting to Find a Job by MBA Scholars STEP Activities Total N Weighted Average NA Don't Know Rank Percentage 1 2 3 4 5 A. Leadership activities and training 26 2.69 1 0 19.2% 26.9% 30.8% 11.5% 11.5% B. Pre-departure workshop 24 2.46 3 0 20.8% 29.2% 37.5% 8.3% 4.2% C. Career and job placement support 23 2.04 4 0 52.2% 17.4% 17.4% 0.0% 13.0% D. Internships 21 2.14 6 0 42.9% 33.3% 4.8% 4.8% 14.3% * Scale of 1 to 5 (one indicating the lowest and five highest rating) NA: Not Applicable 71 | STEP END OF TERM EVALUATION REPORT USAID.GOV Table 35: Average Rating* of Usefulness of Program Support in Finding or Attempting to Find a Job by for STEM Scholars STEP Activities Total N Weighted Average NA Don't Know Rank Percentage 1 2 3 4 5 A. Leadership activities and training 18 2.83 3 1 5.6% 44.4% 27.8% 5.6% 16.7% B. Pre-departure workshop 18 2.44 3 1 27.8% 22.2% 33.3% 11.1% 5.6% C. Career and job placement support 18 2.67 3 1 16.7% 33.3% 22.2% 22.2% 5.6% D. Internships 18 3.28 3 1 16.7% 11.1% 22.2% 27.8% 22.2% * Scale of 1 to 5 (one indicating the lowest and five highest rating) NA: Not Applicable Table 36: Distribution of STEP Scholars (MBA and STEM) by Their Receipt of a Stipend from STEP MBA Scholars STEM Scholars N % N % Yes 27 96.4 22 88.0 No 1 3.6 3 12.0 Total 28 25 Table 37: Mean Amount of Stipend per Month (in USD) that STEP Scholars (MBA and STEM) Received MBA Scholars STEM Scholars Mean amount (in USD) per month $1,665.20 $1,429.29 Table 38: Distribution of STEP Scholars (MBA and STEM) by Additional Costs Paid (in USD) MBA Scholars STEM Scholars N % N % Yes 7 25.0 7 28.0 No 20 71.4 12 48.0 System Missing 0 0.0 6 24.0 Total 28 25 Mean amount of additional costs $2,865.00 $385.71 Table 39: Average Rating* of Usefulness of Academic Program in the U.S. by MBA Scholars Opinions About the Program Average Rank Rank Percentage 1 2 3 4 5 MBA Scholars 4.30 0.0% 7.4% 7.4% 33.3% 51.9% STEM Scholars 4.43 0.0% 0.0% 4.8% 47.6% 47.6% Total Scholars 4.35 0.0% 4.2% 6.3% 39.6% 50.0% * Scale of 1 to 5 (one indicating the lowest and five the most useful) USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 72 Table 40: Distribution of STEP Scholars (MBA and STEM) by their University Similar Activities MBA Scholars STEM Scholars N % N % Yes 21 77.8 17 81.0 No 6 22.2 4 19.0 Total* 27 21 *Some Scholars did not continue the rest of the survey Table 41: Distribution of STEP Scholars (MBA and STEM) by Activities Provided by both STEP and their University Activities MBA Scholars STEM Scholars N % N % A. Leadership activities 15 53.6 15 71.4 B. Career counseling 18 64.3 15 71.4 C. Job placement 10 35.7 4 19.0 D. Internships 9 32.1 9 42.9 E. Extracurricular activities 9 32.1 14 66.7 F. University’s study and academic skills activities 9 32.1 11 52.4 Total 28 21 Table 42: Distribution of STEP Scholars (MBA and STEM) by Activities Not Provided by their University and Needed STEP’s Support Activities MBA Scholars STEM Scholars Total N % N % N % A. Leadership activities 1 3.6 3 14.3 4 8.2 B. Career counseling 5 17.9 6 28.6 11 22.4 C. Job placement 11 39.3 10 47.6 21 42.9 D. Internships 11 39.3 6 28.6 17 34.7 E. Extracurricular activities 2 7.1 1 4.8 3 6.1 F. University’s study and academic skills activities 5 17.9 5 23.8 10 20.4 Other 3 10.7 4 19.0 7 14.3 Total 28 21 49 Table 43: Distribution of MBA Scholars’ Sources of Support by Importance N Rank Average Rank Percentage 1 2 3 4 5 6 A. STEP/STEP staff 28 2.88 21.4% 7.1% 28.6% 21.4% 3.6% 3.6% B. My family and friends in Egypt 28 3.80 17.9% 10.7% 10.7% 10.7% 10.7% 28.6% C. STEP scholars or friends in the U.S. 28 3.38 7.1% 28.6% 7.1% 14.3% 25.0% 3.6% D. The counseling office at my university 28 3.79 7.1% 14.3% 10.7% 21.4% 21.4% 10.7% E. No one 28 2.91 10.7% 10.7% 3.6% 3.6% 7.1% 3.6% * Scale of 1 to 5 (one indicating the lowest and five the most useful source) 73 | STEP END OF TERM EVALUATION REPORT USAID.GOV Table 44: Distribution of STEM Scholars’ Sources of Support by Importance N Rank Averag e Rank Percentage 1 2 3 4 5 6 A. STEP/STEP staff 25 2.88 21.4% 7.1% 28.6% 21.4% 3.6% 3.6% B. My family and friends in Egypt 25 3.80 17.9% 10.7% 10.7% 10.7% 10.7% 28.6% C. STEP scholars or friends in the U.S. 25 3.38 7.1% 28.6% 7.1% 14.3% 25.0% 3.6% D. The counseling office at my university 25 3.79 7.1% 14.3% 10.7% 21.4% 21.4% 10.7% E. No one 25 2.91 10.7% 10.7% 3.6% 3.6% 7.1% 3.6% * Scale of 1 to 5 (one indicating the lowest and five the most useful source) Table 45: Number of Internships MBA Scholars Completed During and After STEP Program During STEP program After STEP program N Valid 26 26 Missing 2 2 Mean 2.4231 1.5385 Median 2.0000 1.0000 Range 3.00 2.00 Minimum 1.00 1.00 Maximum 4.00 3.00 Table 46: Number of Internships STEM Scholars Completed During and After STEP Program During STEP program After STEP program N Valid 21 17 Missing 4 8 Mean 3.2857 1.3529 Median 3.0000 1.0000 Range 3.00 4.00 Minimum 2.00 1.00 Maximum 5.00 5.00 Table 47: Distribution of MBA Scholars’ Internships by Relevance to their Field of Specialization Is the Internship in Scholar’s Field of Specialization? During STEP program After completing STEP program N % N % Yes 13 54.2 8 72.7 No 11 45.8 3 27.3 Total 24 11 USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 74 Table 48: Distribution of STEM Scholars’ Internships by Relevance to their Field of Specialization Is the Internship in Scholar’s Field of Specialization? During STEP program After completing STEP program N % N % Yes 11 52.4 2 66.7 No 10 47.6 1 33.3 Total 21 3 Table 49: Duration of Internships (in Months) by MBA Scholars During and After STEP Program During STEP program After STEP program N Valid 24 11 Missing 4 17 Mean 4.13 1.45 Median 3.00 1.00 Range 17 3 Minimum 1 1 Maximum 18 4 Table 50: Duration of Internships (in Months) by STEM Scholars During and After STEP Program During STEP program After completing STEP program N Valid 20 3 Missing 5 22 Mean 1.40 1.00 Median 1.00 1.00 Range 2 2 Minimum 1 0 Maximum 3 2 Table 51: Distribution of Usefulness of MBA Scholars’ Internships in their Professional Development N Rank Average Rank Percentage 1 2 3 4 5 During STEP program 24 3.38 12.5% 20.8% 16.7% 16.7% 33.3% After STEP program 11 3.00 8.3% 16.7% 0.0% 8.3% 12.5% * Scale of 1 to 5 (one indicating the lowest and five the most useful) 75 | STEP END OF TERM EVALUATION REPORT USAID.GOV Table 52: Distribution of Usefulness of STEM Scholars’ Internships in their Professional Development N Rank Average Rank Percentage 1 2 3 4 5 During STEP program 20 3.05 25.0% 15.0% 15.0% 20.0% 25.0% After STEP program 3 3.00 33.3% 0.0% 0.0% 66.7% 0.0% * Scale of 1 to 5 (one indicating the lowest and five the most useful) Table 53: Distribution of Support Sources for STEP Scholars (MBA and STEM) to Get STEP Internships MBA Scholars STEM Scholars N % N % A. STEP/STEP staff 9 32.1 12 57.1 B. University 6 21.4 3 14.3 C. Friends 4 14.3 4 19.0 D. I found it/them myself 9 32.1 12 57.1 E. Other (mention) 4 14.3 0 0.0 Total 28 21 Table 54: Average Ranking* by MBA Scholars for their Future Career Development STEP Activities Total N Weighted Average NA Don't Know Rank Percentage 1 2 3 4 5 A. I will use my newly learned skills at work 24 2.42 2 0 45.8% 4.2% 29.2% 4.2% 16.7% B. I am confident I will work within my field of specialization when I finish my studies 24 2.58 2 0 29.2% 20.8% 25.0% 12.5% 12.5% C. I am confident, the job placement and career counseling activities conducted by STEP will help me find work 24 3.63 2 0 4.2% 8.3% 33.3% 29.2% 25.0% D. I do not require further job placement and career counseling activities to enable me to find work 25 3.68 0 1 8.0% 12.0% 24.0% 16.0% 40.0% E. I will use my newly learned skills at work 26 3.65 0 0 7.7% 11.5% 19.2% 30.8% 30.8% F. I am confident I will work within my field of specialization when I finish my studies 22 3.64 2 2 18.2% 9.1% 13.6% 9.1% 50.0% * Scale of 1 to 5 (one indicating the lowest and five highest rate) NA: Not Applicable USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 76 Table 55: Average Ranking* by STEM Scholars’ for their Future Career Development STEP Activities Total N Weighted Average NA Don't Know Rank Percentage 1 2 3 4 5 A. I will use my newly learned skills at work 21 4.33 0 0 4.8% 4.8% 4.8% 23.8% 61.9% B. I am confident I will work within my field of specialization when I finish my studies 20 3.95 0 1 5.0% 5.0% 20.0% 30.0% 40.0% C. I am confident, the job placement and career counseling activities conducted by STEP will help me find work 18 2.67 0 3 16.7% 38.9% 16.7% 16.7% 11.1% D. I do not require further job placement and career counseling activities to enable me to find work 20 2.95 0 1 5.0% 25.0% 40.0% 30.0% 0.0% E. I will use my newly learned skills at work 20 2.95 0 1 20.0% 15.0% 25.0% 30.0% 10.0% F. I am confident I will work within my field of specialization when I finish my studies 19 3.95 0 2 0.0% 10.5% 10.5% 52.6% 26.3% G. I am confident I will find work as my specialization is in demand in the Egyptian labor market 20 2.95 0 1 10.0% 35.0% 20.0% 20.0% 15.0% H. I do not require further job placement and career counseling activities to enable me to find work 20 2.15 0 1 30.0% 35.0% 25.0% 10.0% 0.0% I. I am confident I will work within my field of specialization when I finish my studies 20 3.65 0 1 5.0% 10.0% 25.0% 35.0% 25.0% * Scale of 1 to 5 (one indicating the lowest and five highest rate) NA: Not Applicable Table 56: Distribution of Challenges to Finding a Job Related to their Field of Work by MBA Scholars (MBA and STEM) Aggregated by Current and Birth Governorates Current Governorates Categories Birth Governorates Categories Total Greater Cairo and Alexand ria Other Governora tes Outsid e Egypt Greater Cairo and Alexand ria Other Governora tes Challenges N % N % N % N % N % N % A. Lack of jobs to suit post graduates’ qualifications 7 32 2 40 0 0 7 41 2 18 1 5 71. 4 B. Salaries 5 23 1 20 0 0 2 12 4 36 1 3 61. 9 C. Organizations’ work environment 1 5 2 40 1 10 0 0 0 4 36 1 2 57. 1 D. Other (mention) 7 32 2 40 0 0 7 41 2 18 2 9.5 Total 22 5 1 17 11 28 77 | STEP END OF TERM EVALUATION REPORT USAID.GOV Table 57: Distribution of Challenges to Finding a Job Related to their Field of Work by STEM Scholars (MBA and STEM) Aggregated by Current and Birth Governorates Current Governorates Categories Birth Governorates Categories Total Greater Cairo and Alexand ria Other Governora tes Greater Cairo and Alexand ria Other Governora tes Outsid e Egypt Challenges N % N % N % N % N % N % A. Lack of jobs to suit post graduates’ qualifications 9 52. 9 6 54.5 6 60. 0 9 64.3 0 0.0 1 5 71. 4 B. Salaries 8 47. 1 5 45.5 5 50. 0 8 57.1 0 0.0 1 3 61. 9 C. Organizations’ work environment 7 41. 2 5 45.5 5 50. 0 7 50.0 0 0.0 1 2 57. 1 D. Other (mention) 2 11. 8 0 0.0 1 10. 0 0 0.0 1 100 .0 2 9.5 Total 17 11 10 14 1 21 Table 58: Distribution of Currently Employed MBA Scholars by Job Relevance to their Area of Study Is current job relevant to what studied? N % Yes 13 81.3 No 3 18.8 Total 16 Among MBA scholars who are currently employed (N=16) Table 59: Distribution of Currently Employed MBA Scholars by Whether They are Employed by the Same Organization Prior to Traveling Is current job with the same organization prior to traveling? N % Yes 2 12.5 No 14 87.5 Total 16 Among MBA scholars who are currently employed (N=16) Table 60: Distribution of Currently Employed MBA Scholars by Whether They’re in the Same Position with the Same Organization Prior to Traveling Is it the same position? N % Yes 1 50.0 No 1 50.0 Total 2 Among MBA scholars who are currently employed and work in the same organization prior to traveling (N=2) USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 78 Table 61: Distribution of Currently Employed MBA Scholars by Reasons for Joining their Current Organization Reasons to join the current organization N % A. The organization’s work 9 56.3 B. The organization’s status 6 37.5 C. Salary 4 25.0 D. The position is relevant to my field of study 12 75.0 E. Other (mention) 4 25.0 Total 16 Among MBA scholars who are currently employed (N=16) Table 62: Average Duration of Currently Employed MBA Scholars Between Graduation and Job Placement (in completed months) Duration between graduation and job placement (in completed months) N Valid 16 Missing 12 Mean 6.00 Median 5.50 Std. Deviation 5.465 Range 21 Minimum 1 Maximum 22 Among MBA scholars who are currently employed (N=16) Table 63: Distribution of Currently Employed MBA Scholars by Number of Job Applications Submitted Number of Job applications N % None 0 0.0 1-4 1 6.3 5-9 0 0.0 10-14 2 12.5 15-19 0 0.0 20-24 2 12.5 25+ 11 68.8 Total 16 Among MBA scholars who are currently employed (N=16) 79 | STEP END OF TERM EVALUATION REPORT USAID.GOV Table 64: Distribution of Currently Employed MBA Scholars by Number of Interviews Completed Number of interviews Completed N % None 0 0.0 1-4 5 31.3 5-9 7 43.8 10-14 3 18.8 15-19 0 0.0 20-24 0 0.0 25+ 1 6.3 Total 16 Among MBA scholars who are currently employed (N=16) Table 65: Distribution of Currently Employed MBA Scholars by Number of Job Offers They Refused Number of jobs refused N % None 3 18.8 1-4 9 56.3 5-9 3 18.8 10-14 1 6.3 15-19 0 0.0 20-24 0 0.0 25+ 0 0.0 Total 16 Among MBA scholars who are currently employed (N=16) Table 66: Distribution of Currently Employed MBA Scholars by Reasons for Refusing Job Offers Aggregated by Number of Job Offers Refused Among MBA scholars who are currently employed and refused at least one job offer (N=13) Reasons Total Number of job offers refused 1-4 5+ N % N % N % A. Position(s) didn’t match my qualifications/field of study 10 76.9 6 66.7 4 30.8 B. Salary and/or benefits were inadequate 13 100. 0 9 69.2 4 30.8 C. I did not like the attitude of the person(s) interviewing me 2 15.4 1 7.7 1 7.7 D. I did not think the company was progressive enough 5 38.5 1 7.7 4 30.8 E. Other (mention) 1 7.7 1 7.7 0 0.0 Total 13 9 4 USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 80 Table 67: Distribution of Currently Employed MBA Scholars by Current Gross Monthly Salary (in EGP) Gross Monthly Salary (in EGP) N % ≤ 3000 - 4500 2 12.5 4501 - ≤ 6000 0 0.0 6001 - ≤7500 0 0.0 7501 - ≤ 9000 3 18.8 9001 - ≤12000 6 37.5 12001 - ≤14000 1 6.3 Above 14000 4 25.0 Total 16 Among MBA scholars who are currently employed (N=16) Table 68: Distribution of Currently Employed MBA Scholars by Gross Monthly Salary (in EGP) They Received Prior to Traveling to the U.S. Gross Monthly Salary (in EGP) N % ≤ 3000 - 4500 9 56.3 4501 - ≤ 6000 4 25.0 6001 - ≤7500 1 6.3 7501 - ≤ 9000 1 6.3 9001 - ≤12000 1 6.3 12001 - ≤14000 0 0.0 Above 14000 0 0.0 Total 16 Among MBA scholars who are currently employed (N=16) Table 69: Distribution of Currently Employed MBA Scholars by Belief About How Their Salary Compares with Other MBA Holders Salary Comparison with Other MBA Holders N % 1. Lower 6 37.5 2. The same 3 18.8 3. Higher 1 6.3 I don't Know 6 37.5 Total 16 Among MBA scholars who are currently employed (N=16) Table 70: Distribution of Currently Employed MBA Scholars by Whether There’s a Difference in Salary Scale Between Males and Females at Their Present Company Difference in salary scale between males and females N % Yes 3 18.8 No 7 43.8 I don't know 6 37.5 Total 16 Among MBA scholars who are currently employed (N=16) 81 | STEP END OF TERM EVALUATION REPORT USAID.GOV ANNEX III: EVALUATION METHODS, LIMITATIONS, ISSUES DATA COLLECTION METHODOLOGY The end-of-project STEP performance evaluation was implemented between March 4, 2019 and October 25, 2019 (projected) as a descriptive study using a mixed-methods data collection approach, including both qualitative and quantitative research and statistical analysis of online quantitative questionnaires addressing each evaluation question. Eight distinct categories of stakeholders and beneficiaries were identified, for which individual protocols and survey instruments were created. Forty-two key STEP stakeholders and 158 scholar beneficiaries, including six non-returnees, were included in the population of respondents sought. Two teams working in parallel targeted 82 percent of MBA scholars and 87 percent of STEM scholars identified through purposive sampling. Whereas cooperating ministries (MOE, MOHP, MOHSER), USAID Technical Offices, and STEP subcontracted local partners were all targeted at a 100-percent response rate. Three of six non-returnee scholars were sought, and 21 of an unknown number of employers were also targeted. Table 71: Target stakeholders and data collection tools Two teams of two people, each with one sub-team leader and one M&E specialist, collected all primary data. The team leaders led the question and discussion sessions, while both team members took digitized notes for accuracy and later comparison. M&E team members asked clarifying questions as warranted. The first data collection week focused more heavily on face to face KIIs, in part as a contingency measure to allow increased telephone contacts in the second and third weeks should the F2F meetings prove difficult to arrange. This ended up being a useful strategy as most of those willing to meet face to face were exhausted by the middle of the second week of data collection. The lack of scholars willing to meet face-to-face or discuss their views over the phone was a challenge from the outset. USAID did their best to support the team with two emails to each of the scholars/alumni followed by four emails to each from the team. Gradually, as we spoke with more scholars, an increasing number agreed to follow suit and agree to be interviewed. This occurred partially as a result of our attempts to put the interviewees at ease and keep the mood light. We typically found that by the end of the interview, the interviewee was relaxed, relieved, and even enjoying the experience. At the conclusion of the discussion, we would then ask for their assistance to notify their network of scholars/colleagues and help persuade them to complete and interview as well. We also asked everyone we spoke with to fill out an online questionnaire too, which helped strengthen our statistical analysis. It also enabled the team to reach a number of scholars who were USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 82 simply not going to take the time to speak with us, either face-to-face or by phone. Data collection was accomplished using the following methods: 1. A desk review of project-related documents provided by USAID’s activity COR, using thematic content analysis. The desk review included five days of document analysis followed by continued reviews as documents became available. These included: the 2013 STEP Scope of Work, the Bilateral Agreement, the 2014 Performance Management Plan Narrative, all semi-annual reports from 2014 through 2018, performance management plans (PMP) results, performance indicator reference sheets (PIRs) 2014 through 2018, two data quality assessment reports (DQAs), MBA and STEM application packets and selection criteria, candidate status reports, STEP Project Achievements, The 2018 USAID Education Strategy, IIE’s IFP Alumni Tracking Study, and Egypt’s Vision 2030, among others. The team was informed at the outset of data collection activities of the existence of a comprehensive database of scholar information, which was confirmed by the former CoP toward the end of the evaluation. Access to this information would have been very useful from the outset to cross-tabulate a number of scholar variables, such as scholar performance by different profile characteristics. However, only narrow segments of the database were made available to us, some of them quite late in the evaluation. 2. Key informant interviews (KIIs) were conducted face-to-face (F2F) whenever possible to glean nuanced respondent answers, then via telephone, when F2F interviews were not feasible. Virtually all of the MBA scholars willing to meet F2F were interviewed within the first week-and-a-half of the data collection phase, while all others were interviewed by telephone. All STEM scholars were interviewed by telephone, with the exception of one who happened to be in Cairo during the data collection phase. The online questionnaire tool was piloted for use as a dual-purpose instrument—also for use in telephone KIIs—with the intent of increasing the number of responses for statistical analysis. However, its multiple-choice structure was somewhat complicated for telephone use and, as such, the team reverted back to the original KII (also designed for both F2F and phone interviews), which proved more effective. The KII for MBA and STEM scholars was identical except for several skip-logic questions mainly pertaining to the additional employment information needed from the MBAs. 3. Online questionnaires became an invaluable source of data for the STEP evaluation. While this is not always the case, as response rates tend to be low, the online responses filled a critical gap that developed due to a slower than expected scholar response rate. Without an IP to support the evaluation team, our attempts to reach out to scholars typically went unanswered, particularly by the U.S.-based STEMs, even with precursor emails from USAID announcing our task. After multiple attempts to contact scholars by email, then reaching out via WhatsApp for phone interviews, telephone KIIs and completion of online response rates gradually increased. In order to achieve a critical mass of online responses for statistical evaluation, all KII respondents, whether MBA or STEM, were asked to complete an online questionnaire as well. This tactic proved successful as did engendering the support of KII respondents to inform STEP colleagues that their cooperation and feedback were vital to the program. 4. Group discussions preceded by paper and pencil surveys were originally envisioned as a useful, additional means to obtain qualitative and quantitative data efficiently from the same respondents who, in a group setting, might elevate and more fully articulate key findings. However, the difficulties encountered in securing F2F scholar interviews made assembling small groups virtually impossible, and only one discussion group of three participants was held. Another group was scheduled, but just one participant arrived and was interviewed as a normal F2F KII. 83 | STEP END OF TERM EVALUATION REPORT USAID.GOV Table 72: Data Sources, Collection Methods, and Response Rates Achieved Against Targets. Respondent Type Contact Attempts by Phone and Email KIIs Completed F2F & Telephone Online Completed Target % Completed MBAs 207 (t) 383(e) 46 26 (1 par /4 inc) 74 (of 90) 97% STEMs 253 (e) 11 21 (4 par / 7 inc) 54 (of 62) 57% Ministries 4 4 4 (of 4) 100% Local Partners 6 6 6 (of 6) 100% Employers 15 4 21 (of unknown) 19% Non-Returnees 6 0 1 3 (of 6) 33% USAID 4 5 80% TOTALs 874 85 48 (t) = Telephone Calls (e) = Email (inc) = Incomplete (par) = Partially Complete DATA ANALYSIS & QUALITY ASSURANCE Descriptive statistical analyses have been conducted on the quantitative data, whereas a thematic approach was employed on the qualitative data, to identify explicit and implicit patterns in respondents’ answers relating to the evaluation questions. Data analysis actually began during the data collection phase, with three half-day sessions involving the members of both data collection teams. The sessions were used to compare findings, seek data patterns, agree on questions for emphasis (during continuing KIIs), and begin identifying priority findings. As all data collection tools were designed to address necessary themes arising from each evaluation question, the resulting analyses were comparable across all tools. Data collection tools and protocols were piloted to ensure relevance and clarity, and modifications subsequently made to two of them (the scholars’ key informant interview protocol and the scholars’ online questionnaire). Both quantitative and qualitative findings were sought, ensuring that cross-validation between each type was possible when drawing conclusions and developing recommendations. This was most often done by comparing the responses from KIIs with those of online questionnaires, which resulted in strengthened validity and reliability for the observed findings. However, cross-validation was also accomplished through examination of secondary data. One important example is derived from the scholar application packets (31 MBAs and 31 STEMs), which were invaluable to validate primary data obtained about the application process in greater detail, including: the application scoring and rubric, the interview scoring and rubric, the workshop/camp scoring and rubric; and the English test scoring and process. To ensure data entry accuracy and completeness, both members of both data collection teams entered responses digitally into laptops, which proved very helpful in later comparing and contrasting notes within sub-teams and between teams in near real time. Two further measures were undertaken to mitigate the fact that no implementing partner was available to validate interim findings. An informal meeting was held with STEP’s former CoP, which was very helpful to fill in data gaps and validate findings. In addition, a more formal validation workshop with the Activity COR was also held; the meeting was very useful to discuss and validate what the team perceived as its core preliminary findings. DATA COLLECTION LIMITATIONS AND CHALLENGES 1. Response rates from STEM scholars were 57 percent of the targeted level. This is likely due to: 1) STEMs not knowing or trusting the nature of our intent, as there was no IP introduction made; 2) STEMs not receiving our emails due to spam filtering, and 3) STEMs being too occupied with final exams to respond to the evaluation request. Mitigation efforts included four email contact attempts per scholar by the evaluation team, two email contact attempts by USAID, USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 84 appeals to STEMs who had already completed KIIs to notify other STEM scholars of the importance of their feedback, and giving them the option of simply filling an online questionnaire if they did not have time to speak with the team by telephone. While the combined telephone and online response rates did not achieve their targets, these efforts brought the numbers to a level sufficient for a substantive analysis. 2. Response rates from employers of the MBAs were also well short of the targeted level. This is likely the result of scholars’ reticence to allow us to speak with their employers, as most were relatively new in their jobs and had not yet developed strong relationships. This category of respondent was added as a source of contextual information about MBA profile attractiveness and was never required for data validity. The few responses garnered therefore remain useful. 3. 3. As of the date of this evaluation, none of the STEM scholars had yet completed their four-year degrees and, therefore, had not returned to Egypt. This makes the evaluation premature to gauge program outcomes for STEM students pertaining to leadership, employment, or the likelihood they will remain in Egypt and contribute to high-level development objectives. 4. ROI calculations are only possible at a high level (total scholar component ROI), because detailed cost data per category (MBA or STEM), per university, or according to other parameters were not provided. 5. The team had no access to a comprehensive STEP scholar database, which exists with numerous sortable fields, as referenced in the April 2017 Semi-Annual Report: The database is tailored to project requirements and contains detailed participant data covering all phases including participant application, selection, University Preparation Program (UPP, extra-curricula workshops, academic standing, and internships. The evaluation team requested access to detailed information files on each scholar from program inception during the Team Planning Meeting, which we later learned existed in database form but was never fully released. The information it contained could have greatly enhanced our understanding of scholar specifics such as employment against various isolated factors, e.g., GPA or degree attained. The former CoP was very helpful; however, the team was only able to speak with her near the end of the fifth week of the evaluation. While access to the CoP or the IP were never promised, the information they possessed would have been extremely useful, especially at the desk review stage to enhance the assessment team’s efficiency throughout the process. 6. Some IP information received was handpicked, whereas the evaluation team requested the ability to randomly select certain data for ourselves: Applicant selection packets – 15 MBA rejected, 15 MBA selected, 15 STEM rejected and 15 STEM accepted were requested as randomly selected, but it is not apparent this was undertaken. ANNEX IV: WORK PLAN TIMELINE No. Task Description M ar 23-28 M ar 30- Ap r 4 A pr 6- 11 Ap r 13-18 Ap r 20-25 Ap r 27- M ay 2 M ay 4- 9 M ay 11- 16 M ay 18-23 M ay 25-30 Ju n Jul Au g Sep O ct 3.0 USAID Team Planning M eeting (TPM ) /preparatio n 3.1 Prepare a nd rehearse TPM presentatio n 23 3.2 USAID Te am P lan ning M eetin g (TPM ) 2:00 -- D EL 1 24 3.3 M odify w o rk plan item s resulting from U SAID feedback 25 3.4 Field test data co llectio n to o ls 26 3.5 Subm it final w o rk plan w ith revisions 31 4.0 Field data collection 4.1 Field data co llectio n 25 to 11 5.0 USAID M id Fieldw o rk Debrief 5.2 USAID M id -Field w ork D eb rie f -- D EL 2 4 6.0 Data analysis 6.1 D ata analysis 13 27 6.2 Validatio n m eeting w ith CO R 20 7.0 R eport w riting 7.1 D raft repo rt w riting 16 to 5 7.2 Inco rpo rate U SAID debrief (see 8.3) revisions in draft 25 7.3 Send to Q ED fo r review - AM U S tim e ra pid turna round 2 7.4 M ake draft repo rt revisio ns, add co ntent 3 8.0 USAID End of Eval Prelim D ebrief 8.1 Subm it exec sum m ary and slides (30-35) 20 8.2 Prepare a nd rehearse presentatio n 23 8.3 End o f Evalu atio n In tern al D e b rief -- D EL 3 24 8.4 Revise presentatio n based o n feedback 27 9.0 USAID End of Evaluation D ebrief 9.1 Subm it slides (25-40) to U SAID 29 9.2 Prepare a nd rehearse presentatio n 4 9.3 End o f Evalu atio n M issio n-W ide D eb rie f -- D EL 4 5 10.0 Draft Evaluatio n R esults R eport 10.1 Su bm it d raft rep o rt in Eng lish -- D EL 5 3 11.0 Final Deliverables 11.1 Su bm it fin al re p o rt an d an n exes -- D EL 6 1 11.2 Su bm it d raft A rab ic translation o f e xec sum -- D EL 7 25 11.3 Su bm it d raft in fograph is -- D EL 8 7 11.4 Su bm it an o n ym ized d atasets -- D EL 9 23 11.5 Su bm it fin aL Arab ic exe c su m -- D EL 10 26 11.6 Su bm it fin al in fo grap h ic -- D EL 11 5 11.7 Su bm it d raft A rab ic tran slatio n o f in fograph ic -- D EL 12 19 11.8 Su bm it fin al Arabic tran slatio n of in fo grap h ic -- D EL 13 13 11.9 Po st ap p ro ved repo rt an d info grap hic on D EC -- D EL 14 23 11.10 Su bm it b o un d co pie s o f rep ort/in fo graph ic -- D EL 15 25 KEY: # = period o f activity w ith start an d stop d ate USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 86 ANNEX V: REPORT REFERENCES GENERAL REFERENCES Amal Khairy Amin Mohamed (CAPMAS, Egypt). An Analysis of the Gender Pay Gap in the Egyptian Labour Market Paper prepared for the IARIW-CAPMAS Special. The Bayt.com Middle East and North Africa, Salary Survey 2017, April 2017. Buenviaje1 et. al, Employability and Skills of MBA Graduates from Literature Review as Input to Student Development Program (2016) , Journal of Research in Business and Management, Volume 4 ~ Issue 5(2016) pp: 16-21 ISSN(Online) : 2347-3002 www.questjournals.org. Challenges Faced by Graduate Business Education in Southern Africa: Perceptions of MBA Participants. Temtime and Mmereki, Quality Assurance in Education, 19(2), 2011. Conference: “Experiences and Challenges in Measuring Income, Wealth, Poverty, and Inequality in the Middle East and North Africa” Cairo, Egypt November 23-25, 2015. Employability and Skills of MBA Graduates as Input to Student Development Program, Buenviaje1 et. al, Journal of Research in Business and Management, Volume 4, Issue 5 (2016). Egypt Sustainable Development Strategy, Education and Training Pillar. Egypt Sustainable Development Strategy, Health Pillar. Gaining an Employment Edge: The Impact of Study Abroad on 21st-Century Skills and Career Prospects in the United Stated, IIE, October 2017. General Guidelines for reviewing scholarship applications, University of Washington, Office of Merit Scholarships, Fellowships and Award. In Pursuit of Scarcity: Transnational Sudents, Employability, and the MBA, Waters, J., Environment and Planning A, 41(8), 2009. J. Academic Research International, 4(5). J. L. Waters, In Pursuit of Scarcity: Transnational Students, Employability, and the MBA. Environment and Planning A, 41(8), 2009, 1865-1883. Leveraging Higher Education to Promote Social Justice, Alumni Tracking Study, IIE, March 2019. Leveraging Higher Education to Promote Social Justice: Evidence from the IFP Alumni Tracking Study, Ford Foundation International Fellowships Program, Alumni Tracking Study, Report No. 5, March 2019, Martel, M. Institute of International Education. Macatangay, Tracer Study of BSCS Graduates of Lyceum of The Philippines University from 2004-2009. Academic Research International, 4(5), 2013, 361. Ministry of Higher Education and Scientific Research Strategy in accordance with the Egypt Sustainable Development Strategy 2030. Nihal El-Megharbel, Sustainable Development Strategy: Egypt's Vision 2030 And Planning Reform. 27 May 2015. Study Abroad Matters Linking Higher Education to Contemporary Workplace through International Experience, global education research papers, IIE &AIFS Foundation. Temtime & Mmereki, Challenges Faced by Graduate Business Education in Southern Africa: Perceptions of MBA Participants. Quality Assurance in Education, 19(2), 2011, 110-129. 87 | STEP END OF TERM EVALUATION REPORT USAID.GOV Tracer Study of BSCS Graduates of Lyceum of The Philippines University from 2004-2009, Macatangay, Scholarship Interview Guidelines and Rubric, Texas Public Services Association (TPSA). USAID Education Policy, November 208? USAID. Higher Education Scholarships: A review and summary of findings of nine long-term scholarship programs. June 14, 2. REFERENCES FOR RETURN ON INVESTMENT ACT, STEM Education in the U.S.: Where We Are and What We Can Do? Commonwealth Scholarship Commission in the Commonwealth UK, study of research methodology used in evaluations of international scholarship schemes for higher education. Evaluating Commonwealth Scholarships in the United Kingdom: Assessing Impact in Key Priority Areas. Evaluating Commonwealth Scholarships in the United Kingdom: Assessing Impact in Key Priority Areas￾Executive Summary. Evaluation of HED/USAID Women’s Leadership Program-Rwanda (2012-2015), September 4, 2015. International Scholarship Graduates Influencing Social and Economic Development at Home: The Role of Alumni Networks in Georgia and Moldova, Current Issues in Comparative Education (CICE) Volume 19, Issue 1, Fall 2016. Juan A. B. Belt , STRENGTHENING COST-BENEFIT ANALYSIS IN USAID, 2011-2013: LESSONS LEARNED AND FUTURE DIRECTIONS, March 2014. Maureen Woodhall, Cost-Benefit Analysis in Educational Planning- Fourth Edition, UNESCO International Institute for Educational Planning, Paris 2004. Olivier Vardakoulias, Measuring the Economic Impact of Commonwealth Scholarships: Identifying Methodologies for Cost Benefit Analysis and Value for Money, October 2012 NEF Consulting. USAID, Higher Education for Development Program Closeout Report, October 1, 2005 September 30, 2015. USAID, Mid-Term Evaluation of the Indonisian Higher Education Leadership and Mangement (HELM) project, March 2015. USAID, USAID Lebanon University Scholarship Program Mid-Term Evolution, September 2015. USAID/Egypt, Final Performance Evaluation of the Leadership for Education and Development (LEAD), September 2016. USAID, Feasibility Study of the U.S.-Pakistan Knowledge Corridor Scholarship Program. USAID/Kosovo Transformational Leadership Impact And Performance Evaluation Project Impact Evaluation of the Scholarship Program, Baseline Study, January 2016. USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 88 ANNEX VI: RETURN ON INVESTMENT ROI METHODOLOGY RETURN ON INVESTMENT (ROI) OF STEP SCHOLARSHIPS COMPONENT METHODOLOGY & CALCULATIONS METHOD OF ANALYSIS A. I.A Basic Scope: The Cost-Benefit Analysis (CBA) method of evaluation will be used. CBA analysis will be focused on the perspective of Egypt’s national economy. In other terms, the CBA emphasizes the social (as opposed to the private) viability of investments/recurrent costs incurred in the course of STEP Scholarship Program. The scope of CBA analysis covers: i) stream of the program’s costs; ii) estimates of lifetime earnings of STEP scholars, and iii) the appropriate social discount rate, which reflects the opportunity cost of capital as estimated/used in the context of Egypt’s national planning/policy-making endeavors. B. I.B Limitations: Social/Economic CBA shall not be confined to the direct/monetary estimates of costs and benefits. Rather, the analysis shall extend to cover the indirect/non-monetary elements of both the costs and benefits sides. However, the complexities and potential controversial results of attempts to include estimates of the indirect/non-monetary costs and benefits have led the evaluation team to deal with these types of costs and benefits in a qualitative/descriptive manner. This would enhance the quantitative/monetary analysis. CBA INDICATORS A. II.A Basic Scope: CBA indicators estimated are: Net Present Value (NPV): the difference between the present value of benefits and the present value of costs. Benefit/Cost (B/C) Ratio: Ratio of the present value of benefits to the present value of costs. Internal Rate of Return (IRR)/Return on Investment (ROI): IRR and ROI are synonymous in the context of cost-benefit analysis. It is equal to the discount rate that results in a zero NPV. B. II.B Limitations: Calculations of CBA indicators would not be further subject to benchmarking analysis due to: i) absence of information on alternative programs of investment, which could have used the same resources allocated to STEP or, ii) non-availability of other recent CBA analyses of high-education programs. An exception is the end-of-term evaluation for USAID’s LEAD scholarship program (2016), which estimated program-specific ROI of 11 percent to 19 percent depending on field of study. A corresponding ROI calculation was made for higher education in Egyptian private universities estimated at 18 percent to 21 percent [LEAD End-of-Term Evaluation, 2016]. 89 | STEP END OF TERM EVALUATION REPORT USAID.GOV SOCIAL DISCOUNT RATE A. III.A Basic Scope: The evaluation team approached the Egyptian Ministry of Planning, Monitoring, and Administrative Reform to get the social discount rate used in Egypt’s Sustainable Development Strategy (SDS) 2030, which was updated in 2017. The rate reported is 7 percent. Accordingly, this rate is used for benchmarking results of STEP’s ROI calculations. B. III.B Limitations: No information is available about a higher education-specific social discount rate for Egypt. Investments in higher education are worth high prioritization, hence, a lower social discount rate. The rate provided by the Ministry of Planning is across-the-board for all sectors, i.e. it undermines the specific priority of education in general and higher education in particular. RESULTS OF COST-BENEFIT ANALYSIS/ROI CALCULATIONS A. Summary Results Table 73: Summary: MBA Scholarships Program - STEM Scholarships Program MBA Scholarships Program STEM Scholarships Program Total cost per-scholar (inclusive of tuition fees, and all scholarship-related costs) – U.S.$ 143,598 267,096 CBA Indicators/ROI Estimates Net Present Value (NPV)/Scholar (Social Discount Rate: 7%) US$ 74,938 US$ -11,572 Benefit/Cost Ratio (B/C) 1.58 : 1 0.95 : 1 Internal Rate of Return (IRR) 11% 7% USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 90 B. Detailed Calculations Table 74:STEP/MBA Scholarships Program Cost-Benefit Analysis/ROI Calculations STEP/MBA Scholarships Program Cost-Benefit Analysis/ROI Calculations US$ Year Costs/ Scholar Earnings/Scholar (Projected Salaries + Fringe Benefits) Net Earnings/Scholar -2 71,799 0 -71,799 -1 71,799 0 -71,799 1 0 7,059 7,059 2 0 7,765 7,765 3 0 8,541 8,541 4 0 9,395 9,395 5 0 10,335 10,335 6 0 11,368 11,368 7 0 12,505 12,505 8 0 13,756 13,756 9 0 15,131 15,131 10 0 16,644 16,644 11 0 18,309 18,309 12 0 20,140 20,140 13 0 22,154 22,154 14 0 24,369 24,369 15 0 26,806 26,806 16 0 29,486 29,486 17 0 32,435 32,435 18 0 35,679 35,679 19 0 39,246 39,246 20 0 43,171 43,171 21 0 47,488 47,488 22 0 52,237 52,237 23 0 57,461 57,461 24 0 63,207 63,207 25 0 69,528 69,528 Net Present Value (NPV)/Scholar (Social Discount Rate: 7%) 129,814 204,752 74,938 91 | STEP END OF TERM EVALUATION REPORT USAID.GOV STEP/MBA Scholarships Program Cost-Benefit Analysis/ROI Calculations US$ Year Costs/ Scholar Earnings/Scholar (Projected Salaries + Fringe Benefits) Net Earnings/Scholar Benefit/Cost Ratio (B/C) 1.58 : 1 Internal Rate of Return (IRR) 11% Table 75: STEP/STEM Scholarships Program Cost-Benefit Analysis/ROI Calculations STEP/STEM Scholarships Program Cost-Benefit Analysis/ROI Calculations US$ Year Costs/ Scholar Earnings/Scholar (Projected Salaries + Fringe Benefits) Net Earnings/Scholar -4 66,774 0 -66,774 -3 66,774 0 -66,774 -2 66,774 0 -66,774 -1 66,774 0 -66,774 1 0 8,471 8,471 2 0 9,318 9,318 3 0 10,249 10,249 4 0 11,274 11,274 5 0 12,402 12,402 6 0 13,642 13,642 7 0 15,006 15,006 8 0 16,507 16,507 9 0 18,157 18,157 10 0 19,973 19,973 11 0 21,971 21,971 12 0 24,168 24,168 13 0 26,584 26,584 14 0 29,243 29,243 15 0 32,167 32,167 16 0 35,384 35,384 17 0 38,922 38,922 18 0 42,814 42,814 USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 92 STEP/STEM Scholarships Program Cost-Benefit Analysis/ROI Calculations US$ Year Costs/ Scholar Earnings/Scholar (Projected Salaries + Fringe Benefits) Net Earnings/Scholar 19 0 47,096 47,096 20 0 51,805 51,805 21 0 56,986 56,986 22 0 62,684 62,684 23 0 68,953 68,953 24 0 75,848 75,848 25 0 83,433 83,433 Net Present Value (NPV) - Social Discount Rate: 7% 226,178 214,606 -11,572 Benefit/Cost Ratio (B/C) 0.95 Internal Rate of Return (IRR) 7% 93 | STEP END OF TERM EVALUATION REPORT USAID.GOV ANNEX VII-A: DATA COLLECTION TOOLS (ENGLISH) STEP END-OF-PROJECT PERFORMANCE EVALUATION – DATA COLLECTION TOOLS TOOL #1: KII - GROUP DISCUSSIONS PROTOCOLS – SCHOLARS (MBA & STEM) Date of the meeting: Time of the meeting: Interviewer/ other team members: Notes taken by: Interviewee Names & Titles: Scholar Name: Age: Marital status: Cohort: Original Governorate: Governorate of Residence: University: Area of Study: Cumulative GPA: Introduction Thank you for agreeing to meet with us today. We are conducting an evaluation of the USAID STEP (Scholarships and Training for Egyptian Professionals) program. You have been asked to participate in this discussion because you are considered an important stakeholder in STEP’s activities. USAID/Egypt has contracted with the SIMPLE Project (Services to Improve Performance Management, Enhance Learning and Evaluation) to conduct this evaluation. The purpose of our visit is to gather information about the effectiveness of the USAID funded STEP program to assist USAID in strengthening the design of future programs. As a scholar supported by USAID STEP, we would like to ask you some questions about your experiences under the program. Your responses will be kept confidential and will not be publicly released. The discussion should require not more than 30 minutes, Thank you for your valued contribution to this important initiative. Outreach and Selection 1) How did you hear about the STEP program? (EQ2) 2) Why do you think you were selected for this program? (EQ2) 3) How effective do you think the program’s outreach efforts were in reaching disadvantaged communities? (EQ2) (The definition of disadvantaged is financial need and geographically located in governorates) 4) How would you assess the selection process and criteria, were they fair? Why? Y/N (EQ2) 5) What are your recommendations to improve outreach and selection, if any? (EQ2) USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 94 STEP Program Activities 6) How did STEP affect you personally and professionally? How did it impact you? (EQ1A) 7) This program is all about leadership, what is your definition of leadership? Did STEP help you gain leadership skills? If yes, how? (EQ4B) 8) Which four STEP activities benefitted you the most? And why/how they affected you either personally and professionally?(EQ4B) a. Leadership trainings and workshops b. Cultural activities c. Career counselling provided by STEP d. Job placement support (how to write CV, etc…) provided by STEP e. Pre-departure orientation f. Re-entry workshops g. Your action plan h. Internships provided by STEP i. Other internships j. University placement process by STEP 9) Was there any duplication of these activities at your university? From a similar list, select any activities/services you think the university provided that were equal to or better than those provided by STEP, if any. (EQ4B) a. Leadership trainings and workshops b. Cultural activities c. Career counselling d. Job placement support e. Internships f. Your action plan 10) How was your cultural experience in the US? What were the challenges faced during this experience? Did STEP fully prepare you for that experience? Was anything missing? (EQ4B) 11) How will you implement these newly learned skills and knowledge? At what type of organization? (EQ1A) 12) In what ways do you believe your university experience will impact your professional life after graduation? (EQ1A/EQ4B) 13) Did/does STEP/IIE pay you a stipend? (EQ3A) – yes/no 14) If yes, what was/is the amount per month? (EQ3A) 15) Did you incur any additional direct costs (study related) that were not paid by the program? If yes please mention amount in US dollars. (EQ3A) 16) Who is/was supporting (emotionally/support system) you in the US? (EQ3B) 17) Did you consider dropping out of the program completely at any point? Why? (illness or disability, maternity leave, personal/family responsibilities, academic pressure, others) Did STEP help you to overcome these issues-if any? (EQ3B) 18) What do you believe the reasons for the non-returnees are? How could have STEP have prevented that from happening? (EQ3B) Questions for STEM only 19) Do you think you are fully prepared to find a job in your relevant field? Was there some other support or training you wished you had? (EQ4A/EQ4B) 20) What are your future plans and how do you believe they fit into Egypt’s development (EQ1) 21) Are you currently a member of any networking/scholars network? (EQ1/EQ4B) Questions for MBA only 22) Do you think STEP fully prepared you to find a job in your relevant field? Was there some other support or training you wished you had? (EQ4A/EQ4B) 23) How do you see yourself contributing to Egypt’s development? (EQ1A) 24) Are you currently member of any networking/scholars network? (EQ1/EQ4B) 25) For unemployed, what are the challenges in terms of job placement? 95 | STEP END OF TERM EVALUATION REPORT USAID.GOV For employed MBA only 26) What was the activity that helped you most to land a job? Why? (EQ4) 27) Please indicate your gross monthly salary (in EGP) and benefits package you are currently receiving? A. Gross Monthly Salary Only: 1. ≤ 3000 - 4500 2. 4501 - ≤ 6000 3. 6001 - ≤7500 4. 7501 - ≤ 9000 5. 9001 - ≤12000 6. 12001 - ≤14000 7. Above 14000 B. Benefits description: 28). Please indicate your gross monthly salary and benefits package you used to receive prior to travelling to the US? A. Gross Monthly Salary Only: 1. ≤ 3000 - 4500 2. 4501 - ≤ 6000 3. 6001 - ≤7500 4. 7501 - ≤ 9000 5. 9001 - ≤12000 6. 12001 - ≤14000 7. Above 14000 B. Benefits description: 29) Compared to other MBA holders who received their degrees from Egypt, do you believe your salary is: 1. Lower 2. The same 3. Higher Final Question for all students 30) Thinking about STEP activities and looking at the big picture, in your opinion what aspects STEP programs helped you become more employable (or would be in the future)? Do you think anything was missed to help you become more employable in the Egyptian labor market? (EQ1/EQ4) 31) Do you have any additional comments or recommendations to further improve STEP 32) If we decide to repeat STEP, would you recommend it to your family and friends? Why/Why not? This concludes our questions. Thank you for your valuable inputs. TOOL #2: SCHOLARS’ QUESTIONNAIRE (MBA & STEM) Questionnaire ID: MBA/STEM Cohort # Respondent ID 1 MBA 2 STEM 1st Cohort 1 2nd Cohort 2 USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 96 The data of this survey are confidential and will not be used in non-project purposes Hello, my name is …………………, I work for ………. and I am conducting an evaluation of the USAID STEP (Scholarships and Training for Egyptian Professionals) program. You have been asked to participate in this questionnaire because you are considered an important stakeholder in STEP’s activities. The purpose of this questionnaire is to gather information about the effectiveness of the USAID funded STEP program to assist USAID in strengthening the design of future programs. As a scholar supported by USAID STEP, we would like to ask you some questions about your experiences under the program. I want to reassure you that any information you provide will remain confidential, will not be publicly released and will only be used to improve this program. You have the right to stop the interview at any time and you do not have to answer any question that makes you uncomfortable. I estimate that this questionnaire will take no longer than 20 min. Do you Agree to start this questionnaire?  Yes, I agree → Continue to questionnaire  No, I do not agree → End the Interview 1. Date of Interview: ….. /… /2019 2. Interviewer Name: Interviewer Code: 3. Reviewer: Supervisor: Section 1: Basic Information 101. Name:……………………………………………………………………………………. 102. Age in completed years: 103. Marital Status:: 1. Never Married 2. Married 3. Widowed 4. Divorced 104. Original Governorate: 105. Current Governorate of Residence: 1) Alexandria 2) Aswan 3) Asyut 4) Beheira 5) Beni Suef 6) Cairo 7) Dakahlia 8) Damietta 9) Faiyum 10) Gharbia 11) Giza 12) Ismailia 13) Kafr El Sheikh 14) Luxor 15) Matruh 16) Minya 17) Monufia 18) New Valley 19) North Sinai 20) Port Said 21) Qalyubia 22) Qena 23) Red Sea 24) Sharqia 25) Sohag 26) South Sinai 27) Suez 97 | STEP END OF TERM EVALUATION REPORT USAID.GOV 106. University: ………………………………………………………………………… 107. Area of Study: …………………………………………………………………………… 108. Cumulative GPA: IF you are a STEM scholar → SKIP to Next Section 109. Employment Status (last 12 Months): 1. Employed 2. Self-Employed 3. Unemployed and seeking 4. Unemployed and not seeking IF you don’t Choose Employed (Choice 1 & 2) → SKIP to Next Section 110. Title: ……………………………………………… 111. Employer: …………………………………………….…… 112. Address of Employer: …………………………………………….………… 113. Supervisor Contact Information: A. Name: ………………………………… B. Email: ………………………………… C. Telephone: …………………………… 114. Duration of current employment (in months): Section 2: Outreach and Selection 201. How did you hear about the program? 1. Family/friends 2. AFS Egypt 3. STEP Outreach 4. Newspaper 5. Social Media 6. Meeting with AFS/STEP in my home governorate 8. Other (mention) …………………………………… IF you don’t choose Meeting with AFS/STEP in my home governorate→ SKIP to Q203 202. If you met with STEP/AFS staff in your home governorate. Where was your meeting? 1. Mosque/Church 2. School 3. Center 4. Other (mention) ………………………… 203. Please rate the following statements on a scale of 1 to 5 Rate: One being the lowest and five the highest 1 2 3 4 5 The STEP program is well known in my community/home governorate The time between announcement and deadline was sufficient The selection process for the scholarship gave everyone an equal opportunity (i.e. camps, home visits, English placement, interviews, …etc.) The scholarship requirements selected those from disadvantaged communities (underserved governorates) USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 98 204. Do you have suggestions to improve the outreach and/or selection process? If yes, please specify a) ………………………………………………………………………………………… b) ………………………………………………………………………………………… c) ………………………………………………………………………………………… Section 3: STEP Program Activities and Support Questions 301. Please rate the level of usefulness of the following on a scale of 1 to 5 for STEP activities, training and services Rate: One being the lowest and five the highest 1 2 3 4 5 A. Leadership trainings and workshops B. Cultural activities C. Career counselling provided by STEP D. Job placement support provided by STEP E. Pre-departure orientation F. Re-entry workshops G. Your action plan H. Internships provided by STEP I. Other internships J. University placement process by STEP 302. Please rate the following statements on a scale of 1 to 5 Rate: One being the lowest and five the highest 1 2 3 4 5 A. STEP program fully prepared me to study and live in the US as a student B. STEP contributed to my professional development C. STEP contributed to my personal development D. The STEP internships contributed to my professional development E. STEP staff were responsive to my needs F. For STEM only, if not STEM scholar→ SKIP TO Q303. After STEP has ended, my needs are still being met 303. Which STEP activities were most useful to your personal growth? (1 is least useful, and 5 is most useful) 1 2 3 4 5 A. Leadership activities and training B. Pre-departure workshop C. Career and job placement support D. Internships E. Other (mention) 99 | STEP END OF TERM EVALUATION REPORT USAID.GOV 304. Which STEP activities were most useful to your professional development? (1 is least useful, and 5 is most useful) 1 2 3 4 5 A. Leadership activities and training B. Pre-departure workshop C. Career and job placement support/Counseling D. Internships E. Other (mention) 305. Which STEP activities do you believe were most helpful in finding or attempting to find a job? (1 is least helpful, and 5 is most helpful) 1 2 3 4 5 A. Leadership activities and training B. Pre-departure workshop C. Career and job placement support/Counseling D. Internships E. Other (mention) 306. Did STEP pay you a stipend? 1. Yes 2. No → (SKIP TO Q308) 307. What was the amount (in USD) per month? 308. Did you incur any additional direct costs (study related) that were not paid by the program? If yes please enter amount in US dollars if not please enter 0 Section 4: University Activities Questions 401. Please rate the level of usefulness of your academic program in the US to your professional growth. (One being the lowest and five the highest) 402. Did your university provide similar activities and support to those provided by STEP 1. Yes 2. No → (SKIP TO Q405) 403. Which activities were provided by both STEP and your university? (Mark All that Apply) A. Leadership activities B. Career counseling C. Job placement D. Internships E. Extracurricular activities F. AUC’s study and academic skills activities G. Other (mention) 404. Referring to the same list of activities and support, which ones do you think were most important for STEP to provide, i.e., your university did not offer the services or provide them effectively and you needed STEP’s support? USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 100 (Mark All that Apply) A. Leadership activities B. Career counseling C. Job placement D. Internships E. Extracurricular activities F. University’s study and academic skills activities G. Other (mention) 405. Who of the following do you/did you consider as your most important support system, including who do you talked to when problems arose? (Please Order from 1 to 6) Rank Only A. STEP/STEP staff B. My family and friends in Egypt C. STEP scholars or friends in the US D. The counselling office in my university E. No one F. Other (mention) Section 5: Employment Questions A. During STEP program B. After completing STEP program 501. How many internships did you complete? 502. Was the internship in your field of specialization? (please circle) 1.Yes 2.No 1.Yes 2.No 503. Average duration of the internship? (in months) 504. On a scale from 1-5 (with (1) being the least and (5) the most), How useful to your professional development was the internship? (please circle) 1 2 3 4 5 1 2 3 4 5 505. Who helped you get the STEP internship? (Mark All that Apply) A. STEP/STEP staff B. University C. Friends D. I found it/them myself E. Other (mention) IF you are a MBA scholar → SKIP to Q507 506. Please rate the following statements Strongly Agree Agree Neutral Disagree Strongly Disagree A. I will use my newly learned skills at work 101 | STEP END OF TERM EVALUATION REPORT USAID.GOV B. I am confident I will work within my field of specialization when I finish my studies C. I am confident, the job placement and career counseling activities conducted by STEP will help me find work D. I am confident the leadership activities provided by STEP will help me find work E. I am confident the internships provided by STEP will help me find work F. I am confident I will find work as I am an alumni of a US university G. I am confident I will find work as my specialization is in demand in the Egyptian labor market H. I do not require further job placement and career counseling activities to enable me to find work I. I am confident I will work within my field of specialization when I finish my studies IF you are a STEM scholar → SKIP to Q508 507. Please rate the following statements Strongly Agree Agree Neutral Disagree Strongly Disagree A. The job placement and career counseling activities conducted by STEP helped me find work B. STEP Internships helped me find work C. My field of specialization helped me find work D. The fact I am a graduate of a US university helped me find work E. I use my learned skills from STEP they are supporting my career development F. I require further job placement and career counseling activities to enable me to find work/better opportunities 508. In your opinion, what are/will be the challenges to find a job related to your field of work? (Mark All that Apply) A. Lack of jobs to suit post graduates’ qualifications B. Salaries C. Organizations’ work environment D. Other (mention) IF you are not employed/self-employed MBA scholar → SKIP to Q521 509. Is your current job relevant to what you studied? 1. Yes 2. No 510. Is your current job with the same organization you used to work for prior to travelling? 1. Yes 2. No → (SKIP TO Q512) USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 102 511. If yes, Is it the same position? 1. Yes 2. No 512. What made you a join your current organization? (Mark All that Apply) A. The organization’s work B. The organization’s status C. Salary D. The position is relevant to my field of study E. Other (mention) 513. What was the duration between graduation and job placement (in completed months) 514. How many job applications did you submit? 515. How many interviews did you complete? 516. How many job offers did you refuse? IF none → SKIP to Q518 517. If job offers were refused, please indicate the reasons (Mark All that Apply) A. Position(s) didn’t match my qualifications/field of study B. Salary and/or benefits were inadequate C. I did not like the attitude of the person(s) interviewing me D. I did not think the company was progressive enough E. Other (mention) 518. Please indicate your gross monthly salary (in EGP) and benefits package you are currently receiving? A. Gross Monthly Salary Only: 8. ≤ 3000 - 4500 9. 4501 - ≤ 6000 10. 6001 - ≤7500 11. 7501 - ≤ 9000 12. 9001 - ≤12000 13. 12001 - ≤14000 14. Above 14000 B. Benefits description: 519. Please indicate your gross monthly salary and benefits package you used to receive prior to travelling to the US? A. Gross Monthly Salary Only: 8. ≤ 3000 - 4500 9. 4501 - ≤ 6000 12. 9001 - ≤12000 13. 12001 - ≤14000 103 | STEP END OF TERM EVALUATION REPORT USAID.GOV 10. 6001 - ≤7500 11. 7501 - ≤ 9000 14. Above 14000 B. Benefits description: 520. Compared to other MBA holders who received their degrees from Egypt, do you believe your salary is: 1. Lower 2. The same 3. Higher 4. I don’t know 521. At your present company have you seen any indications for a difference in salary scales between males and females? 1. Yes 2. No 522 . Do you have any suggestions that would improve a future STEP type of program? a) ……………………………………………………………………………………………… b) ……………………………………………………………………………………………… c) ……………………………………………………………………………………………… You have completed the questionnaire. Thank you for your valuable input TOOL #2A: TELEPHONE KII– FROM SCHOLARS’ KII (MBA & STEM) Date of the meeting: Time of the meeting: Interviewer/ other team members: Notes taken by: Interviewee Names & Titles: Scholar Name: Age: Marital status: Cohort: Original Governorate: Governorate of Residence: University: Area of Study: Cumulative GPA: Introduction Thank you for agreeing to meet with us today. We are conducting an evaluation of the USAID STEP (Scholarships and Training for Egyptian Professionals) program. You have been asked to participate in this discussion because you are considered an important stakeholder in STEP’s activities. USAID/Egypt has contracted with the SIMPLE Project (Services to Improve Performance Management, Enhance Learning and Evaluation) to conduct this evaluation. The purpose of our visit is to gather information about the effectiveness of the USAID funded STEP program to assist USAID in strengthening the design of future programs. As a scholar supported by USAID STEP, we would like to ask you some questions about your experiences under the program. Your responses will be kept confidential and will not be publicly released. The discussion should require not more than 30 minutes, Thank you for your valued contribution to this important initiative. USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 104 Outreach and Selection 1) How did you hear about the STEP program? (EQ2) 2) Why do you think you were selected for this program? (EQ2) 3) How effective do you think the program’s outreach efforts were in reaching disadvantaged communities? (EQ2) (The definition of disadvantaged is financial need and geographically located in governorates) 4) How would you assess the selection process and criteria, were they fair? Why? Y/N (EQ2) 5) What are your recommendations to improve outreach and selection, if any? (EQ2) STEP Program Activities 6) How did STEP affect you personally and professionally? How did it impact you? (EQ1A) 7) This program is all about leadership, what is your definition of leadership? Did STEP help you gain leadership skills? If yes, how? (EQ4B) 8) Which four STEP activities benefitted you the most? And why/how they affected you either personally and professionally?(EQ4B) a. Leadership trainings and workshops b. Cultural activities c. Career counselling provided by STEP d. Job placement support (how to write CV, etc…) provided by STEP e. Pre-departure orientation f. Re-entry workshops g. Your action plan h. Internships provided by STEP i. Other internships j. University placement process by STEP 9) Was there any duplication of these activities at your university? From a similar list, select any activities/services you think the university provided that were equal to or better than those provided by STEP, if any. (EQ4B) a. Leadership trainings and workshops b. Cultural activities c. Career counselling d. Job placement support e. Internships f. Your action plan 10) How was your cultural experience in the US? What were the challenges faced during this experience? Did STEP fully prepare you for that experience? Was anything missing? (EQ4B) 11) How will you implement these newly learned skills and knowledge? At what type of organization? (EQ1A) 12) In what ways do you believe your university experience will impact your professional life after graduation? (EQ1A/EQ4B) 13) Did/does STEP/IIE pay you a stipend? (EQ3A) – yes/no 14) If yes, what was/is the amount per month? (EQ3A) 15) Did you incur any additional direct costs (study related) that were not paid by the program? If yes please mention amount in US dollars. (EQ3A) 16) Who is/was supporting (emotionally/support system) you in the US? (EQ3B) 17) Did you consider dropping out of the program completely at any point? Why? (illness or disability, maternity leave, personal/family responsibilities, academic pressure, others) Did STEP help you to overcome these issues-if any? (EQ3B) 18) What do you believe the reasons for the non-returnees are? How could have STEP have prevented that from happening? (EQ3B) Questions for STEM only 19) Do you think you are fully prepared to find a job in your relevant field? Was there some other support or training you wished you had? (EQ4A/EQ4B) 20) What are your future plans and how do you believe they fit into Egypt’s development (EQ1) 21) Are you currently a member of any networking/scholars network? (EQ1/EQ4B) 105 | STEP END OF TERM EVALUATION REPORT USAID.GOV Questions for MBA only 22) Do you think STEP fully prepared you to find a job in your relevant field? Was there some other support or training you wished you had? (EQ4A/EQ4B) 23) How do you see yourself contributing to Egypt’s development? (EQ1A) 24) Are you currently member of any networking/scholars network? (EQ1/EQ4B) 25) For unemployed, what are the challenges in terms of job placement? For employed MBA only 26) What was the activity that helped you most to land a job? Why? (EQ4) 27) Please indicate your gross monthly salary (in EGP) and benefits package you are currently receiving? A. Gross Monthly Salary Only: 15. ≤ 3000 - 4500 16. 4501 - ≤ 6000 17. 6001 - ≤7500 18. 7501 - ≤ 9000 19. 9001 - ≤12000 20. 12001 - ≤14000 21. Above 14000 B. Benefits description: 28). Please indicate your gross monthly salary and benefits package you used to receive prior to travelling to the US? A. Gross Monthly Salary Only: 15. ≤ 3000 - 4500 16. 4501 - ≤ 6000 17. 6001 - ≤7500 18. 7501 - ≤ 9000 19. 9001 - ≤12000 20. 12001 - ≤14000 21. Above 14000 B. Benefits description: 29) Compared to other MBA holders who received their degrees from Egypt, do you believe your salary is: 4. Lower 5. The same 6. Higher Final Question for all students 30)Thinking about STEP activities and looking at the big picture, in your opinion what aspects STEP programs helped you become more employable (or would be in the future)? Do you think anything was missed to help you become more employable in the Egyptian labor market? (EQ1/EQ4) 31)Do you have any additional comments or recommendations to further improve STEP 32)If we decide to repeat STEP, would you recommend it to your family and friends? Why/Why not? This concludes our questions. Thank you for your valuable inputs. USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 106 TOOL #2B: GROUP DISCUSSIONS PROTOCOLS –SCHOLARS’ KII (MBA & STEM) Date of the meeting: Time of the meeting: Interviewer/ other team members: Notes taken by: Interviewee Names & Titles: Scholar Name: Age: Marital status: Cohort: Original Governorate: Governorate of Residence: University: Area of Study: Cumulative GPA: Introduction Thank you for agreeing to meet with us today. We are conducting an evaluation of the USAID STEP (Scholarships and Training for Egyptian Professionals) program. You have been asked to participate in this discussion because you are considered an important stakeholder in STEP’s activities. USAID/Egypt has contracted with the SIMPLE Project (Services to Improve Performance Management, Enhance Learning and Evaluation) to conduct this evaluation. The purpose of our visit is to gather information about the effectiveness of the USAID funded STEP program to assist USAID in strengthening the design of future programs. As a scholar supported by USAID STEP, we would like to ask you some questions about your experiences under the program. Your responses will be kept confidential and will not be publicly released. The discussion should require not more than 30 minutes, Thank you for your valued contribution to this important initiative. Outreach and Selection 1. Please advise the steps and sequence that you undertook for the selection of the program? Beginning with how did you hear about the program (EQ2) (Question no.4 from original protocol) STEP Program Activities 2. This program is all about leadership, what is your definition of leadership? Did STEP help you gain leadership skills? If yes, how? (EQ4B/EQ1) (Question no.7 from original protocol) 3. From these mentioned activities, what were the most important to prepare you for your US experience? Did STEP fully prepare you for that experience? What was missing? (EQ4B/EQ1) - probe before and during (Question no.10 from original protocol) Questions for STEM only 4. Do you think you are fully prepared to find a job in your relevant field? Was there some other support or training you wished you had? (EQ4A/EQ4B) (Question no.19 from original protocol) Questions for MBA only 107 | STEP END OF TERM EVALUATION REPORT USAID.GOV 5. Do you think STEP fully prepared you to find a job in your relevant field? Was there some other support or training you wished you had? (EQ4A/EQ4B) (Question no.22 from original protocol) 6. For unemployed, what are the challenges in terms of job placement? (EQ4) (Question no.25 from original protocol) 7. Do you feel that your earnings are equal to an Egyptian MBA holder? Why? (Question no.29 from original protocol) Final Question for all students 8. Are you currently member of any networking/scholars network? If yes, what activities you are involved? or any other professional/voluntary activities your are involve with? (EQ1/EQ4B)? (Question no.21 for STEM and no. 24 for MBA from original protocol) 9. Do you have any additional comments or recommendations to further improve STEP ? (EQ1) (Question no.31 from original protocol) This concludes our questions. Thank you for your valuable inputs. USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 108 TOOL #3: KII - NON-RETURNEES Date of the meeting: Time of the meeting: Interviewer/ other team members: Notes taken by: Organization Name & Location: Interviewee Names & Titles: Scholar Name: Age: Introduction Thank you for agreeing to meet with us today. We are conducting an evaluation of the USAID STEP (Scholarships and Training for Egyptian Professionals) program. You have been asked to participate in this discussion because of your involvement in the program. USAID/Egypt has contracted with the SIMPLE Project (Services to Improve Performance Management, Enhance Learning and Evaluation) to conduct this evaluation. The purpose of our discussion is to gather information about the effectiveness of the USAID funded STEP program to assist USAID in strengthening the design of future programs. As a scholar supported by USAID STEP, we would like to get your feedback on the activities, how you perceive them and how they contributed to your life plans. Your responses will be kept confidential and will not be publicly released or linked to you directly. This call should not take more than 20 minutes and you are welcome to raise all issues you think are important. Questions 1) From your point of view why were you selected to join the program? Were the selection criteria objective and fair? What aspects/criteria, if any, do you think should change? (EQ2B) 2) STEP conducted several activities before and during the scholarship program (leadership workshops, career guidance and others); have you participated in any of them? Did you find them useful? If Yes, which did you find most useful? Why? If No? Why? (EQ2B) 3) Leadership development was an explicit component in STEP, in your opinion did you gain any leadership skills? If yes, can you give an example of applying/utilizing these skills? (EQ4) 4) Did you face any challenges during the program period in the US? If Yes, what were these challenges? Were you prepared by the Program to face such challenges? If yes, how? (EQ4) 5) In general, did you receive the support you expected from STEP? Please elaborate and recommend how it can be improved in the future. (EQ4) 6) Were there any challenges on the academic level? If yes, what were they? Did you receive support to address them? If yes, who provided you with that support and how (EQ4) 7) Did you make use the services at your university for professional and personal development? (Career counseling, placement, advising, CV development). Do you feel there was an overlap between these services and what you were offered through STEP? Which services were typically more effective (STEP or your university)? (EQ2B) 8) We read the project reports and they show that you took the decision of not coming back to Egypt. The reports mention specific reasons, but we would like to learn what were the reasons behind your decision? Could anything have been done by STEP to change this decision? If Yes, what? Do you plan to come back at any point to work in Egypt? If Yes, when? (EQ 2B) 9) Is it possible to share with us how your life is progressing after finishing your studies? Have you found a job? Is this job related to your field of study? Were the skills you acquired through STEP helpful in getting this job? (EQ4) 10) Do you have any recommendations to help in better design future similar scholarship programs, especially for women? (EQ 2B) This concludes our questions. Thank you for your time and valuable input. TOOL #4: KII - TA BENEFICIARIES 109 | STEP END OF TERM EVALUATION REPORT USAID.GOV (Ministry of Education, Ministry of Health) Date of the meeting: Time of the meeting: Interviewer/ other team members: Notes taken by: Ministry: Interviewee Names & Titles: Introduction Thank you for agreeing to meet with us today. We are conducting an evaluation of the USAID STEP (Scholarships and Training for Egyptian Professionals) program. You have been asked to participate in this discussion because you are considered an important stakeholder in STEP’s activities. USAID/Egypt has contracted with the SIMPLE Project (Services to Improve Performance Management, Enhance Learning and Evaluation) to conduct this evaluation. The purpose of our visit is to gather information about the effectiveness of the USAID funded STEP program to assist USAID in strengthening the design of future programs. STEP activity that has provided technical assistance to the ministry at some point during the past 4 years upon the Ministry’s request. During this interview, we would like to hear your reflection on the technical assistance activities that STEP provided to you and how you perceive the value of these activities in achieving the ministry’s strategic goals. Your responses will be kept confidential and will not be publicly released. The discussion should require between 30 and 45 minutes, Thank you for your valued contribution to this important initiative. Questions 1) What technical assistance activities did the STEP activity provide to your Ministry? Please list the activities you can recall with approximate dates. (EQ5) 2) How do you think STEP’s technical assistance activities conducted with the Ministry contribute to Egypt’s strategic objective of preparing an educated workforce that adequately responds to labor market needs? (EQ1 & EQ5) 3) How did STEP technical assistance activities contribute to your Ministry’s strategic goals? (EQ1 & EQ5) 4) On a scale from 1-5, how would you rate the usefulness of STEP technical assistance activities that were conducted with the Ministry? (5 extremely useful), Please list the activities you can recall and rate each in this manner. (EQ5) 5) In your point of view, what are the critical factors for success for technical assistance activities in USAID￾funded programs? (EQ1) 6) Could you list specific capacity you think were developed in the Ministry as a result of participating in STEP technical assistance activities? (EQ 1 & EQ5) 7) What role did the Ministry play in planning for and/or implementing the TA activities you were involved in during the STEP project? (EQ5) 8) How did/would this help the Ministry in absorbing and sustaining the value of the TA provided to them? (EQ5) 9) How would you advise USAID to improve the technical assistance activities planned for your Ministry under future programs? (EQ5) This concludes our questions. Thank you for your valuable inputs. USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 110 TOOL #5: KII - OFFICIALS IN MOHESR Date of the meeting: Time of the meeting: Interviewer/ other team members: Notes taken by: Organization Name & Location: Involvement with STEP: Interviewee Names & Titles: Introduction Thank you for agreeing to meet with us today. We are conducting an evaluation of the USAID STEP (Scholarships and Training for Egyptian Professionals) program. You have been asked to participate in this discussion because you are considered an important stakeholder in STEP’s activities. USAID/Egypt has contracted with the SIMPLE Project (Services to Improve Performance Management, Enhance Learning and Evaluation) to conduct this assessment. The purpose of our visit is to gather information about the effectiveness of the USAID funded STEP program to assist USAID in strengthening the design of future programs. As you have an employee who graduated with an MBA from a US university, with the support of the USAID STEP program, we would like to ask some questions about your experience with this employee until now. Your responses will be kept confidential and will not be publicly released. The discussion should require between 30 and 45 minutes, Thank you for your valued contribution to this important initiative. Questions 1) Please describe how you were involved with STEP and any events or activities you participated in. (EQ1.B) 2) We would be interested to hear your overall evaluation of the STEP activity, particularly how it serves the strategic goals of the Ministry. Are there any examples you can share to illustrate these? (EQ1.B) 3) The scholarship program focused on two tracks: 1] STEMs and 2] MBAs. In your opinion, are these of equal importance in their ability to contribute to Egypt’s economic growth and development? If Yes how? (EQ1.B) 4) All STEP scholarship recipients are women selected from various governorates. How will the Ministry benefit from this new cadre of skilled women? Are there any plans to capitalize on their skills and knowledge? (EQ1.B) 5) What new skills and knowledge do you anticipate the Ministry will require in future employees and do you think the STEP students fit these requirements? (EQ1.B) 6) From the Ministry’s point of view, are there any changes that should be made to enhance the STEP scholarship program design? If yes, what are they? (EQ1.B) 7) From the Ministry's standpoint, what are Egypt’s priority needs now as they relate to Egypt's development? (EQ1.B) This concludes our questions. Thank you for your time and valuable input. 111 | STEP END OF TERM EVALUATION REPORT USAID.GOV TOOL #6: KII - USAID TECHNICAL OFFICES Date of the meeting: Time of the meeting: Interviewer/ other team members: Notes taken by: Organization Name & Location: Interviewee Names & Titles: Introduction The questions we will ask you now are to get a sense of your perceptions of the STEP activity from the perspective of an interested stakeholder. We would also appreciate the chance to return during the data analysis phase and ask further clarifications during a validation workshop. Overall program questions A) Filter questions for other Technical Offices: Did your Technical Office request any technical assistance from the STEP program between 2014 and 2018 and are you personally aware of the details and outcomes? If either is no skip to Q13. 1) Now that STEP is largely completed, what would you consider the activity’s three or four greatest accomplishments? (EQ1A) 2) In your view, what were the most significant challenges (up to four)? What were the efforts to mitigate these challenges? (EQ1A) 3) As STEP was designed in support of DO22, looking back now, which of the following sub purposes do you think were most important when defining STEP’s success? Please rank them in order of importance. (EQ1A) (select one ranking for each and do not repeat any rankings) a) Sub Purpose 1: Increased knowledge and skills of a pool of professionals capable to advance Egypt’s social & economic development and promote democracy & governance 1 2 3 4 Why this ranking? ____________________________________________________________ b) Sub Purpose 2: Increased professional and leadership skills of Egyptian women in priority sectors 1 2 3 4 Why this ranking? ____________________________________________________________ c) Sub Purpose 3: Strengthened technical, managerial and leadership capabilities in targeted public and private institutions and organizations 1 2 3 4 Why this ranking? ____________________________________________________________ d) Sub Purpose 4: Increased understanding of the US among Egyptian professionals sponsored through training 1 2 3 4 Why this ranking? ____________________________________________________________ For COR only (Questions 4 to 15) 4) In the original scope of work, dated May 9, 2013, there were four components: 1) Scholarships for Science, Technology Engineering, Math, (STEM) and Business Administration [appears to have changed little] 2) Democracy and Governance and Health Master’s Degree Scholarships [appears to not have been implemented] 3) Professional Development and Training Opportunities [included extensive government, civil society, and judiciary capacity building, which was not extensively done] 4) Post-Training Support, Follow-On Activities and Technical Assistance [initially designed to “fill a void” where no technical assistance mechanism existed, whereas TA was mainly used to supplement existing USAID technical office work – and virtually all STEM support supplemented the Health and Education Office]. I’d like to briefly discuss the shift in focus and understand the underlying reasons. MBA and STEM scholars component questions 5) In what ways do you think the returning MBAs will be capable of advancing social and economic development, and democracy and governance in Egypt? Please offer a couple of examples. (EQ1A&B) 6) In your view, how did STEP contribute to meeting the objectives of Egypt’s key government, academic, and private sector organizations and have you received any stakeholder feedback to support these? (EQ1B) 7) Do you believe the outreach and selection process attracted the most suitable program candidates: “women of disadvantaged communities?” Looking back, would you change anything about the way outreach and selection were done? (EQ2A) For STEM_____________________________________________________________ USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 112 For MBAs _____________________________________________________________ 8) Do you believe there are other selection criteria that would need to be considered for future participants in similar programs? (EQ2B) For STEM_____________________________________________________________ For MBAs _____________________________________________________________ 9) In addition to the reasons stated in the reports, are there other reasons you think led to some of the participants not returning to Egypt, and do you believe anything could have been done differently to prevent this from happening? (EQ3B) 10) What are the elements or components of “leadership” USAID expects the MBA and STEM participants to demonstrate and do you see this happening yet? Please offer a few examples. (EQ4B) 11) What efforts are needed now to sustain the results of the MBA and STEM scholars component and what if any activities are underway to ensure this happens? (EQ1A) 12) If not addressed in the previous question, how will the STEM cohorts still studying at US universities be supported now that the activity has closed and who will offer them job seeking assistance? (EQ4A) Technical assistance component questions 13) Do you believe the technical assistance in support of the health sector (conference participation and health outreach) added sufficient value to replicate in future STEP kinds of activity designs? (EQ5) (COR & Health) 14) The Early Growth Learning activities comprised most of the TA offered through STEP; how do you foresee these efforts advancing social and economic development in the broader sense? What measures of sustainability exist? How do they contribute to the institutional capacity? (EQ5) (COR & Education) 15) Do you believe institutional strengthening was achieved? Which institutions and based on what feedback? (EQ5) (COR) 16) How did the mechanism to access STEP services function and what were its strengths and weaknesses? (All) 17) If you were designing STEP today, would you structure the TA component differently? If yes, how and what additional sustainability measures (if any) would you like to see? (EQ5) (All) 18) Finally, and once again in consideration of the overall program, what do you believe are the main lessons learned during the implementation of STEP? (EQ1A) (All) This concludes our questions. Thank you for your time and valuable input. 113 | STEP END OF TERM EVALUATION REPORT USAID.GOV TOOL #7: KII - LOCAL PARTNERS Introduction Thank you for agreeing to meet with us today. We are conducting an evaluation of the USAID STEP (Scholarships and Training for Egyptian Professionals) program. You have been asked to participate in this discussion because you are considered an important stakeholder in STEP’s activities. USAID/Egypt has contracted with the SIMPLE Project (Services to Improve Performance Management, Enhance Learning and Evaluation) to conduct this evaluation. The purpose of our visit is to gather information about the effectiveness of the USAID funded STEP program to assist USAID in strengthening the design of future programs. During this interview, we would like to hear your reflection on the activities your organization was involved in with the STEP project; both the activities that deal with STEP’s scholarship component and the Technical Assistance component, if you were involved with STEP on this project component as well. Your responses will be kept confidential and will not be publicly released. The discussion should require between 45 and 60 minutes, Thank you for your valued contribution to this important initiative. Section A: Questions to local partners who dealt with STEP scholars 1) What has your role involved as a local partner with the STEP project? (tick all relevant roles) (EQ5) a. Outreach, Selection Process, Leadership Training, Cultural Activities, Career Counseling, Job Placement, Recruitment, Re-entry workshop, University Preparation Program, Needs Assessment, Monitoring& Evaluation, Technical Assistance Activities to MOE, Technical Assistance Activities to MOH, Other (please specify) 2) Which of the scholars groups you have worked with (select all relevant answers) MBA Cohort 1 MBA Cohort 2 STEM Cohort 1 STEM Cohort 2 3) What do you believe are the key accomplishments of STEP project? (EQ1) 4) What are the critical success factors for these accomplishments? (EQ1) 5) Do you think the STEP selection criteria and selection process for the scholars were appropriate to select the best potential leaders? Y/N (EQ2) 6) What other selection criteria and/or procedures should be considered with future cohorts? (EQ2) 7) Did you perceive different needs among STEP scholars (e.g. between STEM and MBA groups or within the same group)? Y/N (EQ2) 8) If yes, what were these needs and how did you address them? (EQ2) 9) On a scale from 1-5 (with 5 being highly satisfied), how would you rate your satisfaction with the implementation of the following activities that were provided to STEP scholars (rate all those activities that you were involved in) (EQ1) a. Outreach b. Selection Process c. Leadership Training d. Cultural Activities e. Career Counseling f. Job Placement and Recruitment g. Re-entry workshop h. University Preparation Program i. Needs Assessment j. Monitoring& Evaluation k. Technical Assistance Activities to MOE l. Technical Assistance Activities to MOH Date of the meeting: Time of the meeting: Interviewer/ other team members: Notes taken by: Organization Name & Location: Interviewee Names & Titles: USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 114 m. Other ______________________________________________________ 10) How could these activities be improved for future cohorts? (EQ1) 11) In your opinion, what are the specific individual skills/characteristics that are important for successfully completing STEP scholarship program in US universities? (EQ1) 12) From the previous question, do you think STEP scholars have developed these personal skills or characteristics before travelling to pursue their scholarship programs in US? Yes/NO If not, what skills/characteristics they needed to further develop before travelling on their scholarship program to US? (EQ1) 13) In your opinion, what are the specific skills/characteristics that are important for securing employment in Egypt? (EQ1) 14) From the previous question, do you think STEP scholars have developed these personal skills/characteristics on completing STEP scholarship program? Yes/NO If not, what skills/characteristics they need to further develop after the STEP scholarship program to secure employment? (EQ1) 15) On a scale from 1-5, how likely you think STEP alumni will secure a job related to their field of study? (5 being highly likely) (EQ4) 16) From the previous question, what are the primary reasons for any high or low ratings? (EQ4) 17) Are you familiar with the status of returning scholars’ employment and/or business activities? Y/N (EQ3) 18) If your answer to the previous question is ‘yes’ please give examples. (EQ3) 19) What continuing support will STEP alumni require to secure employment in their area of study? Who should provide this support? (EQ4) 20) Reflecting on your experience with STEP scholars, was there any indication that any of them might drop out or not return to Egypt after completing their program of study in US? (EQ2) Y/N/DK 21) If your answer is “yes” to the previous question, please give examples. (EQ2) 22) Should anything be done with current/future cohorts to help prevent/decrease the number of dropouts/non-returnees? (EQ2) 23) How do you view the role your company/organization played with the STEP project contributed to STEP’s goal of “Strengthening technical, managerial, and leadership capabilities of women in Egyptian public and private institutions” (EQ1)? 24) In your view, did STEP operate in a way to sustain its results? Y/N (EQ1) 25) Are there ways STEP might have operated differently to further improve sustainability? (EQ1) Were you involved in other technical assistance activities with STEP other than working directly with STEP scholars? Yes / No If yes, refer to section B questions; if no, the questionnaire is completed 115 | STEP END OF TERM EVALUATION REPORT USAID.GOV Section B: Additional questions for Local Partners who were involved with technical assistance 26) What technical assistance activities to the Egyptian government were you involved with during the STEP program? (EQ5) 27) In your view, how did institutional capacities develop in the Ministries that participated in technical assistance activities with the STEP project? (EQ5 & EQ1) 28) Could you list individual professional capacity you think were developed in the Ministry of Education/Ministry of Health as a result of participating in STEP technical assistance activities? (EQ5 & EQ1) 29) What role did the government counterpart play in planning for and/or implementing the TA activities you were involved in? (EQ5) 30) How did/would this help them in absorbing and sustaining the value of the TA provided to them? (EQ5) 31) What would you suggest to enhance the value-added from technical assistance activities to the Egyptian government under future USAID programs? (EQ5) USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 116 TOOL #8: KII - EMPLOYER Date of the meeting: Time of the meeting: Interviewer/ other team members: Notes taken by: Organization Name & Location: Organization Type: (Corporate, non-profit, public) Sector: Interviewee Names & Titles: Introduction Thank you for agreeing to meet with us today. We are conducting an evaluation of the USAID STEP (Scholarships and Training for Egyptian Professionals) program. You have been asked to participate in this discussion because you are considered an important stakeholder in STEP’s activities. USAID/Egypt has contracted with the SIMPLE Project (Services to Improve Performance Management, Enhance Learning and Evaluation) to conduct this evaluation. The purpose of our visit is to gather information about the effectiveness of the USAID funded STEP program to assist USAID in strengthening the design of future programs. As you have an employee who graduated with an MBA from a US university, with the support of the USAID STEP program, we would like to ask some questions about your experience with this employee until now. Your responses will be kept confidential; your name and your employee’s name will not be publicly released. The discussion should require between 30 and 45 minutes, Thank you for your valued contribution to this important initiative. Questions 1) How long has the STEP MBA holder been an employee of yours? (EQ1A) < 6 months 6 months – 1 year >1 year – 2 years >2 years 2) What is the employee’s job title? (EQ1A & EQ3A) __________________________ 3) Has the employee been promoted since being hired by your organization? (EQ1B & EQ3A) Y/N 4) Would you say the employee’s position description is directly relevant to their MBA area of specialization? (EQ4A) Y/N (if Yes, skip to Q6) 5) If No to the previous question, do you envision eventually moving this employee into a role that is directly relevant to their MBA area of specialization? (EQ4A) Y/N 6) Do you envision this employee is now or will be in a position of leadership within your organization? (EQ4B) Y/N (if Yes, skip to Q8) 7) If you answered no to the previous question, list up to three types of applied knowledge competencies or skills that this employee needs to improve to ensure a future leadership role? (EQ4B) a)______________ b)______________ c)______________ d) I did not hire this person to assume a leadership role e) DK 8) Do you or have you ever had other MBA employees in your organization? (EQ1A) Y/N (if No, skip to Q11) 9) Comparing the STEP MBA holder to other(s), would you say she came into your organization: (EQ1A) a) At a more advanced skill/competencies/ applied knowledge level than other MBA(s) b) At about the same skill/competencies/ applied knowledge level as other MBA(s) c) At a lower skill/competencies/ applied knowledge level than other MBA(s) d) DK 10) How does the STEP MBA holder’s salary and benefits package compare to other MBA holder(s) at the same level in the organization? (EQ1A & EQ3 A) a) Higher b) About the same c) Lower d) DK 11) What is the gross monthly salary of the STEP MBA? (EQ1A & EQ3A) LE______________ 12) What benefits package does the STEP MBA receive? (EQ3A) 13) Were you personally involved in hiring the STEP MBA and are you aware of the details of the negotiation? (EQ4A) Y/N (if No, skip to Q18) 14) What were the three most important applied knowledge/competencies/skills you were seeking when you hired her? (EQ1B) a)_______________________ b)______________________ c)____________________ 117 | STEP END OF TERM EVALUATION REPORT USAID.GOV 15) All new employees require some training and orientation, but do you believe the employee in question possessed the above three types of applied knowledge/competencies/skills to the level of your expectations? (EQ1B) a) Yes, all of them b) Two of the three c) One of the three d) None of them e) DK 16) Thinking back to when you recruited, interviewed and negotiated with the STEP MBA graduate for a position in your organization, how did you perceive their expectations in terms of: (EQ4A & EQ3A) a) salary and benefit level: aa) realistic ab) unrealistic b) starting position level: ba) realistic bb) unrealistic 17) During the negotiation, what was the difference between the salary requested by the employee and the salary that was eventually agreed? LE ____________ or %_____________ 18) Finally, from your perspective, what types of workforce applied knowledge, competencies, skills, and attitude do you believe Egypt most needs now to further its economic development? (EQ1B) This concludes our questions. Thank you for your time and valuable input. USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 118 TOOL #9: RELEASE FORM Services to Improve Performance Management, Enhance Learning and Evaluation (SIMPLE) Release for use of images taken by the SIMPLE activity, implemented by The QED Group, LLC I consent that visual materials like images of myself, individually, or in group shots, may be used in reports, presentations and/or public channels (such as website) or other communication materials by The QED Group, LLC or the donor of this activity, the U.S. Agency for international Development (USAID). I understand that these type of products and communication materials may be disseminated through various USAID and QED Group media, including the corporate website, reports, intranet portal, social media and in any other media or publication circulated by the QED Group or USAID. I release The QED Group, LLC and USAID from any adverse action that may be created for me personally or for my family through use of the said images. By signing this release, I also understand that I give permission to USAID and the QED Group staff to edit, treat and enhance this material in order to make it optimized for use and as many times as needed, without USAID and The QED Group, LLC having to seek my permission for each individual use. ________________________________________ Name (Print or type) ________________________________________ Signature ________________ Date 119 | STEP END OF TERM EVALUATION REPORT USAID.GOV ANNEX VII-B: DATA COLLECTION TOOLS (ARABIC) تقييم نهاية المدة لبرنامج المنح الدراسية والتدريب للمهنيين المصريين STEP أدوات مجع البياانت STEP Performance Evaluation – Data Collection Tools (Arabic) أ داة رمق )1(: دليل مقابةل مع مجموعة نقاش طلبة منحة STEP ادلراس ية (STEM & MBA( اترخي املقابةل: وقت املقابةل: ال باحث / أعضاء الفريق الآخرون: املدون: امس ال ُ شخاص اذلين أجريت معه املقابةل واللقب امس الطالب: السن: احلاةل الاجامتعية: ادلفعة/Cohort : حمافظة املنشأ : إ حمافظة القامة: اجلامعة: التخصص: التقدير الرتامكي GPA: شكرا كل عىل موافقتك عىل لقائنا ال إ يوم. نقوم بجراء تقيمي لربانمج STEP USAID( املنح ادلراس ية والتدريب للمهن ُ يني املرصيني(. وقد طلب منك املشاركة يف هذه املناق شة لنك تعترب واحدا من املس تفيدين امل همني يف أنشطة مرشوع STEP. منية ادلو للت )خدمات لتح إ سني ا دارة الداء وتعزيز التعمل والتق إ يمي( لجراء هذا ا . تعاقدت الواكةل المريكية لية / مرص مع مرشوع SIMPLE لتقيمي منية ادلول ية ال الغرض من زايرتنا هو التعرف عىل مدى فعالية برانمج STEP مريكية ملساعدهتا يف تعزيز تصممي الربامج املس تقبلية، وتأ يت هذه الزاي لت املمول من هيئة ا إ رة يف اطار اعتبارك ً يف الربانمج، ًا. أساس يا ,وبلتاىل ، نود ان نعرف أراءك حول جتربتك مع الربانمج. س إ تعامل اجابتك برسية اتمة، ولن يمت نرشها علن رشياكً ا كل ملسامهتك الق من امل توقع أن تس تغرق املناقشة 30 مية يف هذه املبادرة الهامة ً دقيقة، شكر . إ العالن عن الربانمج ومعلية الاختيار 2EQ .1 كيف مسعت عن برانمج STEP USAID( املنح ادلراس ية والتدريب للمهنيني املرصيني(. .2 ملاذا مت اختيارك يف هذا الربانمج؟ رجاء توض يح الس باب؟ .3 ما مدي كفاءة أسال إ يب العالن عن امل إ نحة للوصول ايل الطلبة من اجمل متعات القل حظا؟ .4 كيف تقمي معلية ومعايري القبول يف الربانمج؟ هل اكنت عادةل؟ ملاذا؟ .5 ما يه توصياتك لتحسني معايري ومعل إ ية العالن عن املنحة، والاختيار من بني املتقدمني؟ USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 120 نشاطات برانمج STEP .6 كيف اثر عليك الربانمج عىل اجلانبني الشخيص واملهين؟ كيف تغريت؟ (A1EQ( .7 جزء كبري من الربانمج يركز عيل همارات القيادة، ما هو تعريفك ملهارات القيادة؟ كيف ساعدك الربانمج عىل اكتساب تكل املهارات؟ (B4EQ( .8 أي من نشاطات برانمج منحةSTEP USAID اس تفدت مهنا اكرث؟ كيف اثرت فيك؟ (B4EQ( منية القيادة ت a. نشاطات وتدريبات b. النشاطات الثقافية c. إ الرشاد املهين املقدم من برانمج STEP d. خدمات املساندة للحصول عىل وظيفة، واملقدمة من برانمج STEP e. ورشة العمل التعريفية قبل السفر orientation departure-Pre Re-entry ورشة .f g. خطة العمل/النشاط plan action اخلاصة بك h. التدريب العميل )internships )مقدم من الربانمج i. التدريبات العمل ية الخرى j. ادلمع اخلاص بلتعيني يف اجلامعة .9 هل اكن ه ناك أي تكرار بني تكل النشاطات يف جامعتك؟ اخرت من القامئة اخلدمات املقدمة من جامعتك واليت تراها اكنت افضل ويف نفس مس تو ي الربانمج. (B4EQ( منية القيادة ت a. نشاطات وتدريبات b. النشاطات الثقافية c. إ الرشاد املهين املقدم d. خدمات املساندة للحصول عىل وظيفة e. خطة العمل/النشاط plan action اخلاصة بك f. التدريب العميل .10 كيف اكنت جتربتك الثقافية يف الولايت املتحدة؟ هل ساعد الربانمج عيل تأ هيكل بصورة اكمةل خلوض تكل التجربة والتحدايت؟ هل اكن هناك قصور ما ىف هذا الشأ ن؟ (EQ4B) .11 كيف تنوي تطبيق املهارات واملعارف املكتس بة؟ ويف أي نوع مؤسسة؟ (A1EQ( .12 كيف تري تأ ثري جتربتك اجلامعية عيل حياتك املهنية بعد التخرج؟ (B4EQ/A1EQ( ً هل اكن الربانمج يدفع كل مرصوفا للنفقات النرثية ؟ (A3EQ( .13 .14 إا إ ذا اكنت الجابة "نعم"، ماذا اكنت القمية الشهرية؟ (A3EQ( .15 هل اكنت ه إ ناك مصاريف اضافية )خاصة بدلراسة( مل يغطها الربانمج؟ (A3EQ( .16 من اكن يقوم مبساندتك مع ً نواي وخالل وجودك يف الولايت املتحدة؟ (B3EQ( .17 هل فكرت يف الان سحاب من الربانمج يف أي وقت؟ ملاذا؟ إ )مرض او اعاقة ،اجازة وضع، ظروف خشصية اوعائلية٬ حتدايت أاكدميية، صعوبت يف ا اخري( كيف ساعدك الربانمج يف هذا إ الطار للتغلب عيل هذه التحدايت-ان وجدت؟ (B3EQ( .18 يف رأ يك، ما يه أس إ باب عدم عودة بعض ادلارسني اىل مرص بعد انهتاء الربانمج؟ ماذا اكن ميكن ان ي إ تخذه الربانمج من اجراءات ملنع ذكل من احلدوث؟ (B3EQ( أ س ئةل خاصة بداريس برانمج العلوم والرايضيات والهندسة STEM .19 يف رأيك، هل اهكل الربانمج للحصول عيل فرصة معل يف جمال ختصصك ادلرايس؟ هل اكن هناك دمع او ن إ شاط اضايف اكن بودك احلصول عليه؟ (B4EQ/A4EQ( منية الاقتصادية والاجامتعية مبرص؟ (1EQ( لت .20 ما يه خططك املس تقبلية؟ وكيف تدخل يف اطار ا .21 هل انت حال يا عضو يف أي مجموعة خرجيني او طلبة هبدف التواصل والتشبيك؟ (B4EQ1/EQ( أ س ئةل خاصة بدارىس ماجس إ تري ا دارة العامل MBA .22 يف رأ يك، هل أسهم الربانمج ىف تأ هيكل للحصول عيل فرصة معل يف جمال ختصصك ادلرايس؟ هل اكن هناك دمع او ن إ شاط اضايف اكن بودك احلصول عليه؟ (EQ4A/EQ4B) منية الاقتصادية والاجامتعية مبرص؟ (A 1EQ( لت .23 كيف تري نفسك مشاراكً يف جمال ا .24 هل انت حال ًيا عضو يف أي مجموعة خرجيني او طلبة هبدف التواصل والتشبيك؟ (B4EQ1/EQ( .25 للطلبة غري العاملني: ما يه التحدايت اليت تواهجك يف احلصول عيل فرصة معل؟ 121 | STEP END OF TERM EVALUATION REPORT USAID.GOV لطلبة الMBA العاملني .26 ما اكرث نشاط ساعدك يف الالتحاق بسوق العمل؟ ملاذا؟ (4EQ( .27 رجاء حتديد املرتب الاجاميل واملزااي الشهرية اليت حتصل علهيا يف معكل احلايل؟ )بجلنيه املرصي( (A3EQ( a. املرتب : 4500 ≥ - 3000 .22 6000 ≥ - 4501 .23 7500 ≥ - 6001 .24 9000 ≥ - 7501 .25 12000 ≥ - 9001 .26 14000 ≥ - 12001 .27 .28 أعىل من 14000 b. املزااي: .28 إ رجاء حتديد املرتب الجاميل واملزااي الشهرية يف معكل السابق قبل السفر ايل الولايت املتحدة ؟ )بجلنيه املرصي(( a. املرتب : 4500 ≥ - 3000 .29 6000 ≥ - 4501 .30 7500 ≥ - 6001 .31 9000 ≥ - 7501 .32 12000 ≥ - 9001 .33 14000 ≥ - 12001 .34 .35 أعىل من 14000 b. املزااي: .29 مقارنة حاميل ماجس تري او MBA من جامعات مرصية، هل مرتبك: (A3EQ( a. اقل b. نفس املس توي c. أعيل d. ل اعرف ال سؤال الخري للك الطلبة .30 عند التفكري بن شاطات الربانمج وبصورة عامة، يف رأيك ما يه اوجه الربانمج اليت ساعدتك يف احلصول عيل وظيفة )او ستساعدك يف املس تقبل(؟ هل تظن انه اكن هناك يشء انقص ملساعدتك يف توظيفك يف سوق العمل املرصي؟ .31 هل دليك أ ية اقرتاحات أو توصيات لتحسني أآداء الربانمج؟ .32 لو مت تكرار الربانمج هل س توصيه لعائلتك واصدقائك؟ ملاذا/ملاذا ل؟ هناية الس ئةل، نشكرك لآرائك القمية.... USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 122 رمق الاس امترة: STEM/MBA رمق ادلفعة رمق املس تجيب MBA 1 STEM 2 1 ادلفعة الوىل 2 ادلفعة الثانية 123 | STEP END OF TERM EVALUATION REPORT USAID.GOV أ داة رمق )2(: اس امترة مسح التقيمي ملنحة STEP ادلراس ية )STEM & MBA) بياانت هذا املسح رسية ولن تس تخدم يف غري أ غراض املرشوع مرحبا اان امسي........ و امعل دلي.......لغرض تقيمي برانمج STEP USAID( املنح ادلراس ية والتدريب للمهنيني املرصيني(. املناق شة ل امل همني يف أنشطة مرشوع STEP. نك تعترب واحدا من ا ُ لقد طلب منك املشاركة يف هذه ملس تفيدين املمول من الواكةل المريكية للت الغرض من زايرتنا هو مجع معلومات حول فعالية برانمج STEP منية ادلولية ملساعدهتا يف تعزيز تصممي الربامج املس تقبلية، وتأ يت هذه الزايرة يف ًا . بعتبارك أساس ًيا يف الربانمج ونود ان نعرف أراءك حول جتربتك مع املرشوع. س تظل أآراؤك رسية ولن يمت نرشها علن رشياكً ونود التنويه بأ ن مشاركتك يف هذا البحث طواع إ ية، ا إ ذ كل احلق يف رفض ال جابة عن أ ي سؤال أو ال إ توقف عن ال جابة يف أي وقت. وطبعا ب إ نحافظ عىل رسية الجابت. هل دليك أس ئةل قبل بدء املناقشة؟ من امل توقع أن يس تغرق هذا الاس تبيان حوايل 15 دقيقة. ا كل ملسامهتك القمية يف هذه املبادرة الهامة. ً شكر كيد املوافقة تأ  أوافق عىل املقابةل  ل أوافق عىل املقابةل  أهناء املقابةل .1 اترخي املقابةل: /...... /..... 2019 .2 امس الباحث كود الباحث: .3 املراجع: املرشف: الق سم الول: الب ياانت الساس ية -101 الامس: ................................................................................................. -102 السن بلس نوات التامة: . أ .2 مزتوج .3 ارمل .4 مطلق -103 احلاةل الاجامتعية 1 عزب USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 124 -104 احملافظة )ماكن امليالد( -105 احملافظة حمل السكن احلايل 125 | STEP END OF TERM EVALUATION REPORT USAID.GOV .1 إ السكندرية .2 أسوان .3 أس يوط .4 البحرية .5 بين سويف .6 القاهرة .7 ادلقهلية .8 دمياط .9 الفيوم USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 126 .10 الغربية .11 اجلزية .12 إ السامعيلية .13 كفر الش يخ .14 القرص .15 مطروح .16 املنيا .17 املنوفية .18 الوادي اجلديد .19 شامل سيناء .20 بورسعيد .21 القليوبية .22 قنا .23 ال بحر المحر .24 الرشقية .25 سوهاج .26 جنوب سيناء .27 السويس 127 | STEP END OF TERM EVALUATION REPORT USAID.GOV USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 128 -106 اجلامعة: .................................................................................... -107 جمال ادلراسة: ........................................................................ -108 اجملموع الرتامكي GPA: اذا كنت من منحة STEM  ان إ تقل اىل القسم التايل -109 احلاةل العملية )أآخر 12 شهر(: .1 يعمل .2 صاحب معل .3 ليعمل و يبحث .4 ل يعمل ول يبحث اذا اكن الاختيار 3 أو 4  ان إ تقل اىل القسم التايل -110 املسمى الوظيفي: ...................................................... -111 هجة العمل: ...................................................... -112 عنوان هجة العمل: ................................................................................. -113 بياانت املدير املبارش: أ. الامس: ...................................................... ب. الامييل: ......................................................... ت. التليفون: .................................................... -114 مدة العمل احلايل )بلشهور(: القسم ال إ ثاين: العالن عن الربانمج ومعلية الاختيار -201 كيف مسعت عن الربانمج؟ .1 العائةل/الصدقاء AFS Egypt .2 .3 دعاية STEP .4 اجلرائد .5 مواقع التواصل الاجامتعي .6 اجامتع مع برانمج STEP/Egypt AFS يف حمافظة املنشأ .7 اخري )رجاء التحديد(.................................. اذا اكن الاختيار لي إ ست الجابة 6  انتقل اىل سؤال 203 -202 هل تقابلت مع فريق العاملني بـ AFS/STEP ىف حمافظتك )حمافظة املنشأ ( .1 املسجد/الكنيسة .2 املدرسة .3 مركز تعلميي .4 أخرى )تذكر(.................................... 129 | STEP END OF TERM EVALUATION REPORT USAID.GOV رجاء تقيمي ال ١إايل ٥ ) ١ القل و ٥ -203 رجاء تقيمي البياانت التالية عيل مقياس من ١إايل ٥ بياانت التالية عيل مقياس من العيل( 5 4 3 2 1 أ. برانمج STEP معروف يف جممتعي/حمافظيت ب. الوقت بني اعالن املنحة واخر ميعاد للتقدمي اكن اكف ًيا ج. معلية الاخت يار أعطت لك خشص فرصة عادةل )املعسكر، الزايرات املزنلية، حتديد مس توي اللغة إ الجنلزيية، املقابالت اخل....( د. معايري املنحة مضنت اختيار الطل بة من احملافظات القل حظا -204 هل دليك اقرتاحات لتح إ سني العالن عن الربانمج ومعلية الاختيار؟ لو نعم رجاء التحديد أ. ......................................................................................................... … ب. ......................................................................................................... … ج. ......................................................................................................... … القسم الثالث: أ س ئةل خاصة بنشاطات ودمع برانمج STEP رجاء تقيمي ال مقياس من ١إايل ٥ ) ١ القل و ٥ -301 رجاء تقيمي مدي الاس تفادة من نشاطات وتدريبات وخدمات برانمج STEP عيل مقياس من ١ ايل ٥ بياانت التالية عيل العىل( 5 4 3 2 1 أ. منية القيادة ت نشاطات وتدريبات ب. النشاطات الثقافية ج. إ الرشاد املهين املقدم من STEP د. خدمات دمع الالتحاق بلوظائف املقدمة من STEP ه. ورشة العمل للتوجيه قبل السفر Predeparture و. ورشة العمل entry-Reعند العودة ز. خطة العمل اخلاصة بك ح. التدريب العميل )Internship )املقدم من الربانمج ط. التدريبات العمل ية الخرى ي. النشاطات اخلاصة بلتعيني يف اجلامعة UPPاملقدم من الربانمج رجاء تقيمي ال ١إايل ٥ ) ١ القل و ٥ -302 رجاء تقيمي البياانت التالية عيل مقياس من ١ ايل ٥ بياانت التالية عيل مقياس من العىل( 5 4 3 2 1 أ. برانمج STEP أهلىن بشلك اكمل لدلراسة يف الولايت املتحدة الامريكية كطالب منييت عيل املس توي املهين ت ب. سامه الربانمج يف USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 130 منييت عيل املس توي الشخيص ت ج. سامه الربانمج يف د. منييت همن ًيا ت سامه التدريب العميل اخلاص بلربانمج يف ه. موظفي STEP اكنوا يس تجيبون لحتياجايت و. لطلبة STEM فقط، ان مل تكن طال ًبا رجاء الانتقال اىل سؤال .٣٠٣ بعد انهتاء الربانمج، احتياجايت مازالت جمابة منيتك عيل املس توي الشخيص؟ بياانت التالية عيل لت رجاء تقيمي ال مقياس من ١إايل ٥ ) ١ -303 أي من نشاطات الربانمج اكنت مفيدة القل فائدة و ٥ كرث فائدة( ال 5 4 3 2 1 أ. منية القيادة ت نشاطات وتدريبات ب. ورشة العمل للتوجيه قبل السفر Predeparture ج. إ الرشاد املهين و خدمات دمع الالتحاق بلوظائف املقدمة من STEP د. التدريب العميل )internships) ه. اخري )رجاء التحديد( منيتك عيل املس توي املهين؟ بياانت التالية عيل مقياس من لت رجاء تقيمي ال ١إايل ٥ ) -304 أي من نشاطات الربانمج اكنت مفيدة ١ القل فائدة و ٥ كرث فائدة( ال 5 4 3 2 1 أ. منية القيادة ت نشاطات وتدريبات ب. ورشة العمل للتوجيه قبل السفر Predeparture ج. إ الرشاد املهين و خدمات دمع الالتحاق بلوظائف املقدمة من STEP د. التدريب العميل )Internships( ه. اخري )رجاء التحديد( رجاء تقيمي ال ١إايل ٥ ) .-305 أي من ن شاطات الربانمج اكنت اكرث فائدة للحصول عيل معل أو س تكون اكرث فائدة؟ بياانت التالية عيل مقياس من ١ القل فائدة و ٥ كرث فائدة( ال 5 4 3 2 1 أ. منية القيادة ت نشاطات وتدريبات ب. ورشة العمل للتوجيه قبل السفرPredeparture ج. إ الرشاد املهين و خدمات دمع الالتحاق بلوظائف املقدمة من STEP د. التدريب العميل )Internships) ه. اخري )رجاء التحديد( -306 هل دفع كل الربانمج مرصوف؟ .3 نعم .4 ل  ان إ تقل اىل السؤال رمق 308 131 | STEP END OF TERM EVALUATION REPORT USAID.GOV قمية املرصوف الشهري املقدم لمك ابدلولر؟ -307 ماذا اكن -308 هل اكنت ه إ ناك مصاريف اضافية غري مبارشة )خاصة بدلراسة( ومل يغط إ ها الربانمج؟ ا إ ذا اكنت الجابة "نعم" اكتب القمية بدلولر القسم الرابع: النشاطات اجلامعية ال -401 رجاء تقيمي مدي الاس تفادة الااكدميية جلامعتك مقياس من ١ ايل٥ ) ١ القل فائدة و ٥ كرث فائدة( -402 هل قدمت جامعتك نشاطات ودمع شبيه لربانمج STEP ؟ .3 نعم .4 ل  ان إ تقل اىل السؤال رمق 405 -403 ما النشاطات الشبهية اليت قدهما الربانمج وجامعتك؟ )اختار لك ما ينطبق( أ. منية القيادة ت نشاطات وتدريبات ب. إ الرشاد املهين ج. و خدمات دمع التعيني الوظيفي د. التدريب العميل )Internships) ه. نشاطات الطلبة و. نشاطات اجلامعة الامريكية بلقاهرة - ميية واملهارات الااكدميية لتعل ا ز. اخري )رجاء التحديد( -404 إبل إ شارة ايل قامئة النشاطات ال سابقة، أي مهنم اكن من املهم لربانمج STEP ان يقدهما؟ مبعين اخر مل يمت تقدميهم بوجه اكمل او مل تقدهما جامعتك وكنت حباجة دمع من برانمج STEP؟ ( أخرت لك ما ينطبق( أ. منية القيادة ت نشاطات وتدريبات ب. إ الرشاد املهين ج. خدمات دمع الالتحاق بلوظائف د. التدريب العميل )Internships) ه. نشاطات الطلبة و. ميية واملهارات الااكدميية لتعل نشاطات اجلامعة ا ز. اخري )رجاء التحديد( -405 من من املدرجني بلقامئة التالية يقوم مبساندتك مع نواي أو تعتربمه س ً ندا كل وانت يف الولايت املتحدة ؟ وذكل يشمل من ت إ تحدث الهيم عندما يكون تواجه مصاعب؟ )رجاء الرتتيب من رمق ١إايل٦( الرتتيب فقط أ. موظفي الربانمج ب. العائةل والصدقاء يف مرص ج. أصدقاء ىف برانمج امل نحة أو يف الولايت املتحدة د. مكتب التوجيه الطاليب يف اجلامعة ه. ل أحد و. اخري )رجاء التحديد( USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 132 القسم اخلامس: التوظيف أ- وقت الربانمج/املنحة ب- بعد اس تكامل املنحة -501 مك عدد التدريبات العملية )Internships )الىت اس تلكمهتا؟ -502 هل اكن التدريب املهين مرت ً بطا ب إ تخصصك ادلرايس؟ )رجاء وضع دائرة عيل الجابة الصحيحة( -1 نعم -2 ل -1 نعم -2 ل -503ماذا اكن متوسط فرتة التدريب العميل )بلشهور( -504 عيل مقياس ١إايل ٥ )١ القل و ٥ العىل( ما مدي الاس تفادة يف تطويرك همن ًيا من التدريب إ العميل؟ )رجاء وضع دائرة عيل الجابة الصحيحة( 5 4 3 2 1 5 4 3 2 1 -505 من ساعدك عيل احلصول عيل التدريب العميل )Internships )اخلاص بربانمجSTEP ؟ )اختار لك ما ينطبق( 1( موظفي الربانمج 2( اجلامعة 3( الصدقاء 4( وجدهتا بنفيس 5( اخري )رجاء التحديد( اذا كنت ممن حصلوا عىل منحةماجس إ تري ا دارة العامل MBA  ان إ تقل اىل السؤال 507 -506رجاء تقيمي البياانت التالية: ً أوافق متاما أوافق حمايد ل أوافق ل أوافق متاما ً أ. يف معيل سأ س تخدم املهارات املكتس بة حديثا ب. أان واثق اين سأ معل يف جمال ختصيص بعد التخرج ج. أ إ ان واثق ان خدمات الرشاد املهين والتعيني الوظيفي اليت قدمت من قبل الربانمج ستساعدين عيل احلصول عيل معل د. منية القيادة اليت قدمت من قبل الربانمج ستساعدين ىف احلصول عيل ت أان واثق ان نشاطات فرصة معل ه. أان واثق ان التدريبات املهنية اليت قدمت من قبل الربانمج ستساعدين ىف احلصول عيل فرصة معل و. أان واثق انين ساحصل عيل معل ح يث أنىن خرجي من جامعة بلولايت املتحدة ز. أان واثق انين ساحصل عيل فرصة معل حيث ان ختصيص مطلوب يف سوق العمل املرصي ح. اان ل احتاج خدمات ارشاد همين او مساندة ىف البحث عن فرصة معل اذا كنت من احلاصلني عىل منحة STEM  ان إ تقل اىل سؤال 508 -507 رجاء تقيمي البياانت التالية: ً أوافق متاما أوافق حمايد ل أوافق ل أوافق متاما أ. إ خدمات الرشاد املهين واملساندة لاللتحاق بوظيفة املقدمة من الربانمج ساعدتين يف احلصول عيل معل ب. التدريب العمىل )Internships )املقدم من الربانمج ساعدين يف احلصول عيل فرصة معل ج. جمال ختصيص ساعدين يف احلصول عيل معل د. كوين خرجي جامعة من الولايت املتحدة ساعدين يف احلصول عيل فرصة معل 133 | STEP END OF TERM EVALUATION REPORT USAID.GOV منيىت همن ًيا ت ه. اان اس تخدم املهارات املكتس بة عن طريق الربانمج وتساعدين يف و. مازلت يف احت إ ياج لالرشاد املهين واملساندة للحصول عيل فرصة معل -508 يف رأيك، ما يه التحدايت للحصول عيل معل يف جمال ختصصك/دراس تك؟ )اخرت لك ما ينطبق( أ. غياب وظائف مناس بة ملؤهالت احلاصلني عىل درجة املاجس تري ب. مس توايت املرتبات واملزااي ج. بيئة املؤسسة/الرشكة د. اخري )رجاء التحديد( اذا كنت من احلاصلني عىل منحة ماجس إ تري ا دارة العامل MBA ، سواء بوظ يفة أو بعمل خاص ولتعمل حاليا  ان إ تقل اىل السؤال 521 ً -509 هل معكل يف نفس جمال ختصصك ادلرايس؟ .3 نعم .4 ل -510هل معكل احلايل يف نفس املؤسسة اليت كنت تعمل هبا قبل سفرك؟ .3 نعم .4 ل  ان إ تقل اىل السؤال رمق 512 -511 إا إ ذا اكنت الجابة "نعم"، هل هو نفس املسمى الوظيفي؟ .3 نعم .4 ل -512 مااذلى جشعك عىل الانضامم ملؤسس تك احلالية؟ )اختار لك ما ينطبق( أ. جمال نشاط املؤسسة ب. مسعة وماكنة املؤسسة ج. مس توى املرتبات واملزااي د. الوظيفة يف جمال ختصصك ادلرايس ه. اخري )رجاء التحديد( -513 ماذا اكنت الفرتة بني التخرج والتعيني الوظيفي )بلشهور املكمتةل( -514 ماهو عدد اس امترات طلب وظيفة الىت تقدمت هبا ؟ -515 ماهو عدد املقابالت الىت اس تمكلهتا؟ -516 ماهو عدد عروض الوظائف الىت رفضهتا؟ إ اذا اكنت الجابة بال يوجد  ان إ تقل اىل السؤال 518 -517 لو مت رفض عرض تعيني وظ يفي، رجاء حتديد الس باب)اختار لك ما ينطبق( USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 134 أ. الوظيفة ل تتناسب مع مؤهاليت/ختصيص ادلرايس ب. املرتب واملزااي غري اكف ج. مل اجعب بأ سلوب ال شخص اذلي أجرى املقابةل معي د. ل أ ظن أن املؤسسة اكنت تشهد تقدما ىف جمال نشاطها بدرجة اكفية ه. اخري )رجاء التحديد( أ - إ املرتب الجاميل الشهري فقط -518 رجاء حتديد املرتب الاجاميل واملزااي الشهرية اليت حتصل علهيا يف معكل احلايل؟ )بجلنيه املرصي( )قبل الرضائب والتأ مينات(: 4500 ≥ - 3000 .36 6000 ≥ - 4501 .37 7500 ≥ - 6001 .38 9000 ≥ - 7501 .39 12000 ≥ - 9001 .40 14000 ≥ - 12001 .41 .42 أعىل من 14000 ب- رشح املزااي: -519 إ رجاء حتديد املرتب الجاميل واملزااي الشهرية يف معكل السابق قبل السفر ايل الولايت املتحدة ؟ )بجلنيه املرصي( أ - إ املرتب الجاميل الشهري فقط )قبل الرضائب والتأ مينات(: 4500 ≥ - 3000 .1 6000 ≥ - 4501 .2 7500 ≥ - 6001 .3 9000 ≥ - 7501 .4 12000 ≥ - 9001 .5 14000 ≥ - 12001 .6 .7 أعىل من 14000 ب- رشح املزااي: -520 مقارنة بطلبة ماجس تري او MBA من جامعات مرصية، هل تعترب مرتبك : .7 اقل .8 نفس املس توي .9 أعيل .4 ل اعرف .521 يف رشكتك احلال ية، هل رأ يت أي مؤرشات لختالف الراتب بني الراجل والس يدات؟ .1 نعم .2ل -522 هل دليك أي اقرتاحات او توصيات لتحسني برانمج منح دراس ية شبيه لربانمج STEP ؟ أ. ............................................................................................................ ب. ............................................................................................................ ج. ............................................................................................................ لقد مت الانهتاء من الاس تبيان. شكرا ملسامهتك الهامة. 135 | STEP END OF TERM EVALUATION REPORT USAID.GOV أ داة رمق )3(: دليل مقابةل الطالبات املتخلفات عن العودة بعد انهتاء املنحة اترخي املقابةل: موعد املقابةل: من أجرى املقابةل/ عضو الفريق املشارك جسل املالحظات: امس ادلارسة )الطالبة( املقدمة شكرا كل عىل موافقتك عىل إاجراء مناقشة معك ال إ يوم. نقوم بجراء تقيمي لربانمج STEP USAID( املنح ادلراس ية والتدريب للمهن ُ يني املرصيني(. وقد طلب منك املشاركة يف هذه املناق شة ل أنشطة مرشوع STEP. نك تعترب واحدا من الرشاكء املهمني يف منية ادلولية / مرص مع مرشوع للت )خدمات لتح إ سني ا دارة الداء وتعزيز التعمل والتق إ يمي( لجراء هذا ا . تعاقدت الواكةل المريكية SIMPLE لتقيمي ل الغرض من هذه املاكملة هو التعرف عىل مدى فعالية برانمج STEP منية ادلول ية المريكية لت املمول من هيئة ا منية ادلولية ملساعدهتا يف تعزيز تصممي الربامج املس تقبلية، لت .بصفتك دارسة من ، نود احلصول عىل مالحظاتك حول النشطة ، وكيف تتصوريهنا وكيف سامهت يف خطط حياتك. ستبقى ردودك رسية ولن يمت ن رشها أو ربطها علن بك بشلك ً قبل STEP USAID ا مبارش. ن توقع أل تس تغرق كرث من املاكملة أ 20 دق يقة، ونرحب أن تناقيش لك املسائل اليت تريهنا هممة. الس ئةل: 1( من وهجة نظرك، ملاذا مت اخت إ يارك لالنضامم اىل برانمج املنحة ادلراس ية؟ هل اكنت معايري الاختيار موضوعية وعادةل؟ ما يه اجلوانب إ /املعايري، ان وجدت، اليت ترين رضورة تغيريها ؟ )EQ2B( يدا أ إ كرث؟ ان اكنت 2( قدم برانمج STEP عدة أنشطة ق بل وأثناء برانمج املنحة ادلراس ية )مثل: ورش معل همارات الق إ يادة، الرشاد املهين وغريها( ؛ هل شاركت فهيا؟ ما اذلي وجدته مف إ ن اكنت الجابة إ إ كرثها افادة، وملاذا. وا إ الجابة "نعم" مفا اذلي اكن أ “ل“، فلامذا ا؟ )B2EQ) 3( اكن تطوير همارات الق ً يادة مكوان أساس ًيا يف برانمج STEP . يف رأيك، هل اكتس بت أية همارات ق إ يادية؟ ا إ ذا اكنت الجابة "نعم"، هل ميك نك أن تعطيين م ً ثال عىل تطبيق/تنفيذ هذه املهارات؟ )4EQ) 4( خالل فرتة الربانمج يف الولايت املتحدة، هل واهج تك أية حتدايت ؟إا إ ذا اكنت الجابة "نعم"، ماىه تكل التحداي ت ؟ هل أعدك الربانمج ملواهجهتا؟ )4EQ) إرشح واعطاء توصيات لتحسيهنا يف املس تقبل؟ )4EQ) 5( يف العموم هل تلقيت ادلمع اذلي توقعته من STEP؟ برجاء ال 6 إ اكدميي؟ ا إ ذا اكنت الجابة "نعم ( هل اكنت ه ناك أية حتدايت عىل املس توى ال "، ماذا اكنت؟ من اذلي دمعك ملعاجلهتا؟ وكيف؟)4EQ) 7( هل اس تخدمت اخلدمات املتاحة يف جامعتك اليت تساعد عىل تطويرك همنيا وخشصيا؟ )املشورة املهنية،والتسجيل يف اجلامعة وتقدمي املشو رة، وتطوير السرية اذلاتية(. هل تشعرين بوجود تداخل بني هذه اخلدمات وما مت تقدميه كل من خالل برانمج STEP؟ ماىه اخلدمات اليت اكنت عاد كرث فعالية ) برانمج ة أ STEP أم اجلامعة(؟ )B2EQ) منية ادلول ية المريكية ستمتكن -مبوجب لت ً باب حمددة لقرارك بعدم العودة. لكن نا نود أن نسمعها من جانبك. هل ميكنك توض يح موقفك وأس باب ه لن هيئة ا 8 ( لقد قرأان تقارير امل رشوع ووجدان أس املعلومات املتصةل مبثل هذا املو قف – من معرفة كيفية حتسني الربامج املس تقبلية؟ )B2 EQ) 9 ( هل من املمكن أن تشاركينا ما تفعلينه بعد الانهتاء من دراس تك؟ هل وجدت وظيفة؟ هل الوظيفة تتصل مبجال دراس تك؟ هل اكنت املهارات اليت اكتسبهتا خالل STEP برانمج مفيدة يف احلصول علهيا؟ )4EQ) 10( هل دليك أية توص يات أخرى للمساعدة يف تصممي برامج املنح ادلراس ية املس تقبلية ب شلك أفضل، خاصة بلنس بة للمرأة؟ )B2 EQ) هناية الس ئةل، نشكرمك لآرائمك القمية USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 136 أ داة رمق )4(: دليل املقابةل مع اجلهات املس تفيدة من أ نشطة املساندة الفنية)وزارة الرتبية و التعلمي – وزارة الصحة( اترخي املقابةل: موعد املقابةل: من أجرى املقابةل/ عضو الفريق املشارك جسل املالحظات: الوزارة: امس ووظ يفة من أجرى املقابةل: املقدمة ً ا ملوافقتك عىل لقائنا ال إ يوم. نقوم بجراء تقيمي لربانمج شكر STEP USAID( املنح ادلراس ية والتدريب للمهنيني املرصيني(. وقد طلب إ نا اجراء هذه املقابةل مع س يادتمك لكونمك من أمه ال رشاكء يف أنشطة برانمج STEP. منية ادلولية / مرص مع مرشوع )مرشوع خدمات حت إ سني ا دارة الداء وتعزيز التعمل وا إلجراء هذا التقيمي. تعاقدت الواكةل المريكية SIMPLE لتقيمي( للت خالل الس نوات ال املمول من ق بل الواكةل المريكية ربعة املاضية، للت الغرض من زايرتنا هو التعرف عىل مدى فعالية برانمج STEP منية ادلولية ملساعدهتا يف تعزيز تصممي الربامج املس تقبلية قدم برانمج STEP أنشطة مساندة فنية للوزارة بناء عىل طلهبا. خالل هذه املقابةل نود ال تعرف عىل رأي س يادتمك ىف أنشطة املساندة الفنية الىت قدهما برانمج STEP لوزارتمك ومدى . تقي ميك لفائدة تكل النشطة ىف حتق يق الهداف الاسرتاتيجية للوزارة ً ستس تغرق املناقشة حوايل 30 و 45 ا ملسامهتمك القمية يف هذه املبادرة اله دقيقة ، شكر امة. الس ئةل: 1( ما ىه أنشطة املساندة الفنية الىت قدهما برانمج STEP للوزارة؟ اذكر النشطة الىت ميك نك تذكرها وتوارخي تكل النشطة بلتقريب؟ (5EQ( 2( كيف سامهت أنشطة املساندة الفنية الىت قدهما برانمج STEPللوزارة ىف حتقيق الهدف الاسرتات إ يجى اخلاص بعداد كوادر مدربة تلىب احتياجات سوق العمل بكفاءة ؟ (EQ1 &5) 3( كيف سامهت أنشطة املساندة الفنية الىت قدهما برانمج STEPللوزارة ىف حتق يق الهداف الاسرتاتيجية للوزارة؟ (5&1EQ( 4( عىل مقياس من 5-1 ، ما هو تقيميك لفائدة أنشطة املساندة الفنية الىت قدهما برانمج STEPللوزارة )5 تعىن فائدة كبرية للغاية( – أذكر أنشطة املساندة الفنية الىت تتذكرها واذكر (5EQ (تقيميك للك مهنا منية ادلولية ؟ (1EQ( 5( ىف رأ يك، ما ىه أ مه عوامل جناح أنشطة ادلمع الفىن الىت تقدهما برامج ا لواكةل المريكية للت 6( ما ىه القدرات املهن ية الىت ترى أهنا تطورت ىف امل شاركني من الوزارة ىف أنشطة الفنية لربانمج STEP؟ (5 & 1 EQ( 7( ما هو ادلور اذلى قامت به الوزارة ىف ختطيط و تنف يذ أنشطة املساندة الفنية الىت قامت هبا مع برانمج STEP ؟ (5EQ( ق مية تكل النشطة و اس تدامهتا؟ (5EQ( كيد 8( كيف ساعد/سيساعد ادلور اذلى قامت به الوزارة مع برانمج STEPىف ختطيط و تنف يذ أنشطة املساندة الفنية ىف تأ ُقدمة للوزارة من خالل الربامج املس تقبلبة؟ (5EQ( منية ادلولية لتقوم ب تعزيز أنشطة املساندة الفنية امل 9( ما ىه امل قرتحات الىت تقدموها الواكةل المريكية للت هناية الس ئةل، نشكرمك لآرائمك القمية.... 137 | STEP END OF TERM EVALUATION REPORT USAID.GOV أ داة رمق )5(: دليل املقابةل مع مس ئويل يف وزراة التعلمي العايل والبحث العلمي اترخي املقابةل: موعد املقابةل: من أجرى املقابةل/ عضو الفريق املشارك جسل املالحظات: امس أو ماكن املؤسسة/الرشكة شلك التعاون مع املرشوع: امس ووظ يفة من أجرى املقابةل: املقدمة شكرا كل عىل موافقتك عىل لقائنا ال إ يوم. نقوم بجراء تقيمي لربانمج STEP USAID( املنح ادلراس ية والتدريب للمهن ُ يني املرصيني(. وقد طلب منك املشاركة يف هذه املناق شة لنك تعترب واحدا من أمه ال رشاكء يف أنشطة برانمجSTEP . منية ادلولية / مرص مع مرشوع للت )مرشوع خدمات حت إ سني ا دارة الداء وتعزيز التعمل والتق إ يمي( لجراء هذا ا . تعاقدت الواكةل المريكية SIMPLE لتقيمي املمول من الواكةل المريكية للت الغرض من زايرتنا هو التعرف عىل مدى فعالية برانمج STEP منية ادلولية ملساعدهتا يف تعزيز تصممي الربامج املس تقبلية، وتأ إ يت هذه الزايرة يف ا طار أن وزارة التعلمي العايل وال بحث العلمي رشيك أسايس يف املرشوع. س إ تعامل ال إ جابت ىف اطار من الرسية، ولن يمت نرشها علن . ًا ا كل ملسامهتك الق من امل توقع أن تس تغرق املناقشة ما بني 30 و 45 مية يف هذه املبادرة الهامة ً دقيقة، شكر . الس ئةل 1( هل ميك نك ذكر أي فعاليات أ و أنشطة شاركت فهيا مع برانمج STEP؟ ماذا اكن ادلور اذلي مقت به . )B1.EQ) 2( نود التعرف عىل تقيميك العام لربانمج STEP، ل س امي كيف خيدم الهداف الاسرتاتيجية للوزارة. )B1.EQ) 3( ركز برانمج املنح ادلراس ية عىل مسارين: 1[ العلوم والتكنولوجيا والهندسة والرايضيات )STEM )و 2[ درجة املاجس إ تري يف ا دارة العامل )MBA). من وهجة نظرك، هل ميكن القول ( ت B1.EQ أن اخلرجيات من الك امل سارين ميكن أن يسهمن يف منية مرص اق إ تصاداي وتطورها؟ ا إ ذا اكنت الجابة ، "نعم" ، كيف يمت ذكل؟ ) 4( لك املس تفيدات من برانمج املنح هن فتيات مت اختيارهن من عدة حمافظات، ك يف ميكن أن تس تفيد الوزارة من هذا الاكدر اجلديد املؤهل؟ هل هناك خطط لالس تفادة من هماراهتن )EQ1.B( ومعارفهن؟ 5 ( من وهجة نظر الوزارة، ما يه اجلوانب أ و ال إ بعاد اليت يقرتح اضافهتا لتحسني تصممي برانمج املنح ادلراس ية STEM؟ ما يه التغ إ يريات اليت ميكن ادخالها؟ )B1.EQ) 6( مكس ئول يف وزارة التعلمي العايل وال بحث العلمي، هل دليك أية توص إ يات اضافية لتحسني الربامج املس تقبلية يف نفس جمال STEP؟ )B1.EQ) هناية الس ئةل، نشكرمك لآرائمك القمية.... USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 138 أ داة رمق )6(: دليل املقابةل مع الرشاكء احملليني اترخي املقابةل: موعد املقابةل: من أجرى املقابةل/ عضو الفريق املشارك جسل املالحظات: امس أو ماكن املؤسسة/الرشكة شلك التعاون مع املرشوع: امس ووظ يفة من أجرى املقابةل. املقدمة شكرا عىل موافقتك عىل لقائنا ال إ يوم. نقوم بجراء تقيمي لربانمج STEP USAID( املنح ادلراس ية والتدريب للمهن يني املرصيني(. وقد مت أختيارك للم شاركة يف هذه املقابةل لنك تعترب واحدا من الرشاكء امل همني يف أنشطة مرشوع STEP. منية ادلولية / مرص مع مرشوع للت )خدمات لتح إ سني ا دارة الداء وتعزيز التعمل والتق إ يمي( لجراء هذا ا . تعاقدت الواكةل المريكية SIMPLE لتقيمي املمول من الواكةل المريكية للت الغرض من زايرتنا هو مجع معلومات حول فعالية برانمج STEP منية ادلولية ملساعدهتا يف تعزيز تصممي الربامج املس تقبلية. خالل هذه الزايرة نود التعرف عىل او حمور املساندة الفنية أ ، سواء اكنت ال STEP ذا انط باعاتك عىل النشطة الىت شاركت فهيا املؤسسة/ الرشكة مع برانمج STEP ن شطة الىت أس هتدفت حمور املنح ادلراس ية ىف برانمج ً يضا. كنت قد شاركت مع برانمج STEP ىف هذا احملور أ ً ستس تغرق املناقشة ما بني 45 - 60 ا كل عىل مسامهتك القمية يف هذه املبادرة الهامة دقيقة ، شكر . الس ئةل الق سم الول: أ س ئةل للرشاكء احملليني الذلين تعاملوا مع طلبة منحة STEP 1( ما اذلى تضم نه دورك كحد الرشاكء احملليني لربانمج منح STEP( أذكر مج يع الدوار ذات العالقة( • إ العالن عن املنحة • معلية الاختيار للحصول عىل املنحة منية همارات القيادة • ت • النشطة الثقافية • إ الرشاد املهين • إاجياد فرص معل • التوظيف • ورشة العمل املنعقدة عند إ العودة اىل مرص • إ برانمج العداد للجامعة • حتديد الاحتياجات • املتابعة والتقيمي • أنشطة املساعدات الفنية لوزارة التعلمي والتعلمي الفىن • أنشطة املساندة الفنية لوزارة الصحة والساكن • أ دوار أخرى )من فضكل حدد( 2( أى من اجملموعات تعاملت مع ها ؟ )أ خرت لك الجابت الصحيحة( MBA Cohort 1 MBA Cohort 2 STEM Cohort 1 STEM Cohort 2 3( من وهجة نظرك، ما ىه أمه اجنازات برانمج املنح ادلراس ية STEP؟ (1EQ( 4( من وهجة نظرك، ما ىه أمه عوامل ال نجاح الىت سامهت ىف تكل الجنازات؟ 1EQ( 139 | STEP END OF TERM EVALUATION REPORT USAID.GOV 5( هل تع تقد ان رشوط و طريقة أختيار املشرتكني ىف منح برانمج STEP مناس بة لخت يار أفضل العنارص الىت دلهيا اس تعداد للقيادة؟ )نعم/ل( (2EQ( 6( ما ىه ال رشوط و/أ إ و ال جراءات الخرى الىت جيب اتباعها بلنس بة ادلفعات املس تقبةل؟ (2EQ( 7( هل لحظت اختالفات مابني احتياجات طالب منح برانمج STEP (عىل سبيل املثال، بني مجموعةSTEMدراسات العلوم والهندسة والرايضيات و مجموعة ماجس إ تري ادارة أو بني طالب اجملموعة الواحدة؟ نعم/ل (2EQ (العامل MBA 8( أ إ ذا اكنت الجابة عن السؤال السابق "نعم"، ماذا اكنت تكل الاحتياجات اخملتلفة وكيف تعاملت معها؟ (2EQ( 9( عىل املقياس من 5-1 )5 تعىن مس ٌ توى مرىض بدرجة كبرية(، كيف تُقمي مس توى رضائك عن تنف يذ النشطة التالية الىت قدمهتا املؤسسة/ الرش كة لربانمج STEP؟ )من فضكل مق بتقيمي مج يع الن شطة الىت أشرتكت فهيا املؤسسة/ الرشكة( (2EQ( • إ العالن عن املنحة • معلية الاختيار للحصول عىل املنحة منية همارات القيادة • ت • النشطة الثقافية • إ الرشاد املهين • إاجياد فرص معل • التوظيف • ورشة العمل املنعقدة عند العودة إاىل مرص • إ برانمج العداد للجامعة • حتديد الاحتياجات • املتابعة والتقيمي • أنشطة املساعدات الفنية لوزارة التعلمي والتعلمي الفىن • أنشطة املساندة الفنية لوزارة الصحة والساكن • أ دوار أخرى )من فضكل حدد( 10( كيف ميكن حت سني تكل النشطة لدلفعات املس تقبلية؟ (1EQ( 11( ىف رأيك، ما ىه امل هارات أو السامت الشخص ية الالزمة لس تكامل املنحة ادلراس ية لربانمج STEPبنجاح ؟ (1EQ( 12( إ من اجابتك عن السؤال ال سابق، هل ترى أن احلاصلني عىل املنح ادلراس ية لربانمج STEP متكنوا من تطوير تكل املهارات و السامت الشخصية قبل التحاقهم بربانمج املنح ادلراس ية ىف اجلامعات المريك إ ية؟ نعم / ل ا ذا اكنت الجابة "ل"، ما ىه املهارات و السامت الشخصية الىت اكنت حتتاج اىل تطوير اكرث قبل التحاقهم بربانمج املنحة ادلراس ية ىف اجلامعات المريكية؟ (1EQ( 13( ىف رأيك، ما ىه امل هارات أو السامت الشخصية الالزمة للحصول عىل وظيفة ىف مرص؟ (1EQ( 14( إ من اجابتك عىل السؤال ال سابق، هل ترى أن احلاصلني عىل املنح ادلراس ية لربانمج STEP متكنوا من تطوير تكل املهارات و السامت الشخصية مع اس تكامهلم لربانمج املنحة السامت ا عات المريك إ ية؟ نعم/ ل ا ذا اكنت ال لشخصية الىت ما زالوا ادلراس ية ىف اجلام جابة "ل"، ما ىه املهارات و حيتاجون اىل تطويرها بعد اس تكامهلم لربانمج املنحة ادلراس ية حىت (1EQ (يمتكنوا من احلصول عىل وظيفة؟ 15( عىل املقياس من ،5-1 كيف ترى احامتلية حصول خرجيى برانمج املنح ادلراس ية عىل وظيفة لها عالقة بلتخصص ادلراىس ىف برانمج منحة STEP( 5 تعىن أ ً حامتل مرتفع جدا( (EQ4 ميك املرتفع أو املنخفض؟ (4EQ( 16( من السؤال السابق، ما هو السبب الرئيىس وراء تقي 17( هل أنت عىل دراية بأ ن شطة العمل أو التوظيف للطالب العائدين من برانمج املنح ادلراس ية STEP؟ نعم/ل (3EQ( 18( أ ذا اكنت أجابتك عىل السؤال ال سابق "نعم"، من فضكل أ ذكر أمثةل (3EQ( ملس متر الىت حيتاهجا الطالب العائدون من برانمج املنح ادلراس ية STEP للحصول عىل وظيفة لها عالقة بلتخصص ادلراىس ىف برانمج منحة STEP؟ من 19( ما ىه أنشطة ادلمع ا اذلى جيب أن يقدم هذا ادلمع؟ (4EQ( ًاى مهنم سيتوقف عن اس تكامل الربانمج ادلراىس أو يرفض الرجوع اىل مرص بعد 20( ىف ضوء الوقت الىت قضيته مع طالب برانمج املنح ادلراس ية STEP، هل اكن ه ناك أ ى مؤرش أ ن أ اس تكامل الربانمج ادلراىس ىف الولايت امل تحدة؟ نعم/ ل /ل أعرف (2EQ( 21( أ ذا اكنت أجابتك عىل السؤال ال سابق "نعم"، من فضكل اذكر أمثةل (2EQ( USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 140 22( هل تقرتح دليك مقرتحات للحد من عدد الطالب احلال يني أو املس تقبليني من ال توقف عن أس تكامل الربانمج ادلراىس أ و عدم الرجوع اىل مرص بعد أس تكامل الربانمج ادلراىس ىف الولايت املتحدة ؟ (2EQ( 23( كيف سامه ادلور اذلى لعبته املؤسسة/ الرشكة مع برانمج STEP ىف حتقيق الهدف العام للربانمج و اخلاص "بتعزيز القدرات الفن ية ، الدارية و الق يادية للمرأه مبؤسسات القطاع العام واخلاص. 1EQ( 24( من وهجة نظرك، هل مت تنف يذ أنشطة برانمج املنح ادلراس ية STEPبطريقة تضمن اس تدامة النتاجئ؟ نعم/ل (1EQ( 25( هل اكن من املمكن تنف يذ أنشطة برانمج املنح ادلراس ية STEPبطريقة خمتلفة لزايدة فرص اس تدامة نتاجئ الربانمج؟ (1EQ( هل أشتركت المؤسسة/الشركة فى أنشطة محور المساندة الفنية لبرنامج المنح الدراسية STEP؟ نعم / ⸀� )فى حالة ⸀�جابة���������جابة عن أسئلة القسم الثانى، فى حالة ⸀�جا���"، يتم إنهاء المقابلة. القسم الثانى: أسئلة أضافية للشركاء المحليين اللذين أشتركوا فى تقديم أنشط ة محور المساندة الفنية 26( ما هى أنشطة المساندة الفنية التى أشتركت فى تقديمها للجهات الحكومية �� برنامج STEP؟ (5EQ( 27 ( فى رأيك، كيف تطورت القدرات المؤسسية فى الوزارات التى اشتركت فى أنشطة المساندة الفنية التى شاركت فيها مع (EQ1 & EQ5) ؟STEP برنامج 28( م ا هى القدرات المهنية التى ترى انها شهدت جوانب تحسن لدى المشاركين )من وزارة التعليم ووزارة الصحة والسكان(فى أنشطة المساندة الفنية لبرنامج STEP ؟(5EQ & 1EQ( 29( ما هو الدور الذى قامت به الجهات الحكومية فى تخطيط وتنفيذ أنشطة المساندة الفنية التى قامت بها المؤسسة/الشركة ����برنامج STEP ؟ (5EQ( 30( كيف ساعد/سيساعدالدور الذى قامت به الجهات الحكومية فى تخطيط وتنفيذ أنشطة المساندة الفنية فى أدراك قيمة تلك ����واستدامتها؟ (5EQ( 31 ُ ( ما هى مقترحاتكم لزيادة القيمة المضافة �نشطة المساندة الفنية المقدمة للجهات الحكومية فى المستقبل من خ�ل برامج ������ريكية؟ (5EQ( 141 | STEP END OF TERM EVALUATION REPORT USAID.GOV أ داة رمق )7(: دل يل مقابةل مع أحصاب العمل اترخي املقابةل: وقت املقابةل: ال باحث / أعضاء الفريق الآخرون: املدون: نوع املؤسسة: غري هادف للرحب، خاص٬ حكويم القطاع: امس ال ُ شخاص اذلين أجريت معهم املقابةل واملسميات الوظيفية املقدمة نشكرمك ملوافق إ تمك عىل اجراء هذا اللقاء ال إ يوم. حنن نقوم بجراء تقيمي لربانمج STEP USAID( املنح ادلراس ية والتدريب للمهن يني املرصيني(. وقد مت أختيارك للم شاركة يف هذه املقابةل لنك تعترب واحدا من الرشاكء امل همني يف أنشطة مرشوع STEP. للت لقد منية ادلولية / مرص مع مرشوع )خدمات لتح إ سني ا دارة الداء وتعزيز التعمل والتق إ يمي( لجراء هذا ا . تعاقدت الواكةل المريكية SIMPLE لتقيمي منية ادلول ية ال الغرض من هذه املقابةل هو التعرف عىل مدى فعالية برانمج STEP مريكية ملساعدهتا يف تعزيز تصم لت املمول من هيئة ا مي الربامج املس تقبلية. وبعتبارمك من املؤسسات والرشاكت الىت ال إ تحقت بلعمل دلهيا احدى خرجيات برانمج منحة ماجس إ تري ا دارة ال إ عامل من احدى اجلامعات بلولايت املتحدة، ومبساندة من برانمج STEP إ ، فاننا نود أن نناقش جتربتمك – حىت اترخيه - مع هذه اخلرجية، كام نود ال كيد بأ إن اجابتمك س تعامل ب إ رسية اتمة، ولن يمت ال إ فصاح عن ا مسمك أو امس خرجية الربانمج الىت تعمل دلى مؤسس تمك. تأ ً ا كل عىل مسامهتمك الق من امل توقع أن تس تغرق املناقشة ما بني 30 و45 مية يف هذه املبادرة الهامة دقيقة، شكر . الس ئةل 1( مىت التحقت خرجية برانمج STEP احلاصةل عىل ماجس إ تري ا دارة العامل بلعمل ىف مؤسس تمك؟ (A1EQ( منذ: أقل من س تة أشهر فرتة ترتاوح بني س تة أشهر وعام واحد فرتة ترتاوح بني عام واحد وعامني فرتة تزيد عن عامني 2( ماهو املسمى الوظيفى خلرجية برانمج STEP احلاصةل عىل ماجس إ تري ا دارة العامل ىف مؤسس تمك؟ (A3EQ & A1EQ( 3( هل متت ترقيهتا وظيف ًيا منذ التحاقها بلعمل دلى مؤسس تمك؟ (A3EQ & B1EQ (نعم/ل 4 ( هل ترون أن املهام الوظيفية املس إ ندة الهيا تتصل عىل حنو مبارش بتخصصها ادلرايس ىف ماجس إ تري ا دارة العامل؟ (A4EQ (نعم/ل إ)اذا اك إ نت الجابة "نعم"، ان إ تقل اىل السؤال رمق 6(. 5 إ( ا إ ذا اكنت الجابة "ل" عن السؤال السابق، هل تنتوون تلكيفها – ً لحقا - مبهام وظيفية ترتبط عىل حنو مبارش بتخصصها ادلرايس ىف ماجس إ تري ا دارة العامل؟ (A4EQ (نعم/ل ً و لحقا – مي كن أن تشغل وظيفة قيادية مبؤسس تمك؟ (A4EQ( 6 ( هل ترون أهنا – ىف الوقت احلايل، أ نعم/ل إ)ا إ ذا اكنت الجابة "نعم"، ان إ تقل اىل السؤال رمق 8( 7 إ( ا إ ذا اكنت الجابة "ل" عن السؤال السابق، نرجو ذكر ثالثة عنارص من املعارف واملهارات الىت حتتاج هذه املوظفة لتحسيهنا يك تكون مؤهةل – مس تق ً بال - لشغل وظيفة قيادية مبؤسس تمك. (EQ4B) أ(______________ ب( ______________ ج( ______________ د( مل يمت تعييهنا بملؤسسة ىك تضطلع مبهام قيادية هـ( ل اعمل 8( هل دليمك بملؤسسة ىف الوقت احلاىل – أو فامي س بق – موظفون/موظفات من احلاصلني عىل ماجس إ تري ا دارة العامل؟ (A1EQ (نعم/ل إ)ا إ ذا اكنت الجابة "ل"، ان إ تقل اىل السؤال رمق 11( ميية 9 إ( ا ذا ما أجريت مقارنة بني خرجية برانمج STEP احلاصةل عىل ماجس إ تري ا دارة العامل وبني غريها من العاملني مبؤسس تمك واحلاصلني عىل ماجس إ تري ا دارة العامل من مؤسسات تعل أ خرى، هل ترون أن: (A1EQ( مي ية أخر أ ( أهنا متتكل معارف تطبيقية وجدارات وهمارات تزيد عام ميتلكه غريها من احلاصلني عىل ماجس إ تري ا دارة العامل من مؤسسات ى؟ تعل ب( أهنا متتكل نفس القد مي ية أخرى؟ ر من املعارف التطبيقية واجلدارات واملهارات الىت ميتلكها غريها من احلاصلني عىل ماجس إ تري ا دارة العامل من مؤسسات تعل مي ية أخر جـ( أهنا متتكل معارف تطبيقية وجدارات وهمارات تقل عام ميتلكه غريها من احلاصلني عىل ماجس إ تري ا دارة العامل من مؤسسات ى؟ تعل د( لأعمل 10 إ( ا ذا ما أجريت مقارنة بني املرتب واملزااي املمنوحة خلرجية برانمج STEP احلاصةل عىل ماجس إ تري ا دارة العامل وبني املرتبات واملزااي املمنوحة لغريها من العاملني مبؤسس تمك واحلاصلني عىل مي ية أ خرى، هل ترون أن ما حتصل عل ماجس إ تري ا دارة العامل من مؤسسات يه خرجية برانمج STEP :) A 3EQ & A1EQ (تعل ا( أعىل ب( عىل نفس املس توى جـ( أقل د( ل اعمل USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 142 11( ما إ هو املرتب الجاميل )قبل اس تقطاع الرضائب والتأ مينات( خلرجية برانمج STEP احلاصةل عىل ماجس إ تري ا دارة العامل مبؤسس تمك؟ (A3EQ & A1EQ( ______________ جنيه مرصى 12( ماىه مجموعة املزااي النقدية والعينية املمنوحة من قبل مؤسس تمك خلرجية برانمج STEP احلاصةل عىل ماجس إ تري ا دارة العامل؟ (A3EQ( ________________________________________________________________ 13( هل شاركمت – خشص ًيا – ىف معلية اختيار وتعيني خرجية برانمج STEP احلاصةل عىل ماجس إ تري ا دارة العامل مبؤسس تمك؟ (A4EQ (نعم/ل إ)اذا إ اكنت الجابة "ل"، ان إ تقل اىل السؤال رمق 18( 14 ( ماىه أمه املعارف التطبيقية/اجلدارات/املهارات الىت كنمت تس هتدفون توافرها عندما مقمت بتعييهنا؟ (B1EQ( أ(______________ ب( ______________ جـ( ______________ 15 إ ( عادة مايكون العاملون اجلدد حباجة اىل بعض التدريب وبرامج التوج يه، هل ترون أن خرجية برانمج STEP الىت نتحدث عهنا اكنت متتكل املعارف التطبيقية /اجلدارات /املهارات الثالث امل إ شار ا لهيا أعاله بملس إ توى اذلى يرىق اىل توقعاتمك؟ (B1EQ( د( مل تكن مت تكل أ هـ(لاعمل ًاي ا( نعم، مجيعها ب( اثنتان من ثالثة جـ( واحدة من ثالثة مهنا 16( حيامن كنمت تقومون بعملية التوظ إ يف واجراء املقابالت والتفاوض مع خرجية برانمج STEP احلاصةل عىل ماجس إ تري ا دارة العامل للعمل مبؤسس تمك، هل ترون أن توقعاهتا أآنذاك: (EQ4A & EQ3A) أ( من حيث مس توى املرتب واملزااي: أ- أ( اكنت واقعية أ-ب( مل تكن واقعية ب( من حيث املس توى الوظيفى عند التعيني:ب- أ( اكنت واقعية ب-ب( مل تكن واقعية 17( خالل معلية التفاوض حول املرتب، ماذا اكن الفارق بني املرتب اذلى طلبته وبني املرتب اذلى مت التفاق عليه؟ ____________جنيه أو ____________ % منية لت ً لهيا ىك متىض قدما ىف جمال ا 18 ( أ ًخريا ، ومن وهجة نظرمك، ماىه أنواع املعارف التطبيقية واجلدارات واملهارات والاجتاهات الىت ينبغى توافرها دلى العاملني والىت ت رون أ إ ن مرصحباجة ا (EQ1B) الاقتصادية؟ هناية الس ئةل، نشكرمك لآرائمك القمية... 143 | STEP END OF TERM EVALUATION REPORT USAID.GOV موافقة عىل اس تخدام مادة تصويرية مرشوع خدمات حت إ سني ا دارة الداء وتعزيز التعلمي والتقيمي )SIMPLE ) موافقة عىل اس تخدام املادة التصويرية اليت يمت التقاط ها ل غراض العامل اليت ينفذها مرشوع )SIMPLE )لصاحل مجموعة )LLC ,Group QED ) امس املرشوع: _______________________________ ماكن التقاط املادة التصويرة: ___________________________ أ وافق أ ان املوقع أ إ دانه عىل اماكنية اس تخدام املواد التصويرية مثل صوري م نفردا أو يف لقطات جامعية، وذكل كامدة يف التقارير والعروض التقدمي ية و / أو القنوات العامة )م ثل موقع الويب( أو مواد اتصال أخرى من ق QED أو LLC منية ادلولية ِبل مجموعة أو اجلهة املاحنة لهذا الن شاط ويه الواكةل المريكية (USAID (علام بأ نين متف هم أن هذا النوع من املنتجات ومواد التصال ميكن للت أن يمت نرش منية ادلولية ومجموعة ك إ يو ا إ ي دي، مبا يف ذكل املوقع اللكرتوين للرش ه من خالل خم إ تلف وسائل العالم ال تابعة للواكةل المريكية كة، وال إ تقارير، وبوابة ال إ نرتانت، ووسائل العالم الاجامتعية للت ويف أ إ ي وسائل ا عالم أ خرى أو مطبوعات تعممها مجموعة QED أو الواكةل منية ادلول ية. وأ نين أحل مجموعة من أ مسؤول ية أو إاجراء ضار قد ينشأ المريكية LLC.QED و USAID ي للت يف شأ ين خشصيا . ً أو يف شأ ن عائليت من خالل اس تخدام الصور املذكورة منية ادلولية وموظفي مجموعة إانه بتوق إ يعي هذه املوافقة فا نين أ إ عطي ال QEDلتعديل هذه املواد ومعاجلهتا وحتس يهنا من أجل جعلها مثالية لالس تخدام املقصود املذكور أعاله ملرة ذن للواكةل المريكية للت منية ادلولية ومجم واحدة أو مرات عديدة ح إ سب احلاجة، بدون الاضطرار ا إ ىل احلصول عىل اذن مين للك اس تخدام بواسطة الواكةل المريكية وعة LLC ,QED . للت ________________________________________ إ المس ________________________________________ التوقيع ________________ التارخي USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 144 ANNEX VIII: PROJECT DOCUMENTS REVIEWED 1- USAID/Egypt- Statement of Work: Scholarships and Training for Egyptian Professionals 2- STEP Workplan for Year 1. January 1, 2014 - September 30, 2014 3- STEP Workplan for Year 2. August 31, 2014 4- STEP Workplan for Year 3, Submitted on: August 31, 2015 5- STEP Workplan FY 18. Submitted on: August 29, 2017 6- Assistance Agreement between the United States of America and the Arab Republic of Egypt for the U.S.-Egypt Higher Initiative Agreement, 20 September 2014 7- PERFORMANCE MANAGEMENT PLAN FOR PERIOD OCT. 1, 2017 - SEPT. 30, 2018, 30 August 2018 8- STEP Performance Indicator Reference Sheet (PRIS)- 2018 9- Performance Management Plan Narrative, February 28, 2014 10- STEP Scope of Work, May 9, 2013 11- Results FY15, August 2015 12- Results FY17, September 2017 13- Results FY18, September 2018 14- FY18-STEP PMP Chart – USAID 15- Achievements Presentation, December 2018 16- IIE STEP PMP -USAID submission-Aug. 30 2018 (OCT. 1, 2017 - SEP. 30, 2018) 17- STEP PMP Narr-Feb 28-2014_Submitted 18- STEP FY15 PMP Results_Submitted 08 31 15 19- STEP PMP Chart-Academic_Oct 2014 to Sep 2015-Subm 20- STEP PMP Chart-Jan 2014 to Dec 2018-S 21- 2015 08 31 STEP PMP Narr-FY16 22- IIE STEP PMP -USAID submission-Aug. 30 2018 23- Results FY17-STEP PMP Chart – USAID 24- STEP - SOW for RFTOP FINAL May 9, 2013 25- STEP FY15 PMP Results_Subm 08 31 15 26- STEP PMP Chart-Academic_Oct 2014 to Sep 2015-Subm 27- STEP PMP Chart-Jan 2014 to Dec 2018-S 28- STEP PMP Narr-YR2_Academic Focused-Revised-Subm 29- Campus Visit Report – Arkansas, Submitted 16 May 2017 30- Campus Visit Report – IIT, Submitted: 30 May 2017 31- Campus Visit Report - Minnesota, Submitted: 2 December 2016 32- Campus Visit Report - ST. CLOUD, Submitted 21 March 2017 33- Campus Visit Report - University of Bridgeport, Submitted 20 April 2017 34- Campus Visit Report – Wyoming, Submitted: 28 July 2017 35- Campus Visit Report -HEI STEM – FIT, Submitted: 3 March 2017 36- Campus Visit Report_SNHU, submitted: 16 September 2016 Data Quality Assessment Report 1- USAID Egypt 2016 Data Quality Assessment. Annex 8. Education And Higher Education Summaries And Checklists 2- IIE (Office of Education & Health (OET)/Education). February 5th, 2018. Data Quality Assessment for Institute for International Education (IIE)- Scholarships & Training for Egyptian Professionals (STEP) and Local Scholarships Program LSP (Private Universities) Activities Project Reports 1. Scholarships & Training For Egyptian Professionals. Short-Term Technical Assistance (STTA). Early Grade Reading. FINAL REPORT. January 31, 2015 2. Youth & Adult Literacy Assessment, IIE. Report completed by: Mariam Ghazala-Career Development & Consultations Academy. August 20, 2015 145 | STEP END OF TERM EVALUATION REPORT USAID.GOV Annexes: - Annex !: Career Development & Consultations Academy (CDC): Scope of Work - Annex 2: Pre-report presentation - Annex 3: Data Collection Questionnaires and Guidelines - Annex 4: Site visits Schedules - Annex 5: Security Clearances and Official Correspondence - Annex 6: Raw Data (Soft Copy) - Annex 7: Data Analysis Table - Annex 8: Arabic Language Questionnaires and Guidelines - Annex 9: Data tables - Annex 10: Photographs - Annex A1: The One Classroom School in Egypt: An Evaluative Study. Final Report.Dr. Mostafa Kamal Helmy, Minister of State for Education &Scientific Research, 1981 - Annex A2: UNICEF Egypt - Programme Profile: Education, 2015 - Annex A3: UNESCO Regional Office Beirut In collaboration with: Ministry of Education of Egypt, World Food Programme and UNESCO Office-Cairo “Friendly Schools for Children in Difficult Circumstances”. Project Document. By Dr. Higazi Idris, Program Specialist in Basic Education UNESCO Regional Office - Beirut. May 2004 - Annex A4: Arab Republic of Egypt; The National Progress Report on Literacy Adult Education Authority, 2008 - Annex A5: The Learn Link Egypt Girls’ Education Training Project, Final Report. By the Academy for Educational Development, Dr. Bruce Miller LearnLink Chief of Party, January 2001, - Annex A6: Literacy and Adult Edcuation in Egypt, Dr. Mohamed Sywelem, Department of Educational Foundations, Faculty of Education, Suez Canal University. 2015 - Annex A7 : Curriculum flyer from EAEA - Annex A8: Community Center Vision, Dr. Moheb Al Rafie, Executive Director, EAEA 3- STEP 2- Last Weekly Report to IIE. EFE.No Date 4- STEP 2- Candidates Case. EFE. No Date 5- Mid Term Performance Evaluation Report: Sustainability of Career Development Centers in Egypt: Public Universities. September 2015 Semi Annual Progress Reports: 1- Semi-Annual Progress Report (April-2014- September 2014), Institute for International Education (IIE), October 30, 2014. 2- Semi-Annual Progress Report (October 2014- March 2015), Institute for International Education (IIE), April 30, 2015. 3- Semi-Annual Progress Report (April 2015- September 2015), Institute for International Education (IIE), October 30, 2015. 4- Semi Annual Progress report (October 2015 - March 2016), Institute for International Education (IIE), April 30, 2016 5- Semi-Annual Progress Report (April 2016 – September 2016), Institute for International Education (IIE), October 30, 2016 6- Semi-Annual Progress Report (October 2016- March 2017), Institute for International Education (IIE), April 30, 2017. 7- Semi-Annual Progress Report (April 2017 – September 2017), Institute for International Education (IIE), October 30, 2017 8- Semi-Annual Progress Report (October 2017-March 2018), Institute for International Education (IIE), April 30, 2018. 9- Semi-Annual Progress Report (April 2018-September 2018), Institute for International Education (IIE), October 30, 2018. Annexes for Semi-Annual Report (April 2017- September 2017) 1- Annex 1 - HEI MBA Cohort 1 Graduate Rawan fathy Video Clip USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 146 Annexes of Semi-Annual Report (October 2017-March 2018): 1- Annex 2: List of HEI MBA Cohort 1 graduates and organisations, Semi-Annual Report (October 2017-March 2018), Institute for International Education (IIE), April 30, 2018. 2- Annex 4: Final status report of HEI MBA Job Placement Assistance Challenges, Semi-Annual Report (October 2017-March 2018), Institute for International Education (IIE), April 30, 2018. 3- Annex 5: Agenda of Amcham conference for HEI MBA Cohort 1 graduates. Semi-Annual Report (October 2017-March 2018), Institute for International Education (IIE), April 30, 2018. 4- Annex 6: Campus Visits Reports, Semi Annual Report (October 2017-March 2018), Institute for International Education (IIE), April 30, 2018. - Campus Report - University of Akron, submitted March 28, 2018 - Campus Visit Report – FDU, submitted December 21, 2017 - Campus Visit Report - Knoxville TN – RP, submitted March 27, 2018 - Campus Visit Report - Montana State University, submitted December 5, 2017 - Campus Visit Report – Rutgers, submitted: December 21, 2017 - Campus Visit Report - University of Missouri, submitted: April 24, 2017 - Campus Visit Report -Missouri State, submitted: April 25, 2018 - Campus Visit Report- Webster University, Submitted: April 26 , 2018 5- Annex 7: HEI Scholar Updates, Semi-Annual Report (October 2017-March 2018), Institute for International Education (IIE), April 30, 2018. 6- Annex 8: HEI MBA Cohort 2 Rabia’s memo, Semi-Annual Report (October 2017-March 2018), Institute for International Education (IIE), April 30, 2018. 7- Annex 11: HEI STEM Cohort 1 scholar, Mayar Allam success story, Semi-Annual Report (October 2017-March 2018), Institute for International Education (IIE), April 30, 2018. 8- Annex 12: HEI STEMCohort 2 scholar, Dina Mohamed success story, Semi-Annual Report (October 2017-March 2018), Institute for International Education (IIE), April 30, 2018. 9- Annex 14: HEI STEM Cohort 2 scholar, Afnan El Hamshari success story, Semi-Annual Report (October 2017-March 2018), Institute for International Education (IIE), April 30, 2018. 10- Annex 14: HEI MBA Cohort 2 scholar, Ola Gomaa success story. Semi-Annual Report (October 2017-March 2018), Institute for International Education (IIE), April 30, 2018. 11- Annex 16: HEI MBA Cohort 2 scholar, Riham Mahmoud success story, Semi-Annual Report (October 2017-March 2018), Institute for International Education (IIE), April 30, 2018. 12- Annex 20: IIE STEPKEYS progress report through March 31, 2018, Semi-Annual Report (October 2017-March 2018), Institute for International Education (IIE), April 30, 2018. Cohort Applications 1- HEI MBA Cohort1 Application - Omnia Ahmed 2- HEI MBA Cohort2 Application Samah Nady Ahmed 3- HEI STEM Cohort (2-A) Application - Sara Montasser 4- HEI STEM Cohort1 Application - Yara Albair 5- HEI STEM Cohort2 Application - Weam Moghazy Project Applications Template 1- Scholarships and Training for Egyptian Professionals (STEP) Program- APPLICATION FOR MASTER’S OF BUSINESS ADMINISTRATION (MBA) SCHOLARSHIPS FOR EGYPTIAN WOMEN AT UNIVERSITIES IN THE UNITED STATES* 2- STEP Selection Process - Required Criteria 3- STEP. Review Rubric 4- STEM: Agenda- Final Selection Camp. September 4-6, 2015. STEM Cohort 2 and 2A Pre￾Departure Orientation- July 17-20, 2016. STEM Reentry Workshop, August 4-6, 2017. STEM Selection Camp September 19-20, 2014. STEM Selection Camp, November 27-28, 2015. 5- Home Visit Rubric (in Arabic) 6- Rubric of Application Assessment (in Arabic) 7- Home Visit and Interview Rubric (in Arabic) 147 | STEP END OF TERM EVALUATION REPORT USAID.GOV 8- HEI MBA and STEM Selection Criteria 9- HEI MBA and STEM Selection Process 10- STEP Application (in Arabic) Project Database: 1- MBA Rejected with Justification_Samah_22Apr19 2- HEI GPA Tracker Cohort1 3- HEI GPA Tracker Cohort2 4- MBA GMAT and TOEFL tracker 5- HEI MBA Scholars’ Achievements / Success stories 6- HEI SUS Scholars’ Achievements / Success stories: 7- HEI MBA Scholars - Basic Data 8- HEI STEM Scholars - Basic Data 9- Employment Status of MBA Graduates 10- Total Number of Applications for HEI MBA and STEM Cohorts I and II 11- Job Placement Challenges EFE 12- HEI MBA and STEM Cohorts Details with Universities Names, USAID, Jan. 31, 2017 13- MBA Scholar Database November 2018 14- HEI MBA I and II Employment Status 112618 15- Expenditures per earmark 16- HEI STEM Scholars - Basic Data 17- SEPT Test MBA1 (Score Results Monday, September 22, 2014 sorted by test score) 18- SEPT Test MBA2 - Part 1 Cairo Residence, July 9, 2015, sorted by score) 19- SEPT Test MBA2 - Part 2 (Sunday, July 26, 2015, sorted by score) 20- Accepted list MBA Cohort 1 and II) 21- Rejected List MBA (Cohort 1 and II) 22- STEM Accepted List (Cohort 1 and II) 23- SEPT Test STEM1 (September 8, 2014, sorted by test score) 24- SEPT Test STEM 2 - Part 1 (Test Score Results, August 2015, sorted by test score) 25- SEPT Test STEM 2 - Part 2 (Test Score Results, September 6, 2015, sorted by test score results) 26- STEM Accepted List (Cohort I and II) 27- STEM Rejected List (Cohort I and II) 28- Accepted List 032519 29- Rejected List 032519 Non-Returnees Status and Notes 1- Non-Returnee 1 - MBA Cohort 1 2- Non-Returnee 2 - MBA Cohort 1 3- Non-Returnee 3 - MBA Cohort 1 4- Non-Returnee 4 - MBA Cohort 1 5- Non-Returnee 5 - MBA Cohort 2 6- Non-Returnee 6 - MBA Cohort 2 Scholars Applications: Accepted- MBA 1 1- Accepted 1 - MBA 1 2- Accepted 2 - MBA 1 3- Accepted 3 - MBA 1 4- Accepted 4 - MBA I 5- Accepted 5 - MBA I 6- Accepted 6 - MBA I 7- Accepted 7 - MBA 1 8- Accepted 8 - MBA 1 USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 148 Accepted- MBAII 1- Accepted 1-MBA II 2- Accepted 2-MBA II 3- Accepted 3-MBA II 4- Accepted 4-MBA II 5- Accepted 5-MBA II 6- Accepted 6-MBA II 7- Accepted 7-MBA II 8- Accepted 8-MBA II Rejected Applications- MBAI 1- Rejected 1 2- Rejected 2 3- Rejected 3 4- Rejected 4 5- Rejected 5 6- Rejected 6 7- Rejected 7 Rejected Applications- MBAII 1- Rejected Applications 2- Rejected 1 3- Rejected 2 4- Rejected 3 5- Rejected 4 6- Rejected 5 7- Rejected 6 8- Rejected 7 Accepted Candidates- STEM 1 1- Accepted 1-STEM I 2- Accepted 2-STEM I 3- Accepted 3-STEM I 4- Accepted 4-STEM I 5- Accepted 5-STEM I Accepted Candidates- STEM II 1- Accepted 1-STEM II 2- Accepted 2-STEM II 3- Accepted 3-STEM II 4- Accepted 4-STEM II 5- Accepted 5-STEM II 6- Accepted 6-STEM II 7- Accepted 7-STEM II 8- Accepted 8-STEM II 9- Accepted 9-STEM II 10- Accepted 10-STEM II 11- Accepted 11-STEM II Rejected Candidates- STEM I 1- Rejected 1-STEM I 2- Rejected 2-STEM I 3- Rejected 3-STEM I 4- Rejected 4-STEM I 149 | STEP END OF TERM EVALUATION REPORT USAID.GOV 5- Rejected 5-STEM I 6- Rejected 6-STEM I 7- Rejected 7-STEM 1 Rejected Candidates- STEM II 1- Rejected 1-STEM II 2- Rejected 2-STEM II 3- Rejected 3-STEM II 4- Rejected 4-STEM II 5- Rejected 5-STEM II 6- Rejected 6-STEM II 7- Rejected 7-STEM II 8- Rejected 8-STEM II USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 150 ANNEX IX: FINDINGS, CONCLUSIONS, RECOMMENDATIONS No FINDING CONCLUSION RECOMMENDATION 1A. To what extent does STEP through the scholarships component contribute to the achievement of USAID/Egypt’s strategic DOs, what are the success factors? F1A.1 – MBA and STEM scholars (MBAs and STEMs) rated their U.S. academic experience as very useful, with MBAs giving it a weighted average of 4.3 of 5 (n=27) and STEMs giving it a weighted average of 4.4 of 5 (n=21). The multi-cultural setting gave them self-confidence, independence, ability to take initiative, and a stronger sense of cultural perspective – “life changing.” [MBA KIIs, STEM KIIs, Online Survey] F1A.2 – Seventy-two percent of all scholars graduated with (or had achieved as of project final reporting) a 3.5 or higher GPA. [Achievement Presentation, MBA and STEM Scholar Databases] F1A.3 – The STEP scope of work lists two primary development objectives, the first is to: Improve the capacity of key government, academic and private sector organizations through the provision of scholarships, training, study tours, and technical assistance. [SoW] F1A.4 – STEP’s objectives are not specifically aligned with either of its two components; instead, they are applied across both the Scholarships and Technical Assistance components (see section 5 for findings related to the technical assistance component). [SoW] F1A.5 – There are no indications of STEP scholarship component activities directly supporting capacity building for government, academic, or private-sector organizations/institutions, other than 18 public￾sector and 8 academic-sector employees being selected for MBA scholarships. [Semi-Annual Reports 1-9, PMP 2014-2018] F1A.6 – Six MBA alumni in the public sector were reached and interviewed. Three of these MBAs returned to government positions with no substantial change in position or salary. The other three joined the public sector for the first time C1A.1 – One-hundred and fifty-eight Egyptian women have undergone a life￾changing cultural and academic experience impacting their personal and professional development. The experience of studying in multicultural settings was as important as the degree itself and led to increased self￾confidence, a sense of independence, and the ability to take initiative. C1A.2 – Seventy-two percent of all scholars with a GPA above 3.5 (as of project final reporting) is impressive and indicates strong intellectual capacity among selected program participants. C1A.3 – The first STEP primary objective – to improve the capacity of key government, academic, and private- sector organizations through the scholarships, training, study tours and technical assistance – has not R1A.1 – We recommend the exclusion of higher￾level objectives – such as building the capacity of government, academic, and private￾sector institutions – from the objectives of scholarship programs unless mechanisms and interventions are clearly defined to ensure they occur (under scholarship activities), e.g., selection of career public-sector employees devoted to government reforms. If the intention is for the TA component to be solely responsible for institutional capacity building objectives, this should be clearly articulated in the design and components aligned with distinct sets of objectives. (Design) R1A.2 – USAID implementing partners should be evaluated, at the proposal stage, for strategies and tactics to build strong relationships with scholars based on support and trust. We recommend requiring 151 | STEP END OF TERM EVALUATION REPORT USAID.GOV No FINDING CONCLUSION RECOMMENDATION (MOHP); each indicated plans to leave after several years of networking for a better salary and position with international organizations. [MBA KIIs] F1A.7 – The scholarships component was not intended to support STEP’s high-level organizational capacity-building objectives; this was the responsibility of the TA component. [CoP Discussion] F1A.8 – The second STEP primary objective is to: create a cadre of highly trained female leaders in the fields of science, mathematics, technology, and business. [SoW] F1A.9 – Sixty-two STEM degrees are being pursued as follows: 32 Engineering, 23 Science, 7 Technology, 0 Mathematics, plus 90 scholars have graduated with MBAs and returned to Egypt. [STEM Scholar Database, MBA Scholar Database, Achievements Presentation] F1A.10 – There are five primary planned outcomes for the scholarships component: [SoW] A cadre of skilled professionals who can apply and transfer new knowledge and skills in priority sectors; Increased women’s leadership and professional advancement; Strengthened technical, managerial, and leadership capabilities in targeted public and private institutions and organizations; Increased goodwill and cultural understanding through training opportunities in the U.S.; Up to 100 women with MBAs from U.S. universities and up to 60 women with bachelor’s degrees in STEM from American universities; F1A.11 – Achievement results for the five primary outcomes [SoW, MBA and STEM Scholars’ Databases, MBA KIIs, STEM KIIs, Online Survey] been measurably achieved by the scholarships component for government, academic, or private-sector organizations (excluding private￾sector companies). There were no clear activities undertaken through the scholarship component directly linked to the achievement of STEP’s organizational capacity building objectives, other than the selection of 18 scholars from the public sector and eight from the academic sector. However, of the six public-sector MBAs reached, none appear likely to assume leadership roles. The scholarships component was not realistically designed to achieve STEP’s high￾level objectives, though this was unclear in STEP’s scope of work, which applies such objectives to the project more broadly. C1A.4 – The second STEP primary objective – to create a cadre of highly trained female leaders in the fields of science, mathematics, technology and a brief plan (one to two pages) identifying relationship-building strategies, frequency, and type of contact, problem resolution mechanisms, and lessons learned addressing previous program scholar concerns. (Design and Implementation) R1A.3 – Future scholarship programs should, from the outset, include planning and activities to systematically link scholars into a cohesive network with support continued not less than six months after their return. Activities should include partially funded networking events, topical private-sector symposia, and meetings organized with policymakers and influential NGOs. The six-month timeframe will also be used to help fully integrate the alumni group into a previously agreed on host NGO, or other suitable institution interested in and capable of taking the alumni network under its wing. In the case of STEP, the National Council for Women would have been a natural fit. Other parallel options such USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 152 No FINDING CONCLUSION RECOMMENDATION F1A.12 – STEP undertook a number of activities while interacting with MBA scholars viewed as inhibiting relationship-building: STEP was perceived by 36% (n=45) of MBA scholars – more heavily weighted toward Cohort One – as having poor planning, long periods without communication and/or short deadlines for final stage applicants, e.g., “your TOEFL is tomorrow,” “you need to be here Monday.” [MBA KIIs] Policies were not consistently communicated to scholars from the outset, as noted in a 2018 project document: “Since communicating program policy has made things more clear, we recommend to share IIE/EFE|EGYPT Policy with HEI MBA Cohort 2 from the outset in order to eliminate resistance, miscommunication or misunderstanding…” [Job Placement Challenges 2018, MBA KIIs] Four MBA scholars – without a formal question being asked – indicated newly accepted applicants were required to sign a contract with STEP but were refused a copy of their own. [MBA KIIs] Seventy-one percent of MBAs (n=45) and 45% of STEMs (n=11) indicated core program objectives were communicated as commitments or promises, and not consistently fulfilled, e.g., relevant internships will be provided, you will be employed on your return, on-campus housing will be provided. [MBA KIIs, STEM KIIs] MBAs were made to believe they had returned to Egypt with unrealistic employment expectations. “Scholars came back with attitude.” [Job Placement Challenges 2018, Local Partner KIIs, MBA KIIs] Returning MBA scholars were asked to sign job rejection letters and told they would not be supported if they rejected two offered positions deemed relevant by IIE. [Job Placement Challenges 2018] business – has been numerically achieved, with 62 STEM scholars pursuing engineering, science, and technology degrees (only mathematics is not represented) and 90 scholars who have graduated with MBA degrees. However, it is premature to conclude to what extent the graduates will assume positions of leadership as not enough time has elapsed. C1A.5 – The five major scholarship component outcomes have been largely achieved: 90 culturally aware MBA graduates with enhanced managerial and leadership skills having returned to Egypt, and 62 skilled and culturally aware STEMs expected to return by the end of May 2020. C1A.6 – STEP undertook a number of activities that inhibited relationship￾building while interacting with MBA scholars, and which contributed to a less cooperative mindset during the critical job￾seeking phase: Inconsistent communication with applicants; insufficient as more systematically tying STEP alumni to IIE’s alumni network would appear appropriate as well. Such activities need to start from day one of new scholarship programs – not wait until scholars are ready to return. We recommend USAID employ these design features in future scholarship activities (medium term) and consider mitigation options for the existing MBA scholars (near term). Perhaps the planned near-term support for STEM scholars, under a costed extension for reintegration and employment assistance, could apply to MBAs as well through this or another contract vehicle. (Design and Implementation) R1A.4 – It is premature for the evaluation team to determine the likelihood of STEM scholars as long-term contributors to program objectives; we therefore recommend a longitudinal study of STEM scholars, in particular, to better understand their future impact as 153 | STEP END OF TERM EVALUATION REPORT USAID.GOV No FINDING CONCLUSION RECOMMENDATION Four MBA scholars – without a formal question being asked – indicated an outplacement consultant worked intensively with individual returning MBAs and committed to provide follow￾on support but did not contact them again, despite assurances otherwise. [MBA KIIs] MBA scholars were unsatisfied with the employment support process upon their return to Egypt. Career counseling and job placement received the lowest weighted average ratings for STEP activities at 1.9 and 1.8, respectively (5 highest) (n=27) see also section 4A. [MBA KIIs, Local Partner KIIs] Twenty-nine percent of MBAs (n=28) and 20 percent of STEMs (n=25) rated STEP staff as first or second (of five ratings) in terms of importance as a source of support. [Online Survey] F1A.13 – Forty-seven percent of MBA scholars (n=45) referenced an alumni Facebook page as their primary networking tool. No STEMs or MBAs reported active use of IIE’s alumni or partner networks aside from receiving the IIE newsletter. [MBA KIIs, STEM KIIs] F1A.14 – There were no clearly articulated actions undertaken by STEP to establish a sustainable scholar network with the possible exception of the creation of the Facebook page, an action claimed by both scholars and STEP. One scholar succinctly expressed her view that without a cohesive network she was less strong: “On my return to Egypt, I tried to contact multiple members of government to share development ideas but, as one person, I didn’t receive a single reply.” [MBA KIIs, STEM KIIs, CoP Discussion, Semi-Annual Reports 1-9] F1A.15 – IIE identifies the benefits of effectively assisting alumni to become networked, as quoted from the IFP Alumni Tracking Study: [IFP-Alumni Tracking Study 2018] “In cases where IFP alumni have been able to collaborate on activities, programs, or in organizations, they have noted a shared sense of academic counseling; mismatched university selection; short travel deadlines; failure to provide copies of contracts to scholars; unfulfilled promises about directly relevant internships; being made to believe they had unrealistic expectations on their return; inadequate job search assistance “as though for fresh graduates;” and threats to stop all assistance if two job rejections occurred – all played a role in seemingly less cooperative MBAs during the job search phase. C1A.7 – A significant missed opportunity involved the lack of cohesive network building among returning alumni (which is a separate issue from lack of networking with employers). MBA alumni represent 90 individuals each attempting to influence a small sphere of friends, family, and colleagues. STEP organized networking events on their behalf, and is said to have created a Facebook page (an action also claimed by the alumni themselves), yet the project undertook no leaders and contributors to Egypt’s development. We also recommend USAID consider an alternate STEM-style program in which more careful attention is paid to scholar fields of study – as they directly relate to labor market demand in Egypt – and for which just one or two terms of study are done in the U.S. with the remainder in Egypt. It is believed these changes will improve the likelihood that scholars will remain to their home country as potential leaders in their fields. More time in Egypt would also allow the project greater access to them for network building. (Design) USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 154 No FINDING CONCLUSION RECOMMENDATION purpose and identity that helps them work together successfully and pursue joint activities. Support of alumni networks can be very successful in bringing together catalysts of change who have a shared identity and who have common characteristics and aspirations. Promoting these alumni and their collective action can speak directly to programs’ sustainable impacts on a greater scale.” F1A.16 – Six of the STEM scholar KII respondents (n=11) indicated plans to study abroad in the U.S. or Europe to pursue post-graduate degrees after a brief repatriation in Egypt, and it is believed there will be STEM non-returnees over the longer term. [STEM KIIs, CoP Discussion] F1A.17 – STEM degrees include aerospace and biomedical engineering. STEP raised a concern with USAID about the unsuitability of the Egyptian job market for certain scholar degree choices but were told the scholars should not be limited in their selection. [STEM Scholar Database, CoP Discussion] F1A.18 – Four-year STEM students required more time to be found, selected, and returned than was available for STEP’s five-year project design. [Semi￾Annual Reports 1-9, STEM Scholar Database] F1A.19 – A costed extension (January 2019 through September 2020) for continuing support to STEM reintegration and employment is ongoing under the USAID Local Scholarships Program (LSP) with four to five staff and managed by a former STEP employee. One local partner is preparing to begin working with the first cohort of STEM scholars on their return in May. [CoP Discussion, Local Partner KIIs] Desk Review Overachieved scholars in career directions program, 35 of 20 FY17 and 30 of 20 FY18 [PMP17, PMP18] Achieved enrollment in alumni networks, 100 of 100 in FY17 and FY18 [PMP17, PMP18] clear process or actions to actually establish a sustainable network – something significantly greater than conducting simple events or establishing a webpage. The result is an alumni group with virtually no leverage – as a group – to raise its profile and share its perspective with policymakers, civil society, and private￾sector organizations. This represents one of STEP’s most notable disappointments and greatly diminishes the potential influence the scholars may have had to more significantly contribute USAID’s and Egypt’s broader development objectives. C1A.8 – A questionable design issue exists regarding sending 17- to 19- year-olds from disadvantaged backgrounds to a country with a radically different culture for four full years. The issue is whether there is an acceptable likelihood of their permanent return to Egypt in support of the country’s development objectives, particularly when their degrees include biomedical 155 | STEP END OF TERM EVALUATION REPORT USAID.GOV No FINDING CONCLUSION RECOMMENDATION 35 of 98 MBA scholars are employed [MBA Employed Status list] Career guidance includes pre and post assessments, five-day career direction workshop, career counseling skills [Sem-An Rep17] Challenges / mitigation: In mid-2017, there were four cases of scholars facing challenges, two abuse related, one health and one legal; STEP directly involved in mitigation efforts [Sem-An Rep17] See other challenges under 1B. IIE uses seven locally contracted partners to deliver services to scholars. [All Semi-Annual Reports] IIE, in collaboration with implementing partner EFE, implemented a one-week Career Directions workshop for HEI MBA cohort 1 graduates. It was tailored to help returning graduates augment their skills for reintegration in the workforce [Semi￾Annual Report April16-Sept.17] SOW section 4.2.4 requires post-training support, follow-on activities, and technical assistance. This includes action planning for participants and follow-up support – both offered to scholars [Semi-Annual Report April18-Sept.18] 35 of 45 returning Cohort 1 MBAs participated in Career Directions workshop to further build practical skills [Sem-An Rep17] Two active STEM cohorts remain as of Dec 2018; USAID therefore extended the contract to support them through Sept. 2020 [Sem-An Rep17- 18] 18 universities were used for 62 STEM scholars and 16 universities were used for 96 MBA scholars, though final presentation states 41 total universities [Scholar Database, Achievement Pres] 62 STEM degrees are being sought: 32 Engineering (including aerospace), 23 Science (including biomedical engineering), 7 Technology, 0 Mathematics [STEM Scholar Database] engineering, aerospace engineering, and interests toward applied research, all of which offer limited employment opportunities in Egypt. It is also not possible to locate, select, and return students pursuing four-year degrees to their home country within a five￾year project horizon. This requires finding alternate mechanisms to support their reintegration, as has been done with the costed extension to support STEM scholars on their return. Over half of STEM seek engineering degrees - this appears overweight as Egypt has significant engineering training capacity. Career counseling did not dissuade STEM scholars from pursuing degrees in fields that Egypt does not have the capacity to employ them in, e.g., biomedical engineering and aerospace engineering. An industry/career needs assessment may have prevented this. USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 156 No FINDING CONCLUSION RECOMMENDATION There is regional diversity in the 34 (or 41) U.S. universities that received scholars across 24 states [Sem-An Rep17, Achievements Presentation] In 2017, 41 MBA and 43 STEM scholars took part in internships April – Sept. 2017 [Sem-An Rep17] STEM internships: 127 (89 in Egypt) [Achievement Pres] MBA internships: 137 (70 in Egypt) falls short of two per-scholar target [Achievement Pres] 72 percent of all scholars graduated or currently have a 3.5 or higher GPA [Achievement Pres] Challenges / mitigation: Difficulty with security clearances / achieved participation and buy-in from MOE [Sem-An Rep15] Scholars proceeding then stopped by English proficiency / began requiring TOEFL scores [Sem￾An Rep15] 12 early program dropouts / increased notice period to employers to allow vacation time allowances to be approved by managers [Sem-An Rep15] Early MBA participants scored low on GMAT / more GMAT practical prep time allocated [Sem￾An Rep15] Internships for scholars in U.S. not consistently found due to changes in labor law and fewer unpaid available, plus most firms don't offer internships until after junior year / STEP allowed paid internships and internships upon returning to Egypt [Sem-An Rep17-18] Returning MBAs are reported to have unrealistically high expectations about job prospects and salaries, often rejecting non￾management offers or those not precisely fitting their qualifications / STEP offered coaching -- outcomes unclear [Sem-An Rep17-18] Internships needed more thought and planning as U.S. unpaid internships were not available and, for STEMs, most companies wait until junior year. One-month internships (as designed) are of questionable value. We need to examine the value of one￾month internships - do some benchmarking. MBAs’ new soft skills and analytical skills are being utilized to the extent possible—more fully in international organizations—but Egyptian organizations are interested in practical knowledge and use this as an excuse to offer MBAs entry-level positions. The lack of organized, systematic networking is a significant missed opportunity. MBAs have had little sustained mentoring to achieve a cohesive network or develop any systematic networking tools. Significantly more emphasis on internship suitability/length and practical employment networking and 157 | STEP END OF TERM EVALUATION REPORT USAID.GOV No FINDING CONCLUSION RECOMMENDATION 2018 continuing challenges with U.S.-based scholars included health/injury (three), program non-compliance (one), legal (one), class failure (one), asylum (one) / STEP remained closely involved in each and proposed solutions -- outcomes not known in all cases [Sem-An Rep17- 18] Program design dictates four years of study in the U.S. for STEM cohorts, typically between the ages of 17 and 19 upon program acceptance [Semi-An Rep April-Oct.14, STEM KIIs] STEP activity directly contributes to the Foreign Assistance Strategic Framework Functional Objective “Investing in People” and to the USAID forward focus on building the capacity of local institutions. It is also designed to serve objectives and expected results under HEI Bilateral agreement. One of its expected results is Increased Employability of Tertiary Education Graduates. One of STEP’s two objectives: Improve the capacity of key government, academic, and private￾sector organizations through the provision of scholarships, training, study tours, and technical assistance and create a cadre of highly trained female leaders in the fields of science, math, technology, and business. KIIs MBAs were unanimous that their U.S. experience was life-changing. It imparted increased self￾confidence, independence, and increased their sense of cultural perspective. Being alone in a different country was the main factor. [MBA KIIs] The majority of employed interviewees are using their new skills and knowledge to some extent in their current jobs. Those in international organizations are most able to use the full extent of their hard skills, whereas improved soft skills are in use by all MBA alumni. [MBA KIIs] Egyptian employers favor practical experience over MBA credentials and are slow to adopt the introductions need to be done. STEP project design has not paid enough weight to contribute to the HEI’s expected results of increasing employment among the program graduates. The program design falls into a linear mode where job placement and employment are dealt with as an exit phase with no tight link to program outreach, applicants’ selection, and university placement to serve HEI’s expected results STEP did not advise students on their major choices and their links to potential jobs in the Egyptian market, hence creating a missed opportunity when seeking employment. While IIE’s stakeholders hold an extensive experience in managing different aspects of the program, the partners are not synchronized in leading them to focus on the program deliverables. This is a missed opportunity to utilize the. USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 158 No FINDING CONCLUSION RECOMMENDATION kinds of practices learned by MBA scholars. [MBA KIIs] MBAs without directly relevant practical experience are typically offered entry-level positions on their return to Egypt, especially in non-international organizations; one interviewee summed this phenomenon up as “the limiting nature of the Egyptian labor market.” [MBA KIIs] Networking among MBAs is ad hoc; no systematic use of IIE, USAID or other networks. Most common use is of university networks and the Facebook network set up by cohorts [MBA KIIs] A majority of MBAs believed internships should have been longer and more directly relevant; career counseling more pronounced from the beginning to help select appropriate majors; and outplacement more practical with networked contacts. [MBA KIIs] Services arranged in the U.S. were stronger than those arranged in Egypt but duplication or overlap in services provided was never an issue. [MBA KIIs] Scholars were not given any academic advising when choosing their majors. [MBA, STEM KIIs] IIE’s partners function in a limited scope, often leading to lack of synchronizing between them. Partners are not aware of each other’s activities, who does what, or the bigger objective of the program. [local partners KIIs] Two partners (AUC and EFE) wished to have been more involved with the students (needs assessment) to better design and tailor to their needs. [local partners KIIs] Scholars and local partners confirmed STEP did not facilitate a learning and sharing environment to partners involved in servicing MBA and STEM scholars. [Local Partner KIIs, MBA KIIs, STEM KIIs] 159 | STEP END OF TERM EVALUATION REPORT USAID.GOV No FINDING CONCLUSION RECOMMENDATION 1B. To what extent does STEP—through the scholarship component—contribute to the priority needs of Egypt’s key government, academic, and private-sector organizations, and what are the success factors? F1B.1 – Egypt’s comprehensive Vision 2030 strategy is based on 12 strategic pillars, four of which are related to STEP: Education and Training, Knowledge, Innovation and Scientific Research, and Health and Economy. [Egypt Vision 2030] F1B.2 – The Vision 2030 Education and Training Pillar (applicable to MOE) includes: [Egypt Vision 2030] Empowering the learner with the requirements and skills of the 21st Century. Providing comprehensive and sustainable professional development for teachers. Developing curricula to cope with global developments and information updates, taking into consideration the age of the learner. Developing the organizational structure of the ministry, educational directorates, departments, and schools to improve educational services. Using effective technological techniques to present and share knowledge among students and teachers. F1B.3 – The Vision 2030 Health Pillar (applicable to MOHP) indicates: Egypt must reduce its rate of Hepatitis C; address the healthcare system crisis resulting from a lack of separation between roles (planning, executive leadership, financing, and regulation); improve governance and accuracy of data; strengthen efficiency, accountability, transparency, and resource management. [Egypt Vision 2030] F1B.4 – The Vision 2030 Higher Education Pillar (MOHESR) includes five pillars: Education, Scientific Research and Innovation, Building Human Character, Health, and Society Service. STEM education is also a priority; MOHESR plans 22 new universities, eight specialized in technology by 2030. [Egypt Vision 2030] C1B.1 – STEP scholarships support the Vision 2030 Education and Training Pillar: providing the necessary skills to students and trainees to think creatively and empower them technically and technologically. Scholarships do not measurably support Egypt’s other Vision 2030 objectives in the short to medium term. Any high-level objectives or institutional capacity building (outside higher education) is within the realm of the TA component (see section 5). One means to add capacity to the public sector, improve transparency, and strengthen governance—through the scholarship program—is to target a number of public￾sector employees for program inclusion. There is no clear indication this was done. MBAs are realistic that they cannot affect Egypt’s development in a measurable way, but they can attempt to USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 160 No FINDING CONCLUSION RECOMMENDATION F1B.5 – Strategic pillars reviewed for MOE, MOHP, and MOHESR identify each as having objectives in alignment with STEP’s TA component outcomes (see also section 5). The scholarships component supports the education and training pillar under MOHESR’s purview. [Egypt Vision 2030] F1B.6 – Twenty-two MBA scholars (n=45) assessed themselves as mainly able to influence their immediate sphere of colleagues, family, and social contacts but realistically unable to affect Egypt’s broader development in a measurable way. One succinct response: “The mindset in Egypt will need to change over next decade for us to truly have influence beyond our immediate friends and colleagues.” Moreover, it is the opinion of the former CoP that the scholarships component was not designed to address high-level objectives; these were the domain of the TA component. [MBA KIIs, CoP Discussion] Desk Review There is a clear STEP scholar objective to build capacity in the public sector and improve governance yet no clear implementation activities to support this. [PMP Narrative, All Semi-Annual Reports, Scholar KIIs] KIIs Responding to how MBA scholars view their potential contribution to Egypt’s development, they assess themselves as mainly able to influence their immediate sphere of colleagues, family, and social contacts [MBA KIIs] Egypt-based internships were received as mostly ineffective. One scholar mentioned that the culture of free internship is not there in Egypt yet. Employers are not ready to share the know-how with interns, since these are temp staff who will not make it into employment. influence their immediate colleagues and social circles to the best of their ability, and nearly all of them claim they are doing this. One very succinct response: “The mindset in Egypt will need to change over next decade for us to truly have influence.” MBA and STEM scholars were not given sufficient career guidance on the selection of their specializations in light of succinct technical priorities for academic and vocational needs in Egypt. Career shifts were allowed without being clear on the consequences or plans for placement in the job market. The number of scholarships and the cluster of scholars around the same technical areas (especially for MBAs) could lead to internal competitions between scholars in seeking employment opportunities— especially with the majority of the MBA scholars ending up in Cairo. 161 | STEP END OF TERM EVALUATION REPORT USAID.GOV No FINDING CONCLUSION RECOMMENDATION Internships in Egypt were the least effective component in the scholarship experience. It was meant to be part of job placement, but it ended up being a burden on both the project and the scholars who had to take Egypt-based internships even if irrelevant to their area of specialization or even if not providing them with real values for employment to get done with this condition in their scholarship agreement. Egypt-based internships involved costs of applicants’ accommodation and allowances. A possible link could be seen between the MBA scholarships and government objectives. One big leak though is the low salaries offered by the government to MBA holders, which could force them to leave their government posts and accept job offers at international organizations that offer much higher salaries, especially for those who work near senior positions at the government given their knowledge of government USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 162 No FINDING CONCLUSION RECOMMENDATION procedures and plans. So, the government tends to lose returned MBA holders after some time working in the government, either because they take offers with international organizations or employment opportunities abroad due to much higher salaries. 2A. To what extent does the beneficiary selection process meet the requirements agreed with USAID? F2A.1 – Outreach: Outreach efforts to solicit applications to academic studies in the U.S aim to focus on underserved and rural areas. Programmatic efforts were planned to reach out to 14 governorates for regional diversity [HEI Bilateral Agreement, PMP Narrative]. F2A.2 - The Selection Process: [Semi-Annual Reports 14-15, Selection Criteria, Selection Process, Cut-off points from CoP, MBA KIIs – apply to all below] A structured candidate selection process and selection criteria were shared with USAID and revised. The selection process as explained in STEP reports has three main filtration points: Stage 1: Achieving a cutoff score for financial eligibility at the application stage. Stage 2: Achieving a cutoff score on a language test. For MBA/STEM Cohort 1: EPE score (an IIE language proficiency test) with a cutoff score of 80 for applicants in Greater Cairo and Alexandria and 74 for the governorates. For MBA/STEM Cohort 2: TOEFL/IELTS score (cutoff score 450/39/4.0). C2A.1 – The outreach process to inform potential applicants of the STEP scholarship opportunity appears to have reached a wide array of people via online vehicles for youth combined with newspapers and regional visits for governorates. The large number of applications received indicates that project outreach efforts certainly generated interest. However, the small percentages of STEP scholarships actually accepted are disproportionate to the outreach results. (9,000+ MBA applications with 117 selected as finalists and 800+ STEM applications with 69 selected as finalists) indicating a more targeted outreach R2A.1 – We recommend a more targeted approach to outreach with target recipients defined with a demographic profile. A quota system should be considered to ensure underserved governorates are adequately represented. And MBA scholarships should be announced through government entities and universities to attract eligible candidates working within them, which will contribute to the objective of strengthening the performance of government and academic institutions. (Implementation) R2A.2- The selection process for MBAs and STEMs should be proposed to clarify 163 | STEP END OF TERM EVALUATION REPORT USAID.GOV No FINDING CONCLUSION RECOMMENDATION Stage 3: Achieving a cutoff score of 2.5 on an individual interview. Group activities are used to screen personality attributes such as group work, initiation, leadership, and creativity; however, it is not clear how group activities scores are weighted. The selection process was viewed by 11 MBAs (n=45), especially Cohort 1, as disorganized and poorly communicated with lengthy periods of no information followed by tight deadlines. Examples from MBA Cohort 1 KIIs: scholars told to report to English TOEFL in five days; had no idea for six months if they were wasting their time; given two days to show up for interview; given one-week notice for international travel, given three days’ notice over weekend for travel from Sohag to Cairo; short notice for applicants to organize themselves for six-months of training at AUC. F2A.3 – Academic Eligibility: The academic performance eligibility criteria for STEM had a minimum acceptable score in the Thanawya Amma exam of 90 percent with a minimum acceptable average grade of 90 percent in science and math. For MBAs, the minimum accepted eligibility criteria for academic performance ranged from moderate (“good”) to low (“pass”) with a diploma. The broad eligibility requirements resulted in a large number of applications. [MBA & STEM Selection Criteria, Achievements Presentation] F2A.4 – Financial Eligibility: The “disadvantaged” financial criterion was used for pass/fail eligibility rather than as late-stage screening and awarded extra points to the very needy. Sub-findings: STEP developed a financial rubric on the application form to assess applicants’ “disadvantaged” financial status. Highest scores were awarded to monthly family incomes of LE 750 or less, candidate salaries of LE 1000 or less, a rented house/apartment, monthly expenses at 75 percent to 100 percent of monthly income, and young applicants working. [Application Packets, Financial Status Rubrics] campaign may have been more efficient and helped streamline the selection process. C2A.2 – The selection process needed more effective planning, communication, and transparency for candidates to know when results would be available, to plan for the life changing activity before them. C2A.3 – The 90- percent academic achievement for STEMs may be too stringent to take into account a balance between study, leadership, and extra￾curricular activities. C2A.4 – Financial Eligibility: The term “disadvantaged” is not applied according to a clearly defined rationale that ensures the most qualified applicants will be selected. It is clear some middle, even upper-middle class applicants were able to gain acceptance, indicating the need for a more consistent and understandable application processes. Also, the negative connotation of “disadvantaged” needs consideration in actual practice, because it and tighten up-front eligibility criteria. These could include years of work experience, age range, and limit (for MBAs), relative weight between academic performance and leadership potential (for STEMs), and defined leadership roles in public and community services (MBAs and STEMs). (Implementation) R2A.3- A clearer definition of “disadvantaged” applicants should be established with the notion that potential leaders may be located in underserved areas but, in some cases, be more deserving of scholarships than those given extra points for living in rented flats or earning less than 1,000 LE /month. Financial status should be used as a screening measure after leadership skills are evaluated and not as criteria for eligibility pass/fail. (Implementation) R2A.4- STEM and non￾STEM school applicants should be dealt with as two separate tracks under STEM scholarship programs with a quota USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 164 No FINDING CONCLUSION RECOMMENDATION Home visits were conducted for STEM candidates not for MBAs. STEP did not perceive home visits appropriate for the MBAs’ age group. Home visits to STEM candidates were not perceived by STEP and its local partners as the best way to verify financial status, e.g., some addresses were given as home visits which were not the actual residence. Home visits were considered most important to verify parents’ consent. [Semi-Annual Reports 1-4, Local Partner KIIs, COP Discussion] Two local partners viewed the selection of scholars with very low financial status as inappropriate for U.S.-based study: “When you focus on the very needy, the culture shock is too much, and they often break out from their communities, it is hard for them to be reintegrated…focus on scholarships inside Egypt for such people…. the pressure from the parents and community is huge on reentry….this can contribute to non-returnees.” [Local Partner KIIs] F2A.5 – Scholar diversity challenges: Large numbers of applications were received versus those selected: One percent of applications submitted were selected for MBA Cohort 1; 1.8 percent of applications submitted were selected for MBA Cohort 2; 7.5 percent of applications submitted were selected for STEM Cohort 1; and 8.3 percent of applications submitted were selected for Cohort 2. [Achievements Presentation] For geographical diversity, 57% of MBAs (n=56) and 52 percent of STEMs (n=32) live in Greater Cairo and Alexandria while the rest are from 16 other governorates. [Scholar Database] Fifty-three percent of STEM scholars (n=33) are selected from Maadi STEM schools. The MOE recommended recruitment be open to all high school students or graduates with required English proficiency, not just STEM school graduates. [Semi-Annual Report 4, Scholar Database] For sector diversity, there is no mention in STEP reports about programmatic efforts to announce may have served as a barrier to people with a well-developed sense of pride. Finally, while STEP did rely on a set of standard rubrics, some of them appear arbitrary for the purpose at hand and/or not easily verifiable. C2A.5 – Diversity: The overweight percentage of scholars from the Maadi STEM School, and MBAs/STEMs more generally from Greater Cairo or Alexandria are at cross purposes with diversity objectives. Scholars from 16 governorates appears diverse, yet small numbers from each indicate otherwise. Among MBAs, five governorates are represented by just two MBAs each and six governorates by just one each. A similar situation exists for STEMs. [MBA & STEM Scholars Database] C2A.6 – The selection process: The selection criteria and process rely on identifiable rubrics and cutoff scores but are not always consistently applied or, if exceptions were required, the reasons for the number of scholarships available for each track. A selection process and criteria should be developed for each track so applicants are competing against each other and not with applicants in the other track. This will enhance the number of non-STEM school applicants who are selected for STEM scholarships. (Implementation) R2A.5: Relative weighting should be given to screening criteria such as for academic performance, underserved applicants, leadership skills, personality traits, language proficiency and GMAT scores, since such criteria should not necessarily all be judged equally. Applicant performance during the extended AUC program (or similar in future) should also be considered as an opportunity for late￾stage screening. (Implementation) 165 | STEP END OF TERM EVALUATION REPORT USAID.GOV No FINDING CONCLUSION RECOMMENDATION MBA scholarships at government entities or universities to attract candidates. Sixty-two percent of MBA scholars are working in the private sector, 19 percent in the public sector and 8 percent in academic institutions (the remainder are unemployed, self-employed, work with NGOs, or undefined). The former CoP stated government employees were not targeted as there was another large program offering MBA scholarships to them. [Scholar Database, CoP Discussion] F2A.6 – Selection of potential leaders: Leadership skills for MBA and STEM applicants were assessed against the number of volunteering community services they performed together with applicants’ answers to essay questions on the application form. Candidates were also screened against a set of skills-based criteria through group activities and individual interviews. [MBA and STEM Application Packets, Application Scoring Rubrics, Semi-Annual Reports 1-4] F2A.7 – The university Preparation Program (UPP) process: All MBA scholars attended UPP for six months at the AUC and prepared for TOEFL and GMAT tests to meet admission requirements by U.S.- based universities. [Semi-Annual Reports 2-5] MBAs with non-scientific backgrounds in some cases retook the GMAT up to three times to improve their scores, and a waiver was negotiated by IIE with some schools. [GMAT Tracker, CoP Discussion] Only non-STEM school students were placed in an AUC preparation program for nine months due to their less advanced scholastic system. The compressed timeframe to send Cohort 2A to the U.S. required hand picking from the STEM Maadi School, which potentially eliminated opportunities for students from non-STEM schools. [Semi￾Annual Reports 3-6] F2A. 8 – Selection process inconsistencies: [MBA and STEM Application Packets, Selection Criteria, not documented. The use of cutoff scores for two different language tests to select candidates within the same cohort, but only in some instances, is not clearly explained/documented. Acceptance of candidates from STEM schools and experimental schools may be perfectly rational, though these too raise questions about how needy these applicants were versus those who could never consider such educational opportunities. How were these decisions made? Certainly outreach efforts were widespread and generated considerable interest. The apparent result of these inconsistencies includes the acceptance of applicants from middle- to upper￾middle-income families. C2A.7 – Selecting STEM scholars from STEM schools alongside government, experimental, and Azhar schools is not seen as equitable competition since screening was done based on academic achievement across USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 166 No FINDING CONCLUSION RECOMMENDATION Selection Process, Cutoff Points Document – apply to all below] Twelve of 15 rejected MBA applicants achieved cutoff scores on the language tests. The reason given for rejection was that they did not achieve the cutoff score on another language test (SEPT score). It is not clear why the additional SEPT test was required if they met TOEFL/IELTS cutoff scores. Four MBA applicants (n=15) who met the language cutoff score and who met the cutoff score during individual interviews (per the documented selection process) were still rejected for language. One selected STEM candidate did not meet the academic application cutoff score; she obtained 30 while the cutoff score was 60. The financial eligibility cutoff scores for both MBAs and STEMs are unclear. At least seven selected STEMs (n=15) achieved low scores on the financial applications that should have disqualified them (the lower the score, the less financially disadvantaged the applicant is). Desk Review According to the HEI bilateral agreement, recruitment outreach will be focused in underserved and rural areas. [HEI Bilateral Agreement] STEP planned to conduct efforts in 14 governorates, including underserved, to solicit applications to academic studies in the U.S. [PMP narrative 14] NM provided Career Self-Assessment sessions (psychometrics) to STEM cohorts to identify fields of study related to interests, career goals, and areas of strength – then discussed during one-on￾one career counseling/coaching sessions with a certified advisor. [Semi-annual, March16] STEM & MBA recruitment and selection are said to follow a structured seven-phase process, which educational systems of different strengths (Thanaweya Amma versus STEM Assessment System). Applicants selected from STEM schools didn’t attend a bridge year like those from experimental and public schools, since STEM students will generally achieve higher scores on TOEFL and GMAT given the more advanced education they receive. On the surface, scholars from 20 governorates appears diverse, yet the small numbers from each tell a different story (in many cases only one or two scholars per governorate) Since only 1.3 percent of MBA and 7.2 percent of STEM applicants were accepted, the initial filter was a crucial selection factor. Needs examination by team. (this could be further elaborated on as screening of the packets are done). The six-month MBA prep at AUC appears intensive and was likely a key factors to scholar academic success, but it may 167 | STEP END OF TERM EVALUATION REPORT USAID.GOV No FINDING CONCLUSION RECOMMENDATION along with selection criteria, were shared with USAID and revised. [Sem-An Rep Apr14-15] MBA graduation ceremony broadcast by Cairo TV news intended to increase awareness and understanding. [Sem-An Rep17] Cohort 2 STEM groups represent 12 governorates and are selected from USAID supported STEM boarding school for girls in Maadi, language schools, governmental schools, and local public universities. [Semi-annual March16] 96 MBA scholars entered as: one unemployed, two self-employed, three NGO, five undefined, 22 public-sector, 63 private-sector. [MBA database] MOE recommended recruitment be open to all high school students or graduates with required English proficiency and not just STEM school graduates. The additional group was selected with high language proficiency level and was not enrolled on AUC’s UPP [Semi-annual March16] Scholars selected from 20 governorates, sent to 41 universities in 24 U.S. states [Achievement Pres] 1.27 percent of MBA applications and 7.2 percent of STEM applications were accepted for program inclusion [Achievement Pres] MBA scholars were offered six months of preparation prior to departure, STEMs received nine months with cohort 2/2A receiving an additional six weeks of academic life skills. [Achievement Pres, Semi-annual March16] Regional info sessions, newspapers, social media, USAID/ministry websites, brochure distribution all used for outreach. [Sem-An Rep April-Oct.14] Inability to get clearance for public school visits were a challenge to outreach efforts. [Sem-An Rep April-Oct.14] Nine of 24 Cohort 1 STEM were put into nine￾month bridge program. [Sem-An Rep April-Oct14] have also been a filter preventing some MBAs from being able to apply due to loss of income. (This could be more true for MBAs, since most of the STEMs were students at the time of selection. The fact STEM Cohort 2 required six additional weeks indicates an unmet need. Requires investigation by team. STEP struggled with security clearance for access to public education institutions. This slowed down the start phase and made partners uneasy. Efforts to clarify need to start earlier by USAID and IP. (This also could indicate a limitation in the outreach). For the first STEM cohort, only nine of 24 were put into the nine- month prep programs. Needs examination by team. The heavy weighting favoring the poorest of the poor (disadvantaged communities) may be at cross purposes with the best STEM applicants, e.g., is it realistic or even USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 168 No FINDING CONCLUSION RECOMMENDATION The “disadvantaged” criterion was used as a primary pass/fail eligibility factor rather than a late￾stage screening measure and awarded extra points to the poorest of the poor. [Application Packets and Rubrics] “Disadvantaged” for STEM was evaluated according to factors such as ownership of a car (regardless of condition or value), ownership of a dwelling (whether inherited or not), income increments of 1,000 LE per month (points awarded to lower amounts), and whether or not the applicant earned a living supporting the family. [Application Packets and Rubrics] Private school graduates were exempted from both STEM and MBA program participation. [All Semi-Annual Reports] STEM acceptance criteria included a 90-percent academic achievement record [cite] KIIs MBA scholars perceive program outreach to have reached a diverse group of “disadvantaged” people via multiple vehicles: Facebook and NGO/development websites (effective for youth), newspapers, and regional visits. (effective for governorates) [MBA KIIs] MBAs spoke of “disadvantaged” as a negative term and believe the concept could have been communicated more effectively. [MBA KIIs] The selection process was viewed by MBAs, especially Cohort 1, as disorganized and poorly communicated with lengthy periods of no information followed by very short deadlines. Examples: told to report to English TOEFL in five days, had no idea for six months if they were wasting their time; given two days to show up for interview, given one week notice for international travel, given three days’ notice (over weekend) for travel from Sohag to Cairo. [MBA KIIs] MBA cultural preparation was more favorably viewed; “STEP was helpful and did the best they could” to prepare them for the culture shift, desirable to award extra points to 17- year-olds who are forced to work? The disadvantaged criterion was not consistently applied, as some daughters of affluent families succeeded in being accepted. Clear definitions are needed for “disadvantaged.” Exemption of private￾school graduates is a clear criterion for screening. The 90-percent academic achievement for STEMs may be too stringent to take into account a balance between study, leadership, and extracurricular activities. The “disadvantaged” objective in itself may characterize a design flaw, as the culture shock for lower income/regional women may be more difficult to overcome and could affect academic performance. “Disadvantaged” should have been communicated in a way that simply implied an opportunity for women who do not have access to MBA studies, as it was 169 | STEP END OF TERM EVALUATION REPORT USAID.GOV No FINDING CONCLUSION RECOMMENDATION though comments were made that an American should have delivered the cultural orientation. [MBA KIIs] A majority of MBAs viewed the bridge program as an important skill builder for their U.S. experience. Some with business or math/science backgrounds found it a waste of time. [MBA KIIs] A majority of STEM scholars were selected from the Maadi STEM school. [STEM KIIs] Some STEM scholars are clearly from affluent families. [STEM KIIs] Selecting STEM scholars from STEM schools as well as government, experimental, and Azhar schools is not seen as fair competition since screening was done based on TOEFL test scores and academic achievement. STEM students are expected to achieve higher marks given the quality education they receive at STEM schools. Picking STEM students to go on STEP scholarships from only one school out of (14 STEM schools, need to validate how many of these 14 schools are for girls) in Egypt violates equality among STEM females. in Egypt. Some rejected MBA scholars from Cohort 1 were allowed to apply for Cohort 2 and sat during the selection process knowing all activities and accepted answers for these activities. Selection activities were assessing applicants possessing multiple skills without clear observation rubrics for the observed skills. Preparing MBA scholars for the GMAT tests at AUC was seen as inefficient, especially placing MBA scholars both with and without math backgrounds in the same class. Short notice given to applicants to free themselves for a six-month training program at AUC was seen as a weak factor in the program design especially that those who left their employment did so on the promise that they would get the program support to be employed upon coming back. A viewed negatively by beneficiaries and may have been a barrier to some. The term may have been better used only internally as a selection criterion. The mixture of media vehicles used to inform potential applicants of the opportunity appears to have reached a diverse array of applicants – online vehicles for youth combined with newspapers and regional visits to reach governorates. The selection process needed more effective communication, transparency, and planning to help the applicants to plan for the life-changing activity before them. (see examples under KII findings). Cultural preparation was thought to be genuinely helpful, and this practice should be continued. Use of an American and/or scholarship alumni may be useful. Efforts to reach STEM applicants in governorates were inadequate to equitably offer opportunities to those in regions – Maadi USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 170 No FINDING CONCLUSION RECOMMENDATION feeling of resentment is high among those who left their employment for the MBA program and couldn’t find a suitable job after they came back. Some MBA scholars wished they were granted a master of science instead of an MBA to be able to find employment in their areas of specialization. Science students complained that they could have focused their time on better things. Verbal part of the GMAT was not useful and the students had to depend on themselves in that component. [MBA KIIs] STEP scholars reporting not knowing the bigger picture or the steps of the selection process and what it entailed. Often leading to frustration when informed on a short notice. [MBA KIIs, STEM KIIs] Both MBA and STEM scholars did not know the reason for their selection. Most scholars believe that the outreach efforts have been effective but extra measures could have been considered. The term “disadvantaged” is explained by scholars as “inability to fund international studies.” STEM was too heavily relied upon as a source of applicants. The MOE may have provided an enhanced vehicle to reach governorates more effectively. Placing non-STEM students only in an AUC preparation program is seen to be a less effective approach in giving all STEM scholars the same opportunities for developing soft skills and university academic skills even if they look better in English and GMAT than their non-STEM colleagues. For MBA scholars, selecting candidates with scientific and non￾scientific backgrounds for an MBA program poses challenges and missed opportunities at different levels. Those MBA scholars who managed to find or go back to their employment with the government are more likely to have an opportunity to transfer their knowledge and skills gained in the MBA program to their jobs. Even if their positions at the government don’t readily allow 171 | STEP END OF TERM EVALUATION REPORT USAID.GOV No FINDING CONCLUSION RECOMMENDATION this, they seem to be ready to take a chance to reveal their new capabilities when they are asked to take on MBA-related responsibilities. 2B. To what extent do applicants meet the criteria for scholarship awards? F2B.1 – USAID’s Empowering Youth Priority: One of the six principles in USAID’s education policy is to focus investments on measurably and sustainably improving learning and educational outcomes. This principle is served by USAID’s priority for empowering youth to gain the skills they need to lead productive lives, gain employment, and positively contribute to society. STEP objectives contribute towards this principle and priority in USAID education policy. [USAID Education Policy 2018] F2B.2 – Potential leaders: Seventy-seven percent of MBA scholars (n=96) achieved a GPA of 3.50 to 4.0, whereas STEM scholars (in-progress grades) indicate 64 percent (n= 62) have achieved a GPA of 3.5 to 4.0. [MBA KIIs, STEM KIIs, Achievement Presentation, GPA Tracker] Eighty-two percent of MBAs (n=45) and 96 percent of STEMs (n=11) believed cultural integration during their U.S.-based universities was a life changing experience. [MBA KIIs, STEM KIIs] Despite challenges finding unpaid internships in the U.S., 100 percent of scholars who were not enrolled in summer classes succeeded in securing an internship at a U.S. organization in their city of residence; however, 66 percent MBAs (n=45) believe internships should have been longer and more directly relevant. [MBA KIIs, Semi-Annual Report 5] Twenty-four percent of MBA scholars (n=45) stated they took the initiative to find a job themselves and 7 percent (n=45) started new C2B.1 – Potential Leaders: STEP scholars performed very strongly in their US programs evidenced by high academic achievements, social integration, and determination in seeking internships. There is also some evidence of leadership and becoming involved in social/community affairs; however, not enough time has elapsed to properly gauge this more broadly. C2B.2 – Career Guidance: While activities such as career self-assessment, career counseling, and action planning were provided to MBAs and STEMs, satisfaction rates with these activities were low to moderate. Efforts to better respond to scholars’ career counseling needs should have been intensively applied from the time they were accepted into R2B.1 – We recommend strengthening and extending the career counseling function to ensure adequate time is given to scholars toward aligning their interests, work experience, university selection, and necessary internships (and likely availability) – all tied back to a realistic assessment of employment market demand. Then ensure their action plans have specific milestones for each step in their career development process. (Implementation) R2B.2 – Facilitate networking activities for MBAs and STEMs alumni where they can learn from each other’s career and job￾market experiences. Encourage alumni to advise new program entrants. (Implementation) USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 172 No FINDING CONCLUSION RECOMMENDATION activities upon coming back. One MBA scholar commented: “After I came back, I started my Academy at El Khanka, this was due to the knowledge, skills, and courage gained during my MBA experience. My way of thinking is now different.” [MBA KIIs] F2B.3 – Career guidance: The STEP program provided career self￾assessment sessions to STEM cohorts identifying fields of study related to interests, career goals, and areas of strength by applying psychometrics tests then discussed these during one-on-one career counseling/coaching sessions with a certified advisor. Forty-five percent of STEMs (n=22) found it useful. [Semi-Annual Report 4, Online Survey] One-week career counseling workshops for MBA graduates were conducted on their return to Egypt to augment their skills for reintegration in the workforce, and for which 37 percent (n=26) considered the workshops useful. [Semi-Annual Reports 6-9, MBA KIIs, Online Survey] Career counseling by STEP is perceived by 93 percent of MBA scholars (n=45) as something that required more time to plan for academic choices, universities, and internships. And, as noted in section 4 findings, no evidence exists to tie counseling to a labor market analysis. [MBA KIIs] Career counseling and job placement received the lowest weighted average ratings for STEP activities at 1.9 and 1.8 respectively (5 highest) (n=27) see also section 1A & 4A. [Online Survey] Sixty-six percent of MBAs (n=25) and 67 percent STEMs (n=21) found the STEP action planning process useful to organize themselves toward career objectives. [Online Survey] Absent clear programmatic actions to establish a sustainable scholar network, 47 percent of MBA scholars (n=45) made reference to an alumni Facebook page as their primary networking tool. [MBA KIIs] the program, with more thought given to linking career counseling to field of study, university selection and relevant job placement. They could have also been more effectively encouraged to support each other more through stronger networking. English proficiency was a constraint even for otherwise qualified applicants, which needs to be factored in at the earliest stages of applicant acceptance. 55 percent of MBAs and 47 percent of STEMs coming originally from Cairo and Alexandria appears overweight and not adhering to regional diversity objectives. Home visits were not done for MBAs and would have likely helped to eliminate some applicants in favor of greater inclusion for disadvantaged scholars. The fact that MBA scholars are university graduates indicate the financial ability of their families over the poverty line depicted 173 | STEP END OF TERM EVALUATION REPORT USAID.GOV No FINDING CONCLUSION RECOMMENDATION As noted in section 1 findings, IIE’s Tracking Study shows the value of networking for scholar development: “...where alumni have collaborated on activities, programs, or in organizations, they have noted a shared sense of purpose and identity that helps them work together successfully.” Desk Review Subsequent phases of the STEM & MBA program selection criteria and application process were modified based on lessons learned in earlier phases. [Sem-An Rep 15] There is evidence of extracurricular activities and civic endeavors by participating scholars. [Sem-An Rep17] No clear diversification of the applicants (geographical) and whether they are from disadvantaged areas. ‘As Egyptian women are underrepresented in study abroad programs… the initiative sought to… reach women from disadvantaged communities.” [SOW, list of cohorts] Summary of MBA and STEM scholars’ academic standing after completion of the first semester of year one shows: 72 percent of MBAs achieved between 3.5-4.0 GPA and six percent (three out of the 53 MBA scholars) are on academic probation with GPAs below 3.0; 71 percent of STEMs achieved between 3.5-4.0 on their GPA and 5 percent (1 out of the 21 STEM scholars) are on academic probation with GPAs below 3.0 [Semi-annual Oct.15 – March16] All scholars who were not enrolled in summer classes (2016) succeeded in securing an internship at a U.S. organization in their city of residence. [Semi-Annual Report Apr-Sep16] Forty-seven percent of STEM scholars are from Greater Cairo and Alexandria.[STEM database] Fifty-five percent of MBA scholars are from Greater Cairo and Alexandria. [MBA database] in the application process. Some selected MBA scholars have a degree in medicine, pharmacy and engineering. (Studying in these colleagues are costly even if public university) The criteria of female leaders as agreed by USAID is not well￾defined and so STEP project ended up screening leadership skills against the number of volunteering community services that applicants conducted in their community together with some easy questions on the application that could be forged. Leadership training was in form of short sessions on some leadership￾related topics during one day in the four￾day pre- departure workshop. Indicators of leadership among STEM scholars are more difficult to screen at this young age based on STEP criteria of community services. USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 174 No FINDING CONCLUSION RECOMMENDATION A financial rubric does exist for selection criteria and EFE designed a scoring rubric for the MBA applications [Sem-An Rep April-Oct.14, Achievement Pres S-6&7] English language tests were administered as a first filter for student applications at the interview stage. [Sem-An Rep April-Oct.14] STEM cohort English proficiency changed for each of three cohorts, scholars were being stopped by English proficiency / STEP began requiring TOEFL scores. [Sem-An Rep15, Achievement Pres S-8] STEM cohorts had a home visit to determine need, whereas this is not listed for MBAs. [Achievement Pres] Rubrics for interviews includes scores for commitment to pursue academic studies and professional career in STEM field/potential for further leadership development. (How this is measured?) MOE rep. were part of the selection activities (how come the security clearance was not issued in time to make announcements inside the schools/placing the ads inside the school?) Basic English Test (cut-of score of 80 (74 for disadvantaged group from underserved gov) (could this be a disqualifying factor for disadvantage students?) 24 assessors evaluating 50 candidates in the group activities camp (no indication how they were assessed for leadership potential, creativity/innovation, adaptability, assertiveness, self-confidence (AFS was in charge of this assessment although they are specialized in cultural awareness). While AFS was involved in the selection process for the STEM, NM and EFE was involved for the selection of the MBA (How leadership and innovations are assessed by the three local partners for the two groups). 175 | STEP END OF TERM EVALUATION REPORT USAID.GOV No FINDING CONCLUSION RECOMMENDATION Group Activity Camp was conducted over three days for a large number of scholars (496 Cohort 1 MBA attended the camp). What is the cost implied and time/effort/resources needed to be in place in making this screening activity early in the selection process? 3A. What is the return on investment of STEP? F3A.1 – The average cost per MBA is $143,598 for the two-year program and $267,096 per STEM for the four-year program, totaling $30,345,360. The average costs are not classified by field of study. [USAID] F3A.2 – Average cost figures are inclusive of indirect and administrative costs. [USAID] F3A.3 – Despite the plethora of research on ROI for higher education, no empirical studies indicate an ROI benchmark for Egypt. An exception is the End-of-Term Evaluation for USAID’s LEAD scholarship program (2016), which estimated the program-specific ROI at a range of 11 percent to 19 percent, depending on field of study. A corresponding ROI calculation was made for higher education in Egyptian private universities estimated at 18 percent to 21 percent. [LEAD End-of-Term Evaluation, 2016] C3A.1 – ROI based on per scholar averages provided by USAID are calculated as 11 percent for MBA scholars and 7 percent for STEM scholars. C3A.2 – Significant non-monetary scholarship benefits occurred: self￾confidence, leadership skills, stronger initiative, managerial skills, technical skills, and cultural awareness (see sections 2 and 4). R3A.1 – For detailed ROI calculations on future scholarship evaluations, we recommend USAID provide cost data classified by field of study and educational institution to allow a more robust analysis. (Design) R3A.2 – For benchmarking of future program results, we recommend USAID sponsors an impact analysis of STEP after the elapse two to three years following its completion. (Design) 3B. What are the reasons for non-returnees? F3B.1 – As of May 2018, six MBA scholars (four from Cohort 1 and 2 from Cohort 2) did not return to Egypt. Six non-returnees of 96 MBA scholars represents a non-return rate of just over 6 percent. [Semi-Annual Reports 7 and 9, Non￾returnee Status Notes] F3B.2 – Four of the non-returnees disclosed the underlying reasons as application for asylum to the USA. [Semi-Annual Reports 7 and 9, Non￾returnee Status Notes] C3B.1 – STEP measures were sufficient to limit the number of non￾returnees to 6 percent, which is not excessive. MBA scholars interviewed believe the non￾returnees had already made their decision before traveling and there was nothing R3B.1 – While the evaluation team has not identified any standard measures or benchmarks to identify applicants who might be susceptible of becoming non￾returnees, further research is warranted. We recommend looking into the possibility of designing USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 176 No FINDING CONCLUSION RECOMMENDATION F3B.3 – STEP, through its implementing partner, enforced mitigation mechanisms to address announced non-returnees through intensive communication. An airport escort process was then enforced. [Semi-Annual Reports 7 and 9, Non-returnee Status Notes] F3B.4 – Fifty-eight percent of MBAs (n=45) and 27 percent of STEMs (n=11) offered opinions why some scholars chose not to return to Egypt, which centered on two factors: limited work opportunities and expected family acceptance problems upon return. None of the respondents believed there was anything STEP could have done to prevent the non-returnees. “When you focus on needy people the culture shock is too much, causing them to break from their communities... this happened in a previous program and we stopped focusing on the needy… scholarships within Egypt may be more effective for such communities... the very needy can contribute to non-returnee numbers.” [MBA KIIs, STEM KIIs, Local Partner KIIs] F3B.5 – A non-returnee confirmed that STEP could not have done anything to prevent her from not returning to Egypt. [Non-returnee Questionnaire] Desk Review As of September 2017, there were four non￾returnees each of which sought asylum (now 6). IIE mitigation included intensive communication with them and an airport escort process. [Sem￾An Rep17, Sem-An Rep18] Non-returnees are MBA students not STEM; also reported challenges are related to MBA not STEM students, as of yet. [Semi-annual reports 5-10] Of the six non-returnees among the MBA scholars, three disclosed the underlying reasons as application for asylum to the USG. [Semi-Annual Reports 9-10] STEP could have done once the scholars were overseas. a screening mechanism using demographic and psychographic markers common to non￾returnees then track the predictive success rate longitudinally. (Design and Implementation) R3B.2 – As one local partner noted, relaxing the disadvantaged/needy selection criteria may be the most likely solution. We recommend making the disadvantaged selection criteria less stringent and award extra points to the needy rather than use such criteria as pass/fail. (Design and Implementation) USAID should reconsider the criteria of just targeting the financially needy as a cutoff score but to be considered among other factors such as leadership skills or community work. 177 | STEP END OF TERM EVALUATION REPORT USAID.GOV No FINDING CONCLUSION RECOMMENDATION Reasons for non-return of the other three scholars were not disclosed. [Semi-Annual Reports 9-10] KIIs STEM students are considering post-graduate studies. KIIs with Cohort 1 student scheduled to return in May expressed her intimidation to go back to Egypt after four years in the U.S. without a post graduate degree. She expressed that many STEM scholars in her cohort are planning for continuing their post-graduate studies in U.S. MBA scholars reported that the ones who got immersed in U.S. culture are the ones who are facing the most challenges fitting in back - especially if they have issues with their families. [MBA KIIs] AFS confirmed that parents have a hard time accepting how their children changed especially upon returning from the U.S.. Also the scholars often come back to reject/resent their own community and culture. They are working to facilitate this reintegration with the returning STEM scholars by a home visit. [AFS KIIs] 4A. Are STEP alumni productively employed in fields related to their academic preparation? Pre-departure support F4A.1 – The university selection process was not consistently well understood by new scholars as 31percent of MBAs (n=45) and 36 percent STEMs (n=11) did not know how they were selected. Scholars reported they often were enrolled in universities not listed in their applications. Two scholars reported not getting the major they requested; one said she was advised to “take it or leave it;” one discovered her university hadn’t required the GMAT due to her work experience; one needed extra courses to finish within the two-year return requirement; yet another had six subjects waived due to previous credits being applied – she returned six months early. [MBA KIIs, STEM KIIs] F4A.2 – Forty-four percent (n=62) of STEM majors are in mechanical/civil/electrical/chemical/ C-4A.1 – Career counseling and job placement assistance was not well planned from the project’s start and less effective than it could have been linking MBAs to employment. Practical relationship building with large employers – not just job search assistance – was not undertaken and we did not see any evidence of consistent attention given to job market demand. No network of linked employers was prepared or R4A1: For future scholarship programs of similar designs, USAID should consider a sustainable holistic cycle approach: At program start, review available labor market assessments to zero-in on specific labor demand as it relates to selected field of study. (Implementation) Begin career counseling weeks before having scholars USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 178 No FINDING CONCLUSION RECOMMENDATION computer/architectural engineering, which are all available majors in Egypt. [Scholar Database] F4A.3 – The evaluation team did not uncover any systematic efforts in project reporting to advise scholars about selecting majors pertaining to specific labor market needs and six MBAs (n=45) plus two STEMs (n=11) reported choosing majors without any support from STEP. Two STEM scholars (n=11) also mentioned limited internship opportunities for aerospace engineering majors in Egypt. STEP raised the unsuitability of the Egyptian job market for certain degree choices (biomedical and aerospace) but were told scholars should not be limited in their selection. [MBA KIIs, STEM KIIs, Semi-Annual Reports 1-9, Egypt Vision 2030] F4A.4 – Universities were not carefully screened for basic logistics such as access to low-cost off￾campus housing and availability of public transportation. Six MBA students (n=45) mentioned these factors were a source of stress for them, especially upon arrival. [MBA KIIs, Semi￾Annual Reports 1-9] US academic/cultural (internships) F4A.5 – Eighteen percent of MBAs (n=45) and 45 percent of STEMs (n=11) mentioned U.S. internships were limited by the need to wait until junior year. Also, few unpaid internships relevant to their field of study were available due to U.S. employment laws. “IIE should have helped us find internships with multinationals, and as it was not paid, I could not use it as job experience.” Initially, there was no mitigation strategy to counteract these obstacles. [MBA KIIs, STEM KIIs, Semi￾Annual Report 8] F4A.6 – There is no evidence of activities to contend with the lack of an internship culture in Egypt. Twenty percent of MBAs (n=45) expressed outright dissatisfaction with the internships. One MBA described reading a book as she was given nothing to do; another mentioned performing secretarial duties such as making photocopies; another stated: “EFE found me an internship with the Saudi-German Hospital, which was a waste of informed of project objectives or offered profiles of returning MBAs as potential new hires. C4A.2 – Internships did not fully satisfy their original purpose: practical work experience and networking for potential employment. Scholars listed as a main challenge difficulty finding suitable unpaid internships and in Egypt, the internships were not consistently relevant to fields of study. C4A-3- MBA scholars faced several challenges finding suitable employment back in Egypt. Being out of the job market for 2-1/2 years, returning scholars needed more opportunities to increase work experience in their field of study as internships were often brief or not directly relevant. They also needed more coaching from the earliest days aimed at managing expectations on their return. However, it appears scholars did return with strong soft skills, which employer select field of study and university, factoring in work experience, career shift interests, and labor market demand. (Implementation) Starting from day one – have a program component focused on demand-side linkage building to strengthen internship opportunities and practical employment linkages well before the scholars return. (Design) 179 | STEP END OF TERM EVALUATION REPORT USAID.GOV No FINDING CONCLUSION RECOMMENDATION my time. The culture of internships in Egypt is not good, people are not supportive, and I didn’t write this junior experience in my CV.” [MBA KIIs, STEM KIIs] F4A.7 – In the scholars’ survey, 33.3 percent of MBAs indicated satisfied or very satisfied (n=27) regarding the usefulness of internships to their professional development. While 27.3 percent of STEM scholars (n=22) stated they were satisfied or very satisfied in terms of internships’ usefulness. [Online Survey] Reintegration F4A.8 – The STEP SoW identifies a project objective to assist scholars to seek employment. For STEMs: The contractor will assist scholarship recipients in identifying internships for the summer periods with organizations and companies that will enable them to obtain valuable hands-on experience. Upon completion of their scholarship, the contractor will assist the graduates in finding appropriate employment back in Egypt. For MBAs: During the last semester of the scholarship recipients’ program, the contractor will assist the student in identifying employment opportunities in Egypt. The contractor will continue to work with the student to identify appropriate employment opportunities after she returns to Egypt. [SoW] F4A.9 – Eighty percent of MBA scholars (n=45) came back with a strong sense of self-confidence. Seventy-three percent of STEM scholars interviewed (n=11) also expressed developing self￾confidence living along in a foreign country. [MBA KIIs, STEM KIIs] F4A.10 – Online networks offered by IIE and USAID are not actively used. Sixty-four percent of MBA scholars (n=45) did not report being a member of the IIE’s alumni network or their partners’ networks. Two MBA scholars reported accessing the IIE network, three others mentioned being USAID alumnus (n=45). However, the Facebook alumni network is actively used by 21 MBA scholars. [MBA KIIs] respondents spoke highly of. C4A.4 – Without a continuing relationship between U.S. universities and STEP alumni and/or universities back in Egypt, which extends beyond the departure of graduates, no continuing education benefits will be possible, e.g., webinars, conference announcements. USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 180 No FINDING CONCLUSION RECOMMENDATION F4A.11 – Sixty percent of MBA scholars interviewed (n=45) indicated they secured jobs by themselves, often helping each other through their informal networks and connections. One recommended an alumni at the Ministry of Health office; one assisted four alumni to work in a tourism company; another helped a colleague apply to a contractor for the European Union; another working at the UN passed a consultancy position to a scholar. Two MBA graduates are currently working in incubators through their own networks. (They are beneficiaries of another USAID programs.) [MBA KIIs] F4A.12 – Thirty-one percent of MBA scholars (n=45) reported that without practical experience an MBA alone was insufficient to find employment in Egypt, and many accepted entry level positions. The job placement local partner concurred indicating the lack of experience and being out of the job market for three years works against the scholars. Four MBA scholars stated employers do not take the MBA into consideration: … there are many with master’s degrees now in Egypt, what makes you special? …go back to what you were doing or accept an entry level job... work experience is the most valued by employers in Egypt… if another MBA holder has the right work experience, s/he can get a higher position with a higher salary.” [MBA KIIs, Local Partner KIIs, Semi-Annual Report 8] F4A.13- Seventy-one percent of MBA scholars (n=45) emphasized the need for an introduction/linkage to relevant employers as a high priority to them – to increase their employability. “Most of us changed careers and our networks were no longer useful... IIE should have promoted us to secure jobs with the private sector even before returning, they had two years to promote us, it could have worked... no one looked in depth into the employment part, if you fix that part, the program would have been very successful… the idea of applying online for jobs isn’t effective, you have to know someone in the company.” [MBA KIIs] 181 | STEP END OF TERM EVALUATION REPORT USAID.GOV No FINDING CONCLUSION RECOMMENDATION F4A.14 – MBAs and STEMs were asked to rank satisfied and highly satisfied pertaining to job placement services and internships, vis-à-vis finding employment: [Online Survey – apply to all below] Twenty-five percent of MBAs (n=24) indicated they were satisfied or very satisfied: STEP Internships helped me find work. Twenty-one percent of MBAs (n=24) indicated satisfied or very satisfied: The job placement and career counseling activities conducted by STEP helped me find work. Thirty-five percent of STEMs (n=25) indicated satisfied or very satisfied: I am confident I will find work as my specialization is in demand in the Egyptian labor market. Twenty-eight percent of STEMs (n=18) indicated satisfied or very satisfied: I am confident, the job placement and career counseling activities conducted by STEP will help me find work. This is further supported by KIIs: 29 percent of MBAs (n=45) viewed STEP career counseling and placement services such as career fairs, lists of online job postings, and attempts to set up interviews as adding little value, and which could have largely been done themselves. [MBA KIIs] F4A.15 – MBAs, in particular, do not express satisfaction with internships and job placement support. [Online Survey] F4A.16 – Of six employers interviewed, four expressed the need for soft skills and positive attitudes in the workplace in addition to specific sector knowledge. One employer indicated the position for which the STEP scholar was hired didn’t require an MBA. It did require experience, yet they waived this due to her personality and communication skills. Another employer hired a scholar for her high adaptability and people skills, as well as her international perspective due to living abroad. While the sample was very small it offers contextual insights about the importance of the soft skills gained. [Employer KIIs] USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 182 No FINDING CONCLUSION RECOMMENDATION F4A.17 – At least fifty-four percent of MBA scholars are believed now to be employed (N=90). Seventy-five percent of MBAs answering an employment question online (n=16) indicated an average time between graduation and employment of six months, and 75 percent of them indicated the position was relevant to their field of study. [Online Survey, STEP and MBA Scholar Database] F4A.18 – MBA and STEM scholars were asked about challenges to find work. Fifty-seven percent of MBAs (n=28) stated the organizations’ work environment as a main challenge followed by 54 percent stating salaries. Seventy-one percent of STEMs (n=21) felt the lack of jobs to suit their qualification will be a challenge followed by 62 percent indicating adequate salaries as a main challenge. [Online Survey] F4A. 19 – One hundred percent of employed MBA scholars (n=13) answering a question about why job offers were refused indicated inadequate salaries and benefits while 77 percent believed the position didn’t match their qualifications/field of study. [Online Survey] F4A-20 It was reported USAID allocated $5 million USD to facilitate the reintegration and employment of STEM scholars. This was confirmed by a local partner who is signing a SOW with IIE to work with returning STEM Cohort 1. Also, two STEM scholars indicated IIE contacted them to clarify their needs after graduation. [CoP Discussion, Local Partner KII, STEM KIIs] F4A.21 – There is no evidence of attempts to continue a relationship between participating U.S. universities and STEP alumni (or Egyptian universities) once the scholars returned and therefore, no means to tap into available continuing education opportunities. [Semi-Annual Reports 5-9] F4A-22 Case study of MBA scholar from Upper Egypt describing STEP’s placement effort. [MBA 183 | STEP END OF TERM EVALUATION REPORT USAID.GOV No FINDING CONCLUSION RECOMMENDATION database, MBA KIIs, Employer KII, EFE Job Placement Challenge Report] Profile: Age 29, from Qena, studied management and leadership from Webster University. Graduated with honors (3.84 GPA). Scholar sought work in the development field. EFE provided her with three opportunities in her field. She was rejected from one, another post was filled and the third she did not attend interview. She was asked to sign a rejection letter and was subsequently dismissed from further STEP support for not doing so. STEP policy: If EFE and IIE agree the offered opportunity matches the candidate’s working experience and field of interest, while the candidate declined the opportunity or interview invitation, a signed rejection letter will be required.” Currently the scholar works in a managerial position in an NGO position she sought on her own. Her manager expects she will be promoted soon. F4A-23 Case study of MBA scholar from Lower Egypt describing STEP’s placement effort. [MBA database, MBA KIIs, Employer KII, EFE Job Placement Challenge Report] Profile: Age 31, from Gharbeya, studied entrepreneurship from Rochester University, GPA 3.4. Scholar sought work in NGOs with a preference toward entrepreneurship. As she has five years of experience in pharmacology, three organizations rejected her because she lacked practical experience. She received an entry level offer from a small NGO, which she rejected. She was asked but refused to sign a rejection letter and was dismissed from any further STEP support. Note: the same position was offered to two other MBA scholars, who also declined it. Currently, the scholar is working at an international NGO that has an entrepreneurial aspect. Her supervisor predicts she will reach a leadership position in the institution. Desk Review USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 184 No FINDING CONCLUSION RECOMMENDATION No STEM students are expected to graduate in 2019 and 2020. [Semi-annual report 8&9] For job placement, STEP Contacted 23 employers for 39 job vacancies in different fields. [cite] Three recruiters in Egypt: Wuzzuf linked scholars with 114 employers (138 vacancies); Jobzella with 13 employers (13 vacancies); AmCham with 24 employers (24 vacancies) [cite] MBA students are said to have unrealistic expectations with regards to job status, often refused job offers not meeting their expectations, rejected jobs offers outside their academic fields. [Semi-Annual Report, March18] Employers demanding work experience in addition to academic qualifications. Only 27 were employed. [Semi-Annual Report, March18] There is no evidence of activities to develop private, public or nongovernmental-sector willingness to understand the nature of the scholar program with an aim to hiring returning graduates. [cite] KIIs The most common response to what activity improved the MBAs’ employability was improved confidence followed by better English skills. [MBA KIIs] The most common response to what activities were missing that would have improved their employability were: internships more directly relevant to their field of study, more practical job￾seeking assistance on their return (not basic CV preparation and career fair notices aimed at the “fresh graduate” level), more real contacts in multinational organizations. [MBA KIIs] The STEP experience was unanimously viewed as valuable by MBAs, particularly as it gave scholars their first opportunity to truly rely on their own initiative. It also gave them a strong credential, the MBA itself. However, these things by themselves did not guarantee significantly greater 185 | STEP END OF TERM EVALUATION REPORT USAID.GOV No FINDING CONCLUSION RECOMMENDATION employability; the Egyptian market requires practical work experience. [MBA KIIs] When MBAs returned, after two years away, Egypt had changed significantly, and they had no active personal network to turn to assist in job seeking. [MBA KIIs] Online/P&P/Tel KII Questionnaires The MBA is not a major factor, but the work experience together with the MBA. If another MBA holder has the right work experience, s/he can get a higher position with a higher salary. (MBA Cohort 1) Interview is an important chance to reveal yourself and your knowledge from the MBA , we could be rejected based on the application only. Literature Review: As per two recent studies linking study abroad to employment -conducted by IIE: students who sought to develop more work-related skills through study abroad reported more positive employment outcomes. i.e establishing intentional goals prior to going abroad and reflection on skills developed through study abroad has a greater influence on employability after graduation. Another suggested there are several best practices that colleges and universities can implement to help support students to link their study abroad and employability: -Cross-training of study abroad and academic advisors designed to have students consider if their choice of program abroad will help enhance their chosen career. -Sessions upon returning to campus that focus on how students might describe study abroad on their résumé. -Student handbooks that offer examples on how to present your international experience on résumés, cover letters, and interviews. USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 186 No FINDING CONCLUSION RECOMMENDATION Source: Study Abroad Matters: Linking Higher Education to the Contemporary Workforce through International Experience, 2018, (IIE publication) Gaining an Employment Edge: The Impact of Study Abroad on 21st Century Skills & Career Prospects, 2017 (IIE publication) 4B. To what extent are STEP alumni positioned to take positions of leadership in academia, government, industry, and business? F4B.1 – There is no evidence of a consistent leadership definition or a framework for its measurement throughout the project. However, there are reported success stories and achievements for MBA and STEM scholars during their stay in the U.S., such as club participation, volunteer activities, awards, and publishing academic papers. [MBA and STEM Scholar Success Stories] F4B.2 – Six MBA scholars received a grant from the US Embassy for “The Dream Project” targeted to improve the lives of 100 orphans and youth as well as social workers and educators. In October 2017, the alumni started implementing their project working with a local NGO. [Semi Annual Report 8] F4B.3 – Eighty-five percent of MBA scholars (n=26) rated satisfied or highly satisfied with the program’s contribution to their personal development and 69 percent to their professional development. While 64 percent of STEMs (n=21) expressed they were satisfied or highly satisfied with the program’s contribution to their personal development and 82 percent to their professional development. [Online Survey] Pre-departure support F4B.4 – At least two workshops were held on self￾leaderships skills in 2016, two days each, one for 34 STEM Cohort 2 and one for 46 MBA Cohort 2 scholars [Semi-Annual Report 4] C4B.1 – Scholars’ leadership skills and soft skills were further developed due to their U.S. academic studies; these are the very factors which offer them an edge over those who have not undergone such an experience. C4B.2 – Insufficient time has elapsed to answer whether leadership is actually being demonstrated in practice during employment or regular community service in Egypt, though several success stories are noted. C4B.3 Scholars were very positive about STEP’s contribution to their personal and professional development. R4B.1 – We recommend critical terms such as “leadership” be clearly defined and shared with each scholar at the application phase then tracked against clear competency indicators throughout their studies and for at least six months after their return. This will also help refine future selection rubrics. (Implementation) 187 | STEP END OF TERM EVALUATION REPORT USAID.GOV No FINDING CONCLUSION RECOMMENDATION F4B.5 – The four-day agenda of the pre-departure orientation lists the following sessions: Leadership and Culture, Emotional Intelligence and Leadership, Style of Management, Team leadership and Eight Characteristics of Team Excellence, Leadership Models, Challenges with Intercultural Teams, Psychological Wellbeing and its Relation to Building Social Capital. Each session lasted between 30-45 minutes. [Pre-departure Orientation Agenda for STEM Cohort 2, July 2016] F4B.6 – In the online questionnaire, scholars reflected on the leadership activities provided by STEP as follows: Thirty-six percent of MBAs (n=25) rated the leadership activities as useful or very useful for their personal development and 35 percent of MBAs (n=26) rated the leadership activities as useful or very useful for their professional development. Thirty-six percent of STEMs (n=22) also rated the leadership activities as useful or very useful for their personal development and 36 percent of STEMs (n=22) ranked the leadership activities as useful or very useful for their professional development. [Online Survey] U.S. academic/cultural F4B.7 – Ninety-six percent of MBAs (n=45) offered definitions of leadership qualities and believe STEP helped further develop them. Most common definitions were: having vision, inspiring others, acting as role models, being an integral part of/supporting a team, understanding people's strengths and weaknesses. Also, 96 percent of MBAs (n=45) and 91 percent of STEMs (n=11) reported gaining self-confidence, becoming more tolerant, and/or feeling more empowered, all strengthened by dealing with people from different cultures while living abroad. [MBA KIIs, STEM KIIs] Reintegration F4B.8 – Three employers (n=6) predict the MBAs are headed into leadership positions in their respective organizations due to their soft skills, USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 188 No FINDING CONCLUSION RECOMMENDATION communications, and personality. [Employers KIIs] F4B.9 – One MBA scholar was promoted to manager within eight months of joining her organization and given a 30-percent salary increase, another received a 20-percent salary increase within a year of employment, while another mentioned she will be promoted soon. [MBA KIIs] F4B.10 – Six of 11 STEM scholars are planning to continue their post graduate studies: Four stated they were already accepted into U.S. programs, while two others plan to study in Europe. [STEM KIIs] F4B.11 – Twenty-three percent of MBA scholars (n=26) agree or strongly agree leadership training helped with employment; 15 percent STEM scholars (n=18) also agree or strongly agree. [Online Survey] Desk Review STEP conducted activities with the scholars to affect professional and personal development: leadership and cultural activities, career counseling and job placement. Students also participated in campus life activities and conducted internships. [Semi-annual reports 5-10] STEP has the following indicator regarding change in participants’ leadership skills: 75 percent of the participants who receive leadership training will demonstrate improvement in their leadership skills measured by comparing baseline assessment prior and after training. [PMP Narr14] Due to restrictions stipulated by the U.S. labor law, the IIE was unauthorized to directly place scholars in internships and scholars must apply to internships on their own. Internship opportunities secured by MBA scholars after February 2016 are as a result of their own self-application following IIE guidelines. [Semi-annual-March16] Two workshops held on Self Leaderships Skills, two days each, one for 34 HEI STEM Cohort 2 189 | STEP END OF TERM EVALUATION REPORT USAID.GOV No FINDING CONCLUSION RECOMMENDATION and one for 46 HEI MBA Cohort 2 scholars. [Semi-annual-March16] Implementing partner NM secured 21 internships for HEI STEM Cohort 1 scholars between May and August 2016. Scholars interned at organizations relevant to their field of study for a period of four weeks. [Semi-annual-March16] In July/August/October 2018, 29 HEI MBA Cohort 2 scholars completed their final Egypt-based internships. Five participants declined participation in summer internships due to personal reasons. [Semi-Annual Report April-Sept.18] No finding on an intentional leadership definition that IIE is following, however, it is reflected in the volunteer work that student are performing at the university. [Semi-Annual Reports] 5A. To what extent does the technical assistance component supplement the activities and projects of other technical offices? F5.1 – The STEP scope of work lists an objective to improve the capacity of key government, academic and private-sector organizations through the provision of scholarships, training, study tours, and technical assistance. These are enumerated in F5.2 – F5.11 below. [SoW] F5.2 – STEP sub-objective: Technical assistance support to government master’s degrees (including facilitating master’s degrees for gov’t and health) – zero completed. [SoW, Achievements Presentation, FY18 Results PMP, Semi-Annual Reports 1-9] F5.3 – STEP sub-objective: Technical assistance support to government and civil society (including training for local gov’t, elections officials, judiciary, members of parliament, local councils, civil society) – zero completed for local government, judiciary, members of parliament, or local councils; one CSO assessment on combatting violence against women. [SoW, Achievements Presentation, FY18 Results PMP, Semi-Annual Reports 2-9] C5.1 – None of the following planned technical assistance component objectives were completed during the course of the STEP activity: Government master’s degrees; Capacity building for local gov’t, elections officials, judiciary, members of parliament, or local councils (one assessment was completed for a civil society organization); Private-sector capacity building for advocacy groups, business associations, chambers R5.1 – We recommend future scholarship programs not be saddled with multiple, overlapping, high-level objectives, which require the addition of a technical assistance component and a distinct set of objectives largely unrelated to those of the scholarship component itself. Instead, align the scholarship objectives with attainable and measurable outcomes resulting from what the scholarships are designed to accomplish: enhancing scholar technical skills, strengthening leadership qualities, USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 190 No FINDING CONCLUSION RECOMMENDATION F5.4 – STEP sub-objective: Technical assistance support to government education (including training for ministries and authorities training with MOE, MOHESR, tertiary institutions and NGOs) – multi-year capacity building interventions including 22 ministry officials and 271 regional supervisors trained, 17,000 CDs with training materials sent to regions, 1800 trainers manuals developed, 72,807 teacher guides, 2,312 training kits plus 15 MOE officials sent to the U.S. for participant training; [SoW, Achievements Presentation, FY18 Results PMP, Local Partner KIIs] F5.5 – MOE capacity building included full cycle support: work plan, performance management plan, assessment/gap analysis, curricula integration, teaching methodology, teacher training, student assessments, program monitoring, and teacher certification. All training was performed in a participatory manner with beneficiaries learning alongside trainers in a ToT setting. [Local Partner KIIs, Semi-Annual Reports 3-8] F5.6 – The MOE is very satisfied with the EGL capacity building and training, which supported Egypt’s priority vision (Education 2.0). They believe they are well positioned to sustainably administer and monitor the newly adopted curricula and ensure teachers are trained in their use. A ministry committee has been formed to continue a cascaded EGL training methodology. [MOE KII, Local Partner KII] F5.7 – STEP sub-objective: technical assistance support to government health (including training for ministry staff, resident physicians at university and government hospitals, university students, nurses, and other hospital and health center staff) – MOHP was supported with 127 participants in hepatitis workshops and 34 MOHP officials were sent abroad for health training/workshops, including nurses and physicians, for infection prevention and control. Both types of support were perceived as technically strong by the ministry. The activity addresses an MOHP priority, the National Hepatitis C Campaign. [Egypt Vision 2030, MOHP KIIs, USAID KIIs] of commerce, and economic growth policymakers. Moreover, just two interventions were found to have been completed for higher academic institutions during STEP’s five-year term. The lack of activity in these areas is perceived as resulting from post￾revolution USAID technical office budget and program changes. It also appears to be the result of an inadequate understanding of STEP TA component objectives and resources – particularly among OEG and ODG – despite the delivery of several information sessions about the availability of STEP TA. C5.2 – The lack of completion of multiple STEP objectives, listed in conclusion C5.1 (above), also raises a project design issue. Each of the sub-bullets arguably comprise stand-alone projects in their own right. Such multiple and overlapping objectives – especially for a project primarily tasked with educating scholars – is counterproductive and and ensuring strong linkages to employment opportunities, all of which will contribute to development objectives in the longer term. In this case, we recommend any technical assistance component be strictly aligned with the objectives of the scholarship program, such as to strengthen employment linkages as further recommended below in R5.3. (Design) R5.2 – In cases where technical assistance is used to send people for participant training or foreign conferences, it is crucial to have clear strategies and mechanisms leading to the adoption of the knowledge gained upon their return, and which have been agreed with the beneficiaries prior to their travel. In fact, we recommend participant selection be based on a well￾articulated mini￾proposal to share the knowledge within their institution/organization upon their return, precisely how this will be accomplished, how it will be sustained, and why institutional 191 | STEP END OF TERM EVALUATION REPORT USAID.GOV No FINDING CONCLUSION RECOMMENDATION F5.8 – The participant training group for infection control found the subject matter very useful, though some aspects would have benefitted from being “less theoretical and more practical.” The group faced difficulties in their ability to effect policy changes on their return. This is believed due to inadequate forethought to the eventual dispersion of knowledge gained and, though selection of operational level participants was seen as a strength (including nurses), the lack of one or two senior level participants meant impactful changes in the near term were unlikely. There is ongoing work on a new health and insurance law, which will achieve sustainable impact if it moves forward. [MOHP KIIs, USAID KIIs] F5.9 – The returning hepatitis conference participants also faced challenges to achieve change within the ministry and hospitals due to a persistent mindset that prioritizes treatment over prevention. Pervasive turnover among ministry staff was also cited as a challenge – three of five ministry participants left their jobs. Moreover, while participants were encouraged to disseminate information on their return – to achieve institutional change – specific mechanisms to do so were not clearly articulated. One suggestion to help enhance the ongoing development of the training group involved a mechanism for sustained communication between the host institution and the Egyptian training group. [USAID KIIs, MOHP KIIs] F5.10 – MOHP indicates it requires a “longer learning cycle which doesn’t end with the completion of the study.” [MOHP KIIs] F5.11 – STEP sub-objective: Unspecified technical assistance support to academic institutions – One assessment of university Career Development Centers completed; One regional adult and youth literacy assessment for MOE-Adult Education Authority completed though unpublished. [SoW, Semi-Annual Reports 3 and 4, Local Partner KIIs, Achievements Presentation, FY18 Results PMP] F5.12 – STEP sub-objective: Technical assistance support to private sector (including capacity obscures programmatic focus. C5.3 – Technical assistance with the MOE was focused on two capacity-building tracks: enabling the MOE to effectively manage the adoption and monitoring of new K-3 Early Grade Learning curricula, and; providing ToT for the teacher training itself. This was a highly leveraged capacity building program with strong chances for sustainability, and well appreciated by the MOE. Key success factors involved STEP’s start to finish integrated approach, which addressed the complete cycle for sustainability from the use of a participatory training methodology to managing and tracking the new curricula in use. C5.4 – Technical assistance was delivered to the MOHP via support to 161 participant training and foreign health conference participants, mainly to improve infection prevention and control and inhibit the spread of Hepatitis. MOHP and beneficiaries greatly valued the change will occur. This should be a precursor for participation. (Implementation) R5.3 – We recommend future scholarship programs be imbued with strong employment objectives and require intensive private sector engagement. Begin developing relationships with large employers from the earliest days of the activity. This should start with an awareness campaign directed to large, progressive employers in sectors that align with the anticipated and actual fields of study undertaken by the scholars. The awareness campaign should be followed by individual meetings between project staff and targeted employers (and those expressing an interest in the program following the awareness campaign) to accomplish several goals: 1) Explain the objectives of the scholarship program and why they are vital to [Egypt’s] development, while appealing to their need for effective CSR programs to give something back; 2) USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 192 No FINDING CONCLUSION RECOMMENDATION building in economic growth issues for policymakers, technical experts in ministries/authorities, advocacy groups, private￾sector business associations, councils, and chambers of commerce) – zero completed. [SoW, Achievements Presentation, FY18 Results PMP, Semi-Annual Reports 1-9] F5.13 – The USAID Office of Education and Health was the only technical office with a clear understanding of the nature and potential use of STEP TA as a resource. Every technical office developed its own plan and did not necessarily factor in STEP TA. The Office of Education naturally accessed available STEP TA, which is believed to be the reason the related Office of Health also participated. STEP made three or four presentations concerning the nature and use of its TA component to all USAID/Egypt technical offices to inform them of STEP TA availability. [USAID KIIs, CoP Discussion] F5.14 – Following the political upheaval of 2011, USAID/Egypt underwent budgetary and internal planning changes, which scaled back and created uncertainty about technical activities and programs. [USAID KIIs] F5.15 – The Scholarships and Technical Assistance components were two projects “stuck together,” neither one designed to support the other. Initially it was envisioned that all USAID technical offices would approach STEP with specific needs. STEP therefore signed subcontractors at the proposal phase with the capacity to address anticipated needs. However, requests for TA were only received from the Education and Health Offices (and one small intervention with Democracy and Governance). [USAID KIIs, CoP Discussion, Semi￾Annual Reports 1-9] F5.16 – Three of six STEP locally subcontracted partners provided services in support of the scholarships component, including outreach, regional information sessions, initial training for selected scholars, and/or developing selection rubrics. One local partner provided career counseling and outplacement services directly to technical content of these events; however, measurable post-event knowledge transfer and policy level outcomes remain unclear. For the infection control training, a contributing factor was the lack of senior level participants to ensure the knowledge gained was given serious consideration by policymakers and adopted on their return. Whereas the hepatitis participants would have benefitted from more forethought and planning to map out and then follow through with specific mechanisms to effect institutional change, monitored and tracked by STEP. C5.5 – Three of six local partners implemented STEP’s technical assistance component as a stand￾alone activity with no relation to, or integration with, the scholarships component. There is no evidence of cross￾fertilization between local partners, which may have otherwise enhanced the planning and delivery of scholarship support initiatives. The SOW Impress upon the employers the rigorous selection process and why the scholars are among the most intellectually gifted in the country; and, 3) Differentiate the value of these students versus their domestic counterparts, e.g., their international perspective, strong ability to take initiative, increased self-confidence, enhanced leadership qualities, and heightened cross￾cultural sensitivity. The final stage will involve matching individual scholar profiles, at least six months before their return, with potential employers then facilitating introductions. We recommend these impactful activities be designed into future scholarship programs by USAID and implemented by IPs as practical means to more precisely tie scholarship initiatives to economic growth. (Design) CSR 193 | STEP END OF TERM EVALUATION REPORT USAID.GOV No FINDING CONCLUSION RECOMMENDATION scholars and alumni. Among the other three local partners, two of which completed two or fewer assignments during the entire STEP program, STEP did not attempt to advocate for their use in support of scholar employment objectives. [Semi￾Annual Report 2, CoP Discussion, Local Partner KIIs] F5.17 – The STEP local partner tasked with job placement for returning MBAs acknowledged an intention to link alumni with their network of employers: “EFE/Egypt will draw upon its network of private-sector employers and link graduates to employment opportunities.” However, there is no indication direct linkages to any network of employers actually occurred. Instead, job placement services were undertaken through three job search firms identifying companies who had posted positions online (see also section 4A). [MBA KIIs, Last Weekly Report Candidate Status 2018, Semi-Annual Reports 6-9, MBA KIIs] F5.18 – Initially STEP was designed with a 100- percent alumni employment mandate, which was later reduced to placement assistance. [CoP Discussion] F5.19 – Other than an initial kickoff meeting, there is no indication of actions by STEP to bring its local partners together for team building discussions or inform them of each other’s activities to cross-fertilize or benefit from their collective knowledge. Local partners were not informed of project events or outcomes outside their immediate scope and not offered any clear opportunities to interact with one another. [Semi￾Annual Reports 1-9, Local Partner KIIs] Desk Review STEP began coordination meetings during Q1 of 2015 with Health, Education, and Democracy and Governance USAID tech offices for TA support, one activity underway with USAID Education same quarter [Sem-An Rep15] March 2015 responded to USAID Education Ofc for needs assessment on eight ministries, CDC envisioned STEP interventions with the private sector, which were never undertaken but could have been directed to develop relationships with large, progressive companies in Egypt as prospective employers for the returning scholars. This represents a significant missed opportunity for STEP and its beneficiaries. A similar process is often undertaken in private sector value chain programs, which coalesce large buyers and small subcontractors by working both sides of the equation simultaneously to sensitize the needs of one to the other. STEP did not conceive of a methodological connection between their technical assistance component resources and strengthening the chances of relevant employment for returning scholars, nor did they advocate for it. Potentials are there to benefit from MBA scholars to strengthen the business administration capacity of government organization. Full circle project integration diagram USAID.GOV END-OF-PROJECT PERFORMANCE EVALUATION OF STEP | 194 No FINDING CONCLUSION RECOMMENDATION asked to submit budget but never received response. [Sem-An Rep15, Partner KII] No clear strategic structure for acceptance of technical assistance activities for education and health sectors, which are unrelated to STEP scholar component. [Semi-annual reports 5-10] STEP continued to implement nationwide training for Ministry of Education (MOE) professionals in Early Grade Learning (EGL), focusing on Early Grade Reading (EGR) and mathematics (EGM) in support of USAID/Egypt’s Development Objectives [Semi-Annual Reports 5-10] “Technical assistance may also be used independently of training to supplement a participant’s work plan or to fill a void where no Mission technical assistance provider exists” [SOW] However, deliberate efforts need to be made to place MBA scholars in public and private organizations. Additionally, reviewing salary scale and benefits are needed for maintaining those scholars in these organizations without getting head hunted by international organizations inside and outside Egypt. 5B. To what extent does STEP through the TA component contribute to the achievement of USAID/Egypt's strategic DOs? What are the critical success factors? TA component for MoE achieved a sustainability model due to the new implementation methodology, in order to maintain that model. However, the long- term planning of the government strategy might not be sustained through that model. TA component for MoH study tour was partially implemented in MoH activities, however, there is no sustainable methodology in place to keep that knowledge. 195 | STEP END OF TERM EVALUATION REPORT USAID.GOV No FINDING CONCLUSION RECOMMENDATION 5C. To what extent does STEP through the TA component contribute to the priority needs of Egypt’s key government, academic, and private-sector organizations? What are the critical success factors? KIIs MoE is in need of a longer-term strategic partner that requires support for more than two years. The year-to-year planning is not very efficient to plan the government’s needs. [TA KIIs] The MoH is in need of a longer learning cycle that doesn’t end upon the end of the study. However, the MoH implemented changes in their day-to-day work. (data analysis, community mobilization) [TA KIIs] USAID.GOV U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523