September 2016 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by the LOTUS Evaluation Team and The QED Group, LLC. Authors: Virginia Lambert, PhD, Ola Hosny, and Doaa Abdelaal Midterm Performance Evaluation of the Leadership Opportunity Transforming University Student Scholarship Program in Egypt Midterm Performance Evaluation of the Leadership Opportunity Transforming University Student Scholarship Program in Egypt MIDTERM PERFORMANCE EVALUATION OF THE LEADERSHIP OPPORTUNITY TRANSFORMING UNIVERSITY STUDENT SCHOLARSHIP PROGRAM IN EGYPT September 23, 2016 Contract No: AID-263-I-15-00001; Order No: AID-263-TO-16-00003 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ACKNOWLEDGEMENTS Services to Improve Performance Management, Enhance Learning and Evaluation (SIMPLE) and the midterm performance evaluation team are very grateful to the implementing partner, the Institute of International Education (IIE). The Leadership Opportunity Transforming University Student (LOTUS) Scholarship Program staff was not only open and fully engaged in meeting and responding to questions asked by the team, they also went out of their way to provide data, project related documentation, and field support when necessary. Given the constrained time frame, this evaluation would not have been as successfully concluded without IIE’s willingness to schedule meetings with scholarship beneficiaries and key stakeholders as well as to provide needed logistical support for conducting field operations. SIMPLE and the evaluation team would also like to acknowledge the assistance of many people at USAID/Egypt: Mary Ishak (Project Management Specialist/LOTUS Agreement Officer’s Representative), for providing guidance and input at all stages of this evaluation; and Arturo Acosta, Ed.D (Deputy Director and Team Leader of Higher Education), Marie El Soussy (Project Management Specialist), Hanan Abbas (Monitoring and Evaluation Specialist from the Program Office), and Albert Carrera and Shaymaa Shaatoot (Contracts Officers from the Procurement Office), for helping to expedite Mission processes wherever possible and facilitating interactions with the implementing partner. Their invaluable support allowed SIMPLE to interview students right before the end of the academic year. SIMPLE and the evaluation team are grateful to North South Consultants Exchange (NSCE), the local subcontractor that hosted the team throughout the evaluation, for providing highly professional and timely logistical and field support. The team further wishes to thank the scholarship recipients themselves who engaged the team in rich conversations and interactions. It was a privilege and pleasure to meet these exceptional young men and women and to benefit from their insights about the LOTUS Program. We wish them continued success as they advance their careers and contribute to Egypt’s development. Finally, SIMPLE wishes to express its deep appreciation to the evaluation team leader (Virginia Lambert, Ph.D.), senior evaluators (Ms. Ola Hosny, Ms. Doaa Abdelaal, and Ms. Youmna Khalil), and senior statistician (May Gadala, Ph.D.) for their professionalism, commitment, and dedicated hard work in meeting deadlines and completing the numerous tasks for which they were responsible. Their diverse technical backgrounds enhanced the design of the evaluation tools, which allowed for deeper levels of analysis. CONTENTS Acronyms........................................................................................................................................................ i Executive Summary.....................................................................................................................................iii Introduction................................................................................................................................................... 1 Background .................................................................................................................................................... 1 Methodology and Limitations .................................................................................................................... 3 Findings, Conclusions, and Recommendations...................................................................................... 5 Challenges................................................................................................................................................... 27 References................................................................................................................................................... 28 Annexes ....................................................................................................................................................... 29 Annex I: Evaluation Statement of Work .............................................................................................. 30 Annex II: Evaluation Methods and Limitations.................................................................................... 43 Annex III: Data Collection Instruments ................................................................................................ 51 Annex IV: Sources of Information and Lists of Participants........................................................... 102 Annex V: Disclosure of any Conflicts of Interest............................................................................. 114 Annex VI: Descriptive Statistics and Statistical Analyses................................................................115 i ACRONYMS AASTMT Arab Academy for Science Technology and Maritime Transport ACU Ahram Canadian University AFRS-Egypt Egyptian Society for Intercultural Exchange Am Cham American Chamber of Commerce AUC American University in Cairo BUE British University in Egypt CAPMAS Central Agency for Public Mobilization and Statistics CDCS Country Development Cooperation Strategy CEDO Career and Entrepreneurship Development Office COR Contracting Officer’s Representative CV Curriculum Vitae DDL Development Data Library DEC Development Experience Clearinghouse DO Development Objective E-ERA Egyptian Association for Educational Resources EMC Evaluation Management Committee FUE Future University in Egypt FY Fiscal Year GI Group Interview GOE Government of Egypt GPA Grade Point Average IDIQ Indefinite Delivery/Indefinite Quantity IIE Institute of International Education IP Implementing Partner IR Intermediate Result ITP International Testing Program KII Key Informant Interview LEAD Leadership for Education and Development Scholarship Program LIA Leadership in Action LOE Level of Effort LOTUS Leadership Opportunity Transforming University Students Scholarship Program LSP Local Scholarship Program M&E Monitoring and Evaluation MENA Middle East and North Africa MOHE Ministry of Higher Education MSA Modern Sciences and Art University NGO Non-Governmental Organization NM Nahdet el Mahrousa ii OEH Office of Education and Health PUA Pharos University of Alexandria PMP Performance Monitoring Plan QED The QED Group, LLC RFA Request for Applications RFTOP Request for Task Order Proposal SIMPLE Services to Improve Performance Management, Enhance Learning and Evaluation SO Strategic Objective SOW Statement of Work STEM Science, Technology, Engineering and Mathematics TOEFL Test of English as a Foreign Language USAID U.S. Agency for International Development iii EXECUTIVE SUMMARY The Egyptian higher education system is challenged by inequitable access, poor quality, and gross inefficiencies. The Government of Egypt (GOE) and policymakers recognize that a serious reform effort is needed to improve the system in order to stay competitive in today’s emerging global society. Over the years, in support of the GOE’s efforts to reform its education system, USAID/Egypt has provided significant support to primary and higher education in the areas of school based reform, institutional support, and scholarships. On May 4, 2010, USAID/Egypt awarded a cooperative agreement to the Institute of International Education (IIE) for US $9,759,921 to implement the Leadership Opportunity Transforming University Students (LOTUS) Scholarship Program through September 30, 2016. The LOTUS undergraduate program is designed to competitively award scholarships to financially disadvantaged Egyptian students with outstanding academic and extra-curricular credentials and a demonstrated commitment to community service. Scholarship recipients were selected from all 27 Egyptian governorates and two former governorates in an effort to achieve geographic and gender diversity. Students entered fields of study that are considered in high demand and critical to Egypt’s sustained economic growth and development of an internationally competitive workforce. Since the signing of the cooperative agreement, eleven modifications were approved up until March 2015. The results have been (a) an expansion of the project size from 100 scholarship recipients in two cohorts to 250 in four cohorts, (b) an extension of the implementation period from September 30, 2016 to September 30, 2019, and (c) an increase in the funding amounts from $9,759,921 to $23,735,013. In order to implement this program, IIE partnered with three non-governmental organizations (NGOs) and six private Egyptian universities. NGOs include Egyptian Society for Intercultural Programs (AFS-Egypt), Egyptian Association for Educational Resources (E-ERA), and Nahdet el Mahrousa. The participating private universities are Arab Academy for Science, Technology and Maritime Transport, Al-Ahram Canadian University, British University in Egypt, Future University in Egypt, Modern Sciences and Arts University, and Pharos University in Alexandria. The scholarship initiative is based on the development hypothesis that increased scholarship opportunities for targeted youth will contribute to a better-educated workforce that responds to Egypt’s labor market needs. EVALUATION PURPOSE AND QUESTIONS The purpose of this midterm performance evaluation is to provide USAID/Egypt with information to help improve the performance of LOTUS and its contribution to USAID/Egypt’s development objectives (i.e., Workforce response to labor market demands improved, which falls under Intermediate Result [IR] 3.1: Access to Quality Tertiary Education Increased). The results will provide information critical to understanding the program’s efficacy and relative importance to the higher education portfolio, as well as making programmatic decisions over the remaining implementation period. The midterm performance evaluation is designed to answer the following questions: 1. To what extent does available evidence suggest that the project is on track to achieve its objectives? 2. To what extent has the LOTUS Program enabled recipients to contribute to development, community service, and leadership activities? 3. To what extent are scholarship recipients satisfied with the academics related and all the other program components like Study Abroad, English language training, leadership in action activities, career counseling, housing, university coordinators, etc.? iv 4. To what extent do scholarship recipients graduate with the academic and soft skills (workforce preparedness) needed to work in jobs suited to their academic preparation? EVALUATION DESIGN, METHODOLOGY, AND ANALYSIS USAID/Egypt issued a Request for Task Order Proposal (RFTOP) to The QED Group, LLC, under the Services to Improve Performance Management, Enhance Learning and Evaluation (SIMPLE) Indefinite Delivery/Indefinite Quantity (IDIQ), on March 3, 2016. The request was for a midterm performance evaluation of the LOTUS Scholarship Program. The evaluation team included five key personnel — an international consultant team leader, three independent Egyptian senior evaluators, and a senior statistician. The evaluation was conducted between April and June 2016. The evaluation team adopted a mixed methods approach in which primary quantitative and qualitative data were collected. The team reviewed more than 65 program-related documents, administered a paper-and￾pencil survey to 159 currently enrolled students, and hosted an online survey targeted at 74 LOTUS graduates. Sources of qualitative data included 12 group discussions with more than 100 currently enrolled students and alumni, and 25 key informant individual and group interviews with university presidents, student support staff and faculty, implementing partner NGOs, and USAID activity managers. Secondary quantitative data was gathered from IIE and participating universities and consolidated with the primary qualitative data collected. Content analysis was used to quantify the qualitative data by identifying themes, which were then tabulated and analyzed. SPSS and STATA statistical analysis packages were used to analyze the data. DATA LIMITATIONS The evaluation faced several limitations: 1. A limited window of opportunity for data collection and analysis due to the closing academic year; 2. Student examination schedules and intervening national and religious holidays shortened the time needed to pilot the instruments and limited the full utilization of the rich set of data collected; 3. Absence of control group data for comparative analysis with non-LOTUS students; 4. An important shortcoming in the online survey was that it did not collect respondents’ identification information. This made it impossible to follow up with employed graduates to get contact information of their employers for the analysis on employability; and 5. The lack of adequate quantitative data on leadership skills and workforce preparedness. The results of the evaluation are outlined below. SALIENT CONCLUSIONS AND ACTIONABLE RECOMMENDATIONS Evaluation Question No. 1: To what extent does available evidence suggest that the project is on track to achieve its objectives? Salient Conclusions 1. LOTUS is on track to achieve USAID/Egypt objectives 1, 2, and 3 for young, financially challenged men and women across Egypt.1 However, regarding IIE’s nine objectives (as reported on IIE’s latest 1 USAID Objectives are to: “1) Identify and empower young women and men who have demonstrated academic excellence, leadership, and involvement in their communities; 2) Give Egyptian students an opportunity to attend private universities that would normally be above their families' financial means; 3) Develop and nurture the recipients' leadership potential, skills sets, and v M&E matrix dated December 27, 2015), IIE has (a) exceeded two (#4-ITP Bridge Year and #9- Female Enrollment), (b) achieved three (#1-Total Student Enrollment, #5-US Study Abroad and #8-Governorates Represented), (c) did not achieve one (#7-University Capacity Building), (d) does not demonstrate significant progress towards the achievement of one (#3-English proficiency), and (e) cannot clearly demonstrate achievement of two performance indicators due to limitations on the interpretation of the reported M&E matrix data (#2-Leadership and #6-Employability).2 2. Current evidence suggests that the hypothesized impact of the short-term achievements of individual students on long-term institutional and national level objectives is questionable. No evidence is yet available to suggest the return of graduates to work in their home governorates, or the transformation of the personal ties among LOTUS graduates and students into a national network of young leaders. 3. Except in the case of Pharos University, Leadership Capacity Building for the universities has been ineffective with no links to ongoing campus training activities. In general, the LOTUS Leadership Capacity Building component for university staff was sporadic, incomplete, and unmemorable. 4. As an unanticipated outcome, LOTUS strengthens partner private universities by enrolling high quality students who challenge the quality of education provided and contribute directly to building future capacity as teaching assistants after graduation; to date, 11 percent of LOTUS alumni are employed as teaching assistants (a full-time paid professional position) in partnering universities. LOTUS is lauded by stakeholders and participants as a valuable resource for marginalized young men and women to expand their options for career advancement and personal growth. 5. Recruitment was comprehensive and on-the-ground. Selection procedures and criteria were clearly defined and applied uniformly across the governorates and applicants. 6. LOTUS students do not see employment options in their fields of study in their local governorates and many want to go abroad to work or study. However, most affirm ties to their home governorates in terms of community service. 7. LOTUS students have established a network of friendships within and across universities and increasingly across cohorts. This network continues among graduates. 8. The gender parity requirement in LOTUS has been an effective method to ensure equal opportunities for young men and women. Gender parity is seen in the program as a whole, and across components as well. Recommendations for IIE Action/To be Taken in the Near Future Build on and formalize the incipient network of LOTUS students and alumni to move toward the objective of establishing a nationwide network of young professionals. Although this is a long-term objective that will ultimately need to be self-sustaining, in the remaining three years of the LOTUS Program, the following steps can be taken: 1. Identify and work with interested LOTUS graduates to reach out to graduates of similar USAID commitment to community and country, so that they are prepared and equipped to become future leaders and advocates for development in their local communities; (4) Enhance the recipients' employability and career options; and (5) Create a network of youth nationwide who are well educated and passionate about Egypt.” 2 IIE’s nine objectives are: (1) Two hundred and fifty financially disadvantaged female and male youths from 27 governorates are enrolled in undergraduate programs of study in private Egyptian universities; (2) At least 90% of LOTUS graduates are empowered to assume future leadership roles and civic responsibilities; (3) At least 90% of LOTUS graduates are proficient in the English language; (4) At least 90% of Cohort 3 and 4 students complete the bridge year program successfully; (5) LOTUS students are placed in Study Abroad program opportunities in the US; (6) At least 90% of LOTUS students demonstrate enhanced employability; (7) At least 100 staff and faculty from partner Egyptian universities receive capacity building training; (8) At least two youths from the majority of Egypt's governorates are provided access to quality higher education (Cohorts 1-4); and (9) At least 125 girls/women are provided with access to higher quality education.” vi programs (e.g. LSP, LEAD). This would broaden the core network. 2. Set up and/or formalize a virtual communication infrastructure. 3. Use the network to communicate employment or volunteer opportunities, project development opportunities, presentations, and conferences of particular interest to LOTUS students; encourage other partnering institutions to contribute as well. Evaluation Question No. 2: To what extent has the LOTUS Program enabled recipients to contribute to development, community service, and leadership activities? Salient Conclusions 1. The Leadership in Action (LIA) component was implemented through a series of sessions and workshops to enhance students’ leadership skills. Students assessed the program positively and stakeholders consider LIA to be effective. 2. LOTUS students report that LIA has deepened their understanding of volunteering and community service. 3. The component could be more effective and less problematic for both students and implementers with improved management and coordination. Recommendations for IIE Action/To be Taken in the Near Future 1. Coordination and Management: Establish a coordination committee among participating organizations to meet on a quarterly basis for information exchange about schedules and student obligations. Identify potential points of coordination among the parties ( e.g., linking career counseling to university career centers, or tapping university resources to identify internships). 2. LIA Scheduling and Prioritization: Establish a list of sessions for the year (or semester) and schedule sessions in each university such that their timing does not conflict with students’ academic or other legitimate obligations. In the case of sessions offered multiple times, allow students who attended a session once to not have to attend repeated sessions unless they desire. Evaluation Question No. 3: To what extent are scholarship recipients satisfied with the academics related and all the other program components like Study Abroad, English language training, leadership in action activities, career counseling, housing, university coordinators, etc.? Salient Conclusions 1. Scholarship recipients welcome the opportunity to learn English and recognize the importance of English proficiency, but are not satisfied with the training modes (intensive supplementary courses and/or bridge year) introduced to date. 2. Tracking the relationship between English proficiency and GPAs shows they do not vary together in three of the six partner universities. 3. The most important benefits of the US Study Abroad experience for the students are cross￾cultural exposure, an enrichment experience that cannot be clearly measured, and such practical benefits as improved employability and expanded job options. Study Abroad has heightened the students’ self-confidence and leadership, improved their English, and strengthened their resumes; all of which have improved the students’ job options. 4. The original intent of using internships to link students to their home governorates has not been achieved; 79 percent of all internships to date take place outside of students’ home governorates. 5. Internships are important to employability and career development but issues of coordination have inhibited the full realization of these benefits. 6. Students face difficulties in finding and reporting internships. vii 7. Implementation of the internship component has been dispersed among partner organizations without a central management point. 8. Communication among students, IIE, and the universities are centralized in IIE, and managed through a one-track communication channel focused on the university and IIE coordinators. Limiting students’ communication with IIE and the university to two individuals makes student feel isolated and disregarded. Students are told what to do (and reprimanded), without an effective voice in managing their lives.3 9. Housing problems affect student integration into university life. 10. Communications between LOTUS students and coordinators are a source of tension, mistrust, and student dissatisfaction. 11. Students report little knowledge of or benefit from the career counseling and mentoring component. Recommendations for Future Design 1. If the bridge year concept is to be repeated, include a variety of activities in addition to English. More time could be given to soft skills training, introduction to university life and studies, and career counseling and practicums. 2. If the bridge year concept is to be repeated, students recommend that career counseling begin during the bridge year to support their selection of a field of study. Recommendations for IIE Action/To be Taken in the Near Future 1. Strengthen the management and integration of the internship components by: a. systematizing the process for identification, reporting, and follow-up on internships, b. compiling and posting internship opportunities, and c. establishing a working relationship with private sector entities interested in internships linked to specific fields of study. 2. Strengthen links between Cohorts 1 and 2 and Cohorts 3 and 4. Consider having graduates meet with current students socially on a regular basis to discuss their challenges and to serve as mentors. 3. Designate assistant coordinators in AASTMT and BUE to be accessible to students and manage program coordination with the implementing partner. 4. Conduct broad-based consultations with students prior to setting LOTUS program policies. 5. Students recommend that career counseling focus on information about the labor market and gender roles relative to career and job choice. In addition, face-to-face counseling and small￾group mentoring should continue across the four years of study and address topics pertinent to seeking an internship (e.g., resume writing, interviewing skills, and later employment search). 3 Group discussions with students in more than one university revealed their dissatisfaction with the way their coordinators from both IIE and their particular universities handled them/their issues. Two highlights in particular were (a) the way coordinators address students, which the latter perceived as demeaning, and (b) depriving them, the students, from participating in the process of making decisions that they think are very much related to their everyday lives at the universities. viii Evaluation Question No. 4: To what extent do scholarship recipients graduate with the academic and soft skills (workforce preparedness) needed to work in jobs suited to their academic preparation? Salient Conclusions 1. A majority of LOTUS graduates are employed and working in areas related to their fields of study. Those who are not employed are in residency, military service, training, or graduate school. A fourth of the recent graduates are unemployed and looking for work. 2. Current students attribute their anticipated success to academic achievement, soft skills training, and internships. 3. The recognition by others that the LOTUS scholarship is merit-based increases student chances for employment. Recommendation for Future Design In collaboration with the private sector and/or partnering universities, consider establishing a financing or grant mechanism, for providing (1) loans or grants to assist recent graduates with specific costs associated with the job search; (2) an incentive fund for start-up businesses or professional practices in scholarship recipients’ home governorates; and (3) start-up seed money to fund graduate/student community service projects that benefit scholarship recipients’ home governorates and/or for incentivizing start-up businesses or professional practices in home governorates. The mechanism also could be used to establish links to other USAID activities focused on workforce development and/or employment generation. Recommendation for IIE Action/To be Taken in the Near Future Encourage participating universities to consider providing support services to recent graduates to assist them in their early job search and career development. 1 INTRODUCTION The purpose of the midterm performance evaluation of the Leadership Opportunity Transforming University Students (LOTUS) Scholarship Program is “to provide USAID/Egypt with information to help improve the performance of LOTUS and its contribution to USAID Egypt’s development objectives. The results will provide information critical to understanding the program’s efficacy and relative importance to the higher education portfolio, as well as in making programmatic decisions over the remaining implementation period” (see Annex I, LOTUS Evaluation Statement of Work). The audience for the evaluation is USAID/Egypt, particularly, the education team within the Office of Education and Health (OEH); the implementing partner, Institute of International Education (IIE); LOTUS scholarship recipients; participating universities; and USAID/Washington. As a public document, it will be shared with other stakeholders, donors, and education-related non￾governmental organizations (NGOs). Once approved, the final report will be submitted to the USAID Development Experience Clearinghouse (DEC) for public access and use. The evaluation statement of work spells out four specific evaluation questions: 1. To what extent does available evidence suggest that the project is on track to achieve its objectives? 2. To what extent has the LOTUS Program enabled recipients to contribute to development, community service, and leadership activities? 3. To what extent are scholarship recipients satisfied with the academics related and all the other program components like Study Abroad, English language training, leadership in action activities, career counseling, housing, university coordinators, etc.? 4. To what extent do scholarship recipients graduate with the academic and soft skills (workforce preparedness) needed to work in jobs suited to their academic preparation? Data collection, analysis, and report writing are built around these four questions. Because this is a midterm evaluation, recommendations are directed primarily to IIE for process improvement, and secondarily to USAID/Egypt for consideration in future higher education programming. BACKGROUND The Egyptian higher education system is challenged by inequitable access, poor quality, inefficiency, and a reputation for producing graduates who lack critical thinking and complex communication skills. The Government of Egypt (GOE) recognizes that reforms are needed to improve the higher education system in order to compete effectively in the global economy. In support of the GOE efforts to reform its education system, USAID/Egypt has provided significant support over the years in the way of school-based reform, institutional support, and scholarships. The LOTUS Scholarship Program builds on this experience. It seeks to close the gap between high school graduates who are able to access high quality education and economically disadvantaged Egyptians who have limited options. This is being done through scholarship awards to highly qualified high school graduates from all 27 governorates in the country to attend private universities that they would not otherwise be able to afford. The scholarship initiative aims to improve their potential for employment in productive and high paying jobs and contribute to Egypt’s development. 2 The LOTUS Scholarship Program began in 2010 with a cooperative agreement (No.263-A-00-10-00026- 00) with IIE/Egypt. The agreement was signed on May 4, 2010. The initial end-of-project date was September 30, 2016, but a series of 11 modifications through March 3, 2015 extended the program to September 30, 2019 and increased funding to US$23,735,013 (see Annex I, Evaluation Statement of Work, Section K). The number of scholarships awarded grew from 100 to 250 with the addition of three additional cohorts of 50 students each. The LOTUS Program is distinguished by several key characteristics: it is national, provides full scholarships, works only with private universities, and uses merit, financial need, and leadership potential as selection criteria. The program was implemented nationwide and recruited and enrolled students from all governorates. Gender parity and geographic considerations were also requirements. Scholarships were to be awarded to an equal number of males and females and, especially for the first two cohorts, awardees from each governorate had to include at least one male and one female. Students were enrolled in five (later six) private universities in Cairo and Alexandria in specific fields of study. These fields are considered in high demand and critical to Egypt’s sustained economic growth and development of an internationally competitive workforce. LOTUS students participated in activities to enhance their leadership skills and commitment to development in Egypt and their home governorates. The five university partners for Cohorts 1 and 2 were British University in Egypt (BUE), Modern Sciences and Arts University (MSA), Future University of Egypt (FUE), Ahram Canadian University (ACU), and Pharos University in Alexandria (PUA). At the start of Cohort 3, MSA withdrew and the Arab Academy for Science, Technology, and Maritime Transport (AASTMT) joined as an additional partner. The ultimate goal of the program is to support development in Egypt by training a cadre of future leaders. This will be done by providing high quality undergraduate education to a limited number of promising Egyptian students with great financial need (determined by documented household income and assets and home visits). It is the hypothesis that increased scholarship opportunities for targeted youths will create a more educated workforce that is able to respond to Egypt’s labor market needs. LOTUS, therefore, fits into USAID/Egypt’s Strategic Framework by supporting Development Objective (DO) 22 (Educated Workforce that Responds to Labor Market Needs) and contributing to Intermediate Result (IR) 1 ( Tertiary and Workforce Development Programs Able to Produce Graduates with Relevant Skills) and Sub IR 1.1 ( Increased Access to Quality Tertiary Education). LOTUS objectives are to: • Identify and empower young women and men who have demonstrated academic excellence, leadership, and involvement in their communities; • Give Egyptian students an opportunity to attend private universities that would normally be beyond their families’ financial means; • Develop and nurture recipients’ leadership potential, skills sets, and commitment to community and country so they are prepared and equipped to become future leaders and advocates for development in their local communities; • Enhance recipients’ employability and career options; and • Create a network of youths nationwide who are well-educated and passionate about Egypt. The midterm evaluation comes at a crucial point in LOTUS’ implementation. The first two cohorts of scholarship recipients are completing their undergraduate degrees and moving into the workforce. In 3 addition, Cohorts 3 and 4 are beginning their studies amid USAID/Egypt changes in the design of the program. These changes include (1) an increased emphasis on recruitment from underserved governorates4 and Sinai; (2) a reduction in the number of participating universities from five to two (i.e., BUE and AASTMT); (3) the introduction of an initial bridge year at BUE to focus on English language instruction as well as some training in leadership and career development soft skills; (4) a change in the available fields of study by eliminating five-year programs in engineering, dentistry, pharmacy, and physical therapy and introducing/emphasizing others, such as logistics and nursing; and (5) a reconfiguration of the roles of partner NGOs. METHODOLOGY AND LIMITATIONS The LOTUS objectives introduced in Evaluation Question 1 served as the foundation for this evaluation of program achievements. The team used a mixed methods approach to bring multiple types of evidence to bear on the analysis of each evaluation question. A desk review of the project documents in Annex IV preceded design of the 15 quantitative and qualitative tools used in data collection (see Annex III). This review provided background information on the program as well as monitoring data and indicators for the six years of implementation. A team of three evaluators, under the auspices of USAID/Egypt SIMPLE, engaged the six universities and management teams from three NGOs, IIE, and USAID/Egypt. A fourth evaluator worked with the team on the design of tools and initial data collection, and a statistician managed the survey and secondary data analysis. The evaluators were supported by a project coordinator, three field coordinators/survey proctors, and a SIMPLE evaluation manager. Field work for the evaluation was conducted in Cairo and Alexandria over a five-week period between April 17 and May 19, 2016. Two surveys, a paper-and-pencil survey of current students and an online survey of graduates, were the primary sources of quantitative data. Given that a total of 250 students received LOTUS scholarships, the surveys were administered to the full population, not a sample, of students and graduates. The response rate to the student survey was 97.5 percent and 54 percent to the online graduate survey.5 IIE also provided several student databases that included demographic information, GPAs and English language proficiency scores, and employability status of LOTUS alumni. The evaluation team merged the student databases with the survey responses to provide a base for statistical analysis of students’ perceptions relative to demographic and achievement characteristics (see Annex II for details about the process). 4 Underserved governorates have several characteristics: low economic and literacy levels, high student dropout rates, preference for male not female education, no or very few higher education institutions, many underrepresented communities/areas, and political instability. 5 According to the article, “Survey Response Rates,” posted to the Surveygizmo website, “surveys that you distribute internally (e.g., among employees) generally have a much higher response rate than those distributed to external audiences (e.g., customers).” Internal surveys generally receive a 30-40 percent average response rate (or more), compared to the average 10- 15 percent response rate for external surveys. As LOTUS scholarship recipients are considered an internal audience, SIMPLE concludes that the graduate online survey response rate is well above the average for internal surveys. 4 Group discussions and key informant and group interviews were further used to collect qualitative data. A subset of 82 students (44 females and 38 males) who completed the survey also participated in group discussions at each university. The 11 student group discussions included three for female students only, five for male students only, and three for both genders. In addition, one group composed four female graduates. Key informant individual and group interviews (KII and GI) at each university targeted university presidents, LOTUS coordinators, and faculty and staff who received LOTUS training. The final set of key informant individual and group interviews was held with the implementing organizations (IIE/Egypt, AFS Egypt, E-ERA, Nahdet el Mahrousa, and USAID/Egypt activity managers). A total of 25 interviews were completed with approximately 72 individuals. The interviews included a telephone interview with IIE/New York staff involved in the Study Abroad component. Notes from the 12 group discussions and 25 interviews were typed and shared with team members. The notes were then coded and entered into one of three Excel tally sheets under key evaluation themes. These themes were first identified from the discussions and then additional items were added for the university interviews and NGOs/implementers in order to systematize the analysis across data sources. Interview and discussion notes are not included in the report in order to protect the confidentiality of the participants. Annex II provides a detailed description of the methods and limitations, including documentation of steps taken to ensure confidentiality. Limitations The principal limitation of data collection and analysis was time. The start of final exams at universities pushed the student survey and group discussions to an early point in the evaluation process so that the design of the questionnaire and protocols were based entirely on document review rather than key informant group interviews. Moreover, as is the case in all evaluations, the time for data coding and analysis was insufficient to utilize the information collected to its fullest extent.6 There are also several limitations as to the scope of the data. First, available resources prevented the use of a comparison group, which would have distinguished findings attributable to the LOTUS Program.7 Second, the online graduate survey, administered using Survey Monkey, did not record demographic information and, thereby, precluded any analysis of the impact of such factors. Responses to the online survey also could not be merged with the IIE graduate employability database. Finally, the analysis for Evaluation Question 4 concerning employability was restricted by the absence of objective data on the skills and aptitudes required for workforce preparedness in the fields of study pursued by LOTUS students. The evaluation team was unable to interview employers or supervisors of LOTUS alumni since IIE does not collect this information and the graduates declined to provide it in the online survey. Therefore, in lieu of direct measures of employability, Question 4 reports perceptions of employability. 6 The initial concern that Christian students would not participate in the group discussions because they were conducted during the Christian holy week was unwarranted. Attendance was higher than expected; absence was not a significant issue. Students were eager to participate and some not on the original list requested to take part. In addition, students not present during the first visit of the evaluation team were invited to complete the survey during the team’s second visit to the universities. These second visits focused primarily on key informant and group interviews with faculty and staff. A third group discussion was also held in AASTMT and BUE during the second visit. 7 IIE has maintained contact with some young people who were offered a scholarship but did not take it and others who were alternates for the award. The LOTUS M&E specialist interviewed a sample of these individuals (n=16) for an internal 2014 report titled “LOTUS Scholarship Program Comparison Group Mid-Term Evaluation Report.” In the evaluation design, USAID decided that the costs to re-contact these individuals, who are widely dispersed, were not justified. The 2014 report showed small differences in most areas except leadership and critical thinking. 5 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS This section presents the findings, conclusions, and recommendations of each of the four evaluation questions. EVALUATION QUESTION 1 (EQ1): TO WHAT EXTENT DOES AVAILABLE EVIDENCE SUGGEST THAT THE PROJECT IS ON TRACK TO ACHIEVE ITS OBJECTIVES? As outlined in the evaluation statement of work (see Annex I), the individual-level objectives of LOTUS are to: 1. Identify and empower young women and men who have demonstrated academic excellence, leadership, and involvement in their communities; 2. Give Egyptian students an opportunity to attend private universities which would normally be beyond their families' financial means; 3. Develop and nurture recipients' leadership potential, skills sets, and commitment to community and country so that they are prepared and equipped to become future leaders and advocates for development in their local communities; 4. Enhance the recipients' employability and career options; and 5. Create a network of youths nationwide who are well educated and passionate about Egypt. The development hypothesis underlying the design of the program is that increased scholarship opportunities for targeted youths will contribute to a better-educated workforce that responds to Egypt’s labor market needs. As a result, the program’s goal is to support development in Egypt by training a cadre of future leaders. Achieving the five short- and long-term objectives will contribute to this overarching goal. The general conclusion regarding this evaluation question is that LOTUS is on track to achieving its individual-level student objectives. However, the extent to which these achievements contribute to Egypt’s long-term national and institutional objectives (e.g., support for development in local communities and a nationwide network of youths) is still unclear. Additional time is needed to evaluate whether these desired outcomes have been accomplished. The first three findings below address progress to date on LOTUS’ individual-level student objectives. The next set of findings examine institutional and national objectives. Conclusion: The LOTUS Scholarship Program has had mixed results insofar as meeting its nine implementation objectives.8 Targets for student enrollment (objective 1) and access to quality higher education (objective 8) are both fully achieved; those for successful completion of the bridge year program (objective 4) and gender access to quality higher education (objective 9) exceeded plans; 8 LOTUS M&E Matrix, dated December 27, 2015. See Annex VI, Part II [Percent of IIE Lotus Targets Achieved] for a list of objectives and achievements to date. 6 objectives 3 (English language proficiency) and 7 (capacity building training in universities) have not been achieved; and, finally, Study Abroad (objective 5) targets are in progress, while student empowerment (objective 2) and greater employability (objective 6) are undetermined due to the unavailability of data to measure achievement in these areas. Findings and Supporting Evidence: IIE and its partners identified and enrolled 250 students in four cohorts between 2010 and 2014. These students match the selection criteria of great financial need, academic excellence, demonstrated leadership, and community service. They come from all governorates, comprise an equal number of males and females, and are diverse in their skills, interests, families, and community backgrounds (see Annex VI, Table VI.1). Students are enrolled and supported in six private universities and have continued to demonstrate academic excellence, leadership, and community service (see Annex VI. Table VI.2).9 To date, only three percent of the scholarship recipients have left the program prior to graduation. Five students were terminated because of unsatisfactory academic performance, and three have voluntarily withdrawn. Most of the indicators used to measure progress are output indicators showing participation in the various LOTUS activities designed to achieve the stated objectives. The indicators do not measure results per se. For example, while the evaluators know that students took part in the Leadership in Action (LIA) program, there is no direct measure of their leadership or empowerment. The area students seem to struggle in the most is English language proficiency, as measured by ITP tests. Although they have achieved intermediate English proficiency as a result of the program, the goal of having 90 percent of students at a high level of proficiency has not been reached. Conclusion: Stakeholders and participants laud the LOTUS Program as a valuable resource for marginalized young men and women to expand their options for career advancement and personal growth. Findings and Supporting Evidence: LOTUS was universally praised in interviews, group discussions, and surveys. Everyone associated with the program, including the partner universities and NGOs, graduates, and students, voiced support for its continuation and expansion.10 Current students and graduates overwhelmingly stated that they would recommend or have recommended LOTUS to their siblings and close friends. Only four out of the 154 students surveyed, one each from Cohorts 2 and 4 and two from Cohort 3, said they would not.11 Nearly all graduates (28 out of 29 surveyed) said “yes” and one did not know whether he or she would recommend the program. In interviews, implementers/partners supported the concept and program results for highly qualified financially challenged students from across Egypt, but noted shortcomings or challenges in implementation. These, however, do not override their general support. The number of applications received increased dramatically from 346 for Cohort 1 to 1,281 for Cohorts 1.2 and 2. Applications for Cohort 3 decreased to 700 but jumped again to 901 for Cohort 9 See also the extended discussions for Evaluation Questions 2 and 3. 10 The evaluation team successfully contacted one student of five who had been terminated for lack of performance, and one of three who departed voluntarily. The terminated student voiced strong support for LOTUS despite his/her individual problems. The student who voluntarily withdrew was the only person who felt the negatives in the program outweighed the positives. 11 Only one of the four students provided a written comment (in English) about the lack of support. The person stated that the program comprised students who only wished to study rather than enjoy themselves. 7 4.12 Compared to the number of applications for Cohort 1, those for all subsequent cohorts were substantially higher. This may reflect the positive recognition and successful branding of the LOTUS Program. NGOs reported that the initial announcement of the scholarship was met with suspicion about USAID motives in supporting students from marginal families in remote areas with 100 percent scholarships. Questions were also posed about why support was being offered to young women. In contrast, by the fourth recruitment cycle, recruiters were receiving calls about when applications would be available. No significant gender differences were observed in the number of applications received from the various governorates. IIE indicated that there were more qualified females than males overall in the selection pool. Conclusion: Recruitment was comprehensive and on-the-ground. Selection procedures and criteria were clearly defined and applied uniformly across the governorates and applicants. Two experienced NGOs (AFS-Egypt and E-era) handled recruitment for LOTUS and participated in the selection process. Findings and Supporting Evidence: The governorates were divided into two groups, based on the strengths of the local networks of each of the recruiting organizations. These organizations used various modes of contact — announcements in El Ahram newspaper; notifications via local offices of the Ministry of Education; local information sessions; and schools and social media — to recruit students. They did not directly target females or males, although parents were concerned about housing and security for their daughters, and about options for fields of study for their sons. Recruiters stated that, in remote and conservative communities, young women and their parents were sometimes reluctant even to attend LOTUS information sessions owing to social pressure against higher education for women, particularly outside local areas. In these cases, recruiters relied upon trusted religious leaders and current female students to encourage prospective female candidates. The Study Abroad component was a further deterrent for some women and their parents. Three staff members from IIE, E-ERA, and AFS screened and scored applications for completeness and eligibility. Each eligible candidate then received a home visit by at least three people who scored the visit according to a pre-set IIE rubric. The final step was participation in a two-day camp, where candidates were once again scored by three people on the basis of participation and interaction in group activities, and an individual interview. All applicants — males and females from across the governorates — were part of a single selection pool for each cohort. The results matched the program objectives in terms of diversity in demographic profile and financial need, gender parity, and representation from remote areas (see Annex VI, Table VI.1). In interviews, university presidents and LOTUS coordinators cited the high quality of LOTUS students in academic achievement and commitment as a defining characteristic of the program and the principal benefit of their participation in the program. Conclusion: LOTUS students do not see employment options in their fields of study in their local governorates and many want to go abroad to work or study. Most affirm ties to their home governorates in terms of community service. Findings and Supporting Evidence: Students in 8 of the 11 discussion groups expressed their desire to travel abroad, usually because of perceived employment opportunities in their fields of study. 12 IIE notes that the decrease in applications for Cohorts 3 and 4 coincides with the reduction in the number of options for fields of study, including such popular choices as engineering and pharmacy. 8 Only two students in dentistry and pharmacy expressed a desire to return to their home communities to work. In contrast, the IIE graduate database shows that only two out of 83 graduates are currently living abroad. Half of the graduates who responded to the online survey said they had done community service in their home communities during the past year, but none of the employed graduates are working in their home governorates.13 Conclusion: LOTUS students have established close friendships and networks within and across universities and, increasingly, across cohorts. This network continues among graduates. Findings and Supporting Evidence: In student discussions there were frequent references to LOTUS as “family” or “my second family.” Seventy-nine percent of 153 students surveyed contacted friends and peers to discuss the challenges of university life, while 78 percent sought other LOTUS students. When asked how often they interacted socially with their LOTUS colleagues, 60 percent said “always” and 29 percent said “often;” 30 percent interacted “always” and 43 percent “often” with other students. Recent graduates discussed their interest in strengthening ties between Cohorts 1 and 2 and Cohorts 3 and 4, which they see as potentially beneficial. IIE has invited them to participate in events involving current students (e.g., speaking with new entrants during orientation sessions, and attending the annual student-led leadership conference). A further indicator of the strength of ties among graduates and LOTUS students is the extent of their current contacts with one another. The online survey of graduates found that other LOTUS alumni and students were their most frequent contacts (an average of 2.83 on a four-point scale ranging from rarely [1] to always [4]). Other alumni from their fields of study had an average score of 2.81, while high school friends scored 2.75. Conclusion: The LOTUS Leadership Capacity Building component for university faculty and staff has been sporadic and incomplete, except in the cases of Pharos University in Alexandria and Ahram Canadian University. Findings and Supporting Evidence: According to IIE’s cooperative agreement with USAID/Egypt, the leadership capacity building activity is intended to “strengthen the leadership of each partner university and imbed an enriched understanding of leadership principles that can then serve the success and potential institutionalization of the Leadership in Action Program at each university. It is imperative to have champions within university faculty of the partner institutions who truly understand the importance of leadership development inclusive of character building and self-discovery, life and work skills, and service. Without these ‘champions,’ programs are rarely sustained and students find fewer outlets for leadership action and service with the university community. As part of the capacity￾building sessions, IIE proposes to also include discussions on scholarship administration.” In its evaluation interview, IIE also explained that the capacity building activity is a direct benefit to the universities, in part, to reward them for their participation and collaboration in the program cost share requirement. Group interviews with faculty and staff from four of the five universities (AASTMT has not yet received training) revealed minimal recall of the training sessions and material. Participants had very little to no knowledge of the LOTUS Program or its components on campus, although at least some in each group 13 See Evaluation Questions 3 and 4 for detailed evidence to support these observations. 9 knew of the presence of LOTUS students. Only two universities have received the full curriculum, though scheduling was sporadic. Participants maintain that IIE has not followed up in spite of interest on the part of university administrators. The exception to the above evidence is Pharos University in Alexandria. PUA is creating a career development center for faculty based on the IIE curriculum to provide comprehensive training to strengthen classroom instruction and faculty-student relations. The training offered in the new center is mandatory for all new faculty. Conclusion: LOTUS-affiliated universities find working with the program beneficial. Administrators and university coordinators cite the high quality of LOTUS students as the primary benefit for their institutions. They praised the selection process for its ability to identify such students. To date, 11 percent of 83 LOTUS alumni are employed as teaching assistants in their universities. SUMMARY TABLE EQ1: EXTENT TO WHICH LOTUS IS ACHIEVING ITS OBJECTIVES Findings 1. IIE and its partners identified and enrolled 250 students in four cohorts between 2010 and 2014. These students match the selection criteria of great financial need, academic excellence, demonstrated leadership, and community service. Scholarship recipients come from all governorates and represent both genders equally. 2. Stakeholders and participants laud LOTUS as a valuable resource for marginalized young men and women to expand their options for career advancement and personal growth. 3. Administrators and LOTUS university coordinators cite the high quality of LOTUS students as the primary benefit of the program. To date, 11 percent of LOTUS alumni are employed as teaching assistants in their universities. Conclusions 1. IIE is making progress in its nine implementation objectives. Targets for student enrollment and access to quality higher education are fully met. The bridge year program has been successfully completed and gender access to quality higher education has exceeded plans. Targets for English language proficiency and capacity building training in universities have not been met; Study Abroad is in progress; student empowerment and greater employability accomplishments are both undetermined. 2. The hypothesized impact of the short-term achievements with individual students on long-term institutional and national-level objectives is questionable. There is no evidence to suggest that graduates return to work in their home governorates, or that personal ties among LOTUS graduates and students transform into a national network of young leaders. 3. Except for PUA and ACU, the Leadership Capacity Building initiative for university staff has been ineffective, with no links to ongoing campus training activities. 4. An unanticipated outcome of LOTUS is the strengthening of partner universities. Enrolling high quality students effectively challenges the quality of education provided and contributes directly to building future capacity as teaching assistants after graduation. 5. Stakeholders and participants view LOTUS as a valuable resource for marginalized young men and women to expand their options for career advancement and personal growth. 6. Recruitment was comprehensive and on-the-ground. Selection procedures and criteria were clearly defined and applied uniformly across applicants and governorates. 7. LOTUS students do not perceive employment options in their fields of study in their local governorates, and many want to go abroad to work or study. Most affirm ties to their home governorates in terms of community service. 8. LOTUS students have established a network of friendships within and across universities and increasingly across cohorts. This network continues among graduates. 9. LOTUS-affiliated universities find working with the program beneficial. 10 Recommendations for IIE Action/To be Taken in the Near Future 1. Build on and formalize the incipient network of LOTUS graduates and students to move toward the objective of establishing a nationwide network of young professionals. Although this is a long-term objective that will ultimately need to be self-sustaining, steps can be taken toward laying the groundwork and infrastructure in the remaining three years of the program. 2. Identify and work with interested graduates to reach out to graduates of similar USAID programs (e.g., LSP, LEAD) to broaden the core network. 3. Set up and formalize the virtual communication infrastructure. 4. Organize intermittent social events for graduates (e.g., a barbecue) to strengthen personal and professional ties within and across programs. 5. Use the network to communicate employment or volunteer opportunities, project development opportunities, and presentations and conferences of particular interest to LOTUS students. Encourage others to contribute as well. EVALUATION QUESTION 2 (EQ2): TO WHAT EXTENT HAS THE LOTUS PROGRAM ENABLED RECIPIENTS TO CONTRIBUTE TO DEVELOPMENT, COMMUNITY SERVICE, AND LEADERSHIP ACTIVITIES? This question examines the Leadership in Action (LIA) component and its perceived effect on students’ capacity to take on leadership roles in the development of their communities. Hence, the focus here is on students’ leadership skills, level of commitment to community service, and actual engagement in community service. Overall, the evaluation team found LIA to be effectively implemented and positively evaluated by students and staff. However, student enthusiasm for the program and ability to take advantage of all the offerings have been compromised by problems in operational planning and coordination. Conclusion: LIA was implemented through a series of sessions and workshops designed to enhance students’ leadership skills. Students assessed the program positively. Findings and Supporting Evidence: Students and LOTUS university coordinators identify LIA as a positive component of the LOTUS student experience. LIA consists of a series of training sessions and activities to strengthen students’ leadership capacity and commitment to development. Training is given in personal growth and development, academic study skills, leadership concepts and fundamentals, leadership skills development, and service learning project and community interaction opportunities. Students have reported that listening, self-discovery, and comfort zone sessions influenced them the most, and that they benefitted from sessions on values, time management, and team work. IIE annual reports document the number of sessions conducted. In 2014, 18 LIA sessions and 32 specialized sessions were conducted at partner universities and attended by 379 students. In 2015, 17 LIA sessions and 100 specialized sessions were conducted at partner universities. Conclusion: Students have indicated that parents and community members now see them as more mature and experienced. This may be indirect evidence of the effectiveness of the LOTUS Program. “I never imagined that I would be a leader. I just didn’t have those characteristics. But LIA showed me that I can.” (Group discussion with students) “LOTUS students are leaders inside their departments.” (Interview with LOTUS university coordinator) 11 “The short notifications did not allow us to be always available.” (Group discussion with students) “Sessions were held on the same day, so we did not have the chance of attending them all.” (Group discussion with students) “We cleaned the streets and talked to the people about the importance of preserving our communities.” (Group discussion with students) “The Student Learning Project gave me an edge and experience, specifically that I implement it in a different governorate.” (Group discussion with students) Findings and Supporting Evidence: Students in all group discussions commented that their parents and community members now seek out and appreciate their opinions and perceptions more so than in the past. They also discussed ways they could affect their communities and resolve certain pressing issues. On a personal level, they feel able to cope more effectively with family conflicts and to build more trust as a result of their LOTUS experience. They all talked about how LIA helped them to understand and deal with different personalities and to approach them constructively. Conclusion: LOTUS students reported that LIA has deepened their understanding of volunteering and community service. Findings and Supporting Evidence: Students in 8 of the 11 discussion groups acknowledged the importance of the community services component. They spoke of the various activities in which they are involved (e.g., in health, environment, and education) on their own and through LOTUS. IIE’s annual reports show that, in 2014, three teams completed service-learning projects, and in 2015, 57 teams completed projects involving 11,751 hours of community service. Out of concern and eagerness to make real changes on the ground, students discussed two ways to increase the effectiveness of this component. First, while they think highly of the service learning project, they believe the application process should be revised to make it less complex and the process for approval and funding made more timely. Second, students lamented that their rigorous academic schedule and extra-curricular sessions leave them less time than in the past to participate in community service activities. The open-ended questions on the student survey offered insights as to other approaches than can be used to maximize the benefit of this component, such as connecting students to funding agencies to support their projects, engaging family members in joint activities with students, and measuring the effects of the service learning project on the community. Conclusion: Students believe operational planning and coordination issues prevent them from making the best use of all LIA sessions. Findings and Supporting Evidence: The discussions and group interviews revealed that the operational planning and coordination of LIA was a challenge for students, IIE staff, and university coordinators alike. Many students had concerns about when and where the LIA sessions were organized and held. IIE academic coordinators highlighted the complexity of organizing these sessions, which involved coordination across six universities, numerous faculties, and presenter schedules. Finally, university coordinators expressed only a “Things are very different in Upper Egypt. Girls don’t speak – now I talk. I interrupt my father. He is pleased.” (Group discussion with students) 12 limited role in making administrative arrangements. At the same time, they are interested in attending and making use of the sessions. SUMMARY TABLE EQ2: EXTENT TO WHICH LOTUS HAS ENABLED DEVELOPMENT, COMMUNITY SERVICE, AND LEADERSHIP ACTIVITIES Findings 1. LIA was implemented through a series of sessions and workshops to enhance students’ leadership skills. Students assessed the program positively. 2. Indirect evidence of the effectiveness of this component is that parents and community members now see students as more mature and experienced as a result of the program. 3. Students say that LIA deepened their understanding of volunteering and community service. 4. Students believe operational planning and coordination issues prevent them from making the best use of all LIA sessions. Conclusions 1. Stakeholders consider the LIA component to be effective. 2. Better management and coordination would make LIA more effective and less problematic for students and implementers. Recommendations for IIE Action/To be Taken in the Near Future In response to (a) students’ sense of overload and perceived repetition of sessions they are required to attend, and (b) lack of knowledge about the program as a whole among partners, the following coordination and management steps can be taken: 1. Establish a coordination committee with representation from all participating organizations to meet on a quarterly basis for information exchange and scheduling, and to identify potential points of coordination among partners (e.g., linking career counseling to university career centers, or tapping university resources to identify internships). 2. Continue meetings between university and IIE coordinators and assistant coordinators to share problems and solutions and to communicate information about schedules and student obligations. Scheduling and setting priorities within LIA: 3. List sessions for the year (or semester) and schedule them so that they do not conflict with students’ other obligations. For sessions offered at multiple times, allow students who attended a session once to not have to attend repeated sessions unless they so desire. 4. Monitor the quality and utility of the sessions in terms of student skills and feedback on presentation. Develop a method or rubric to measure learning in the sessions and topics so as to reduce their total number. EVALUATION QUESTION 3 (EQ3): TO WHAT EXTENT ARE SCHOLARSHIP RECIPIENTS SATISFIED WITH THE ACADEMIC RELATED AND ALL THE OTHER PROGRAM COMPONENTS LIKE STUDY ABROAD, ENGLISH LANGUAGE TRAINING, LEADERSHIP IN ACTION ACTIVITIES, CAREER COUNSELING, HOUSING, UNIVERSITY COORDINATORS, ETC.)? This section of the report discusses student satisfaction with the following LOTUS components: (1) English language training, (2) Study Abroad, (3) internships, and (4) student life. LOTUS scholarship recipients are generally satisfied with the opportunities to learn English and engage in the job market through internships. Those who studied in the United States found the experience exceeded their expectations as it helped develop many of their skills. University life, however, is fraught with problems. Students have experienced difficulties integrating into campus communities, housing and transportation issues, and mistrust in their communications with LOTUS management. 13 English Language Training Conclusion: Students and graduates recognize the importance of LOTUS’ English language training. They believe that improved English language proficiency will increase their chances in the job market. Findings and Supporting Evidence: Graduates think English language training is very important to their careers. Thirteen out of 28 respondents to the online survey considered it “very useful” and eight rated it “useful.” This and Study Abroad were rated significantly higher than the other components. Current students also regarded English language training as effective. Fifty-two percent of surveyed students found the bridge year effective, compared to 47 percent who thought the summer courses were effective and 42 percent who saw the supplementary course as effective. The importance attributed to English and student interest in advancing their English language proficiency are reflected in improved International Testing Program (ITP) scores. Each student in Cohorts 1 and 2 sits for the ITP test three times during his/her participation in LOTUS (i.e., in the first year, after the second year, and before graduation). For Cohorts 3 and 4, the ITP test is conducted before and after the bridge year. Students who scored above 550 did not have to take summer courses. However, the student survey revealed that 27 percent of scholarship recipients “ sometimes” face challenges with the language, while 21 percent are “often” challenged and 13 percent “always” challenged. Conclusion: Improved English language proficiency is not directly correlated with students’ grade point averages (GPAs). Tracking the relationship between the two in three universities shows this to be the case. Findings and Supporting Evidence: Attempts to link improved English scores to improved GPAs were inconclusive. Although students’ average ITP scores improved over time for all cohorts and within all universities, average GPAs fluctuated over the same period. Hence, academic performance cannot necessarily be linked to the improvement in English language scores (see Annex VI, Part III for data tables and charts). Fluctuating GPA scores can be attributed to such reasons as unfamiliarity with the university system of course registration, students over-extending themselves with courses, problems with particular courses, or other factors beyond those at the university. Moreover, classroom instruction in the Science, Technology, Engineering, and Mathematics (STEM) fields at two universities contained a mix of Arabic and English. STEM students at PUA and nursing students at BUE reported in their group discussions that classroom instruction was sometimes given in Arabic, while slides on the board were written in English. English language proficiency was, therefore, a less important factor in student achievement in these fields than in others. The IIE team observed this as well.14 Conclusion: Strong English language proficiency increases students’ chances of studying abroad, and the Study Abroad experience, in turn, is associated with improved proficiency at graduation. 14 IIE Q1-2016 Report. 14 Findings and Supporting Evidence: English language competency is a primary requirement for participation in the Study Abroad component. According to IIE, it is also the principal roadblock to qualifying for travel. Students confirmed this problem as well as their frustration with the test preparation sessions in their discussions with the team. Students are frustrated by the rigidity of this requirement. However, IIE New York has said that US host universities believe LOTUS students are hampered by weak English language proficiency. At the same time, Study Abroad is correlated with improved English language competency. Conclusion: Students are dissatisfied with the two modes of delivery of English language training (i.e., supplementary and bridge year courses). Findings and Supporting Evidence: The methods of English language training differed for Cohorts 1 and 2 and Cohorts 3 and 4. The first two cohorts took intensive English language summer courses at BUE. Cohort 2 also received four supplementary courses (a total of 120 hours) at the American University in Cairo (AUC). Complaints about the timing of courses — in the summer when students wanted to return to their governorates, or after a long day of study, which involved travelling significant distances from their host universities to AUC in downtown Cairo — led to the introduction of bridge year courses for Cohorts 3 and 4. Summer courses continued to be offered on the basis of students’ Test of English as a Foreign Language (TOEFL) ITP test score. LOTUS continues to provide English language training for students. For example, 127 students attended English language courses in the summer of 2015 and 61 during the 2015 academic year.15 IIE is currently discussing dates and schedule with BUE for the 2016 Intensive Summer English program. In summer 2015, 30 students from Cohort 3 and 50 from Cohort 4 were enrolled in BUE’s Intensive Summer English Program. It is anticipated that 30 students from Cohort 4 will participate in the 2016 summer program.16 In order to compare bridge year students (Cohorts 3 and 4) with students who completed intensive English courses (Cohorts 1 and 2), the evaluation team computed the relative change in English ITP scores within the first year for all graduates and current students. Relative change is defined by the difference between the second and first ITP scores. The team found that the mean of the relative change in English proficiency test scores was significantly higher among students who took intensive English courses than among those who took bridge year courses (see Annex VI, Tables VI.8 and VI.9). However, there were different opinions about the English language training methods. Cohort 2 students thought the English curriculum and content at BUE were weak but more useful at AUC. BUE students stated that they were taught grammar they already learned and that they preferred to take English during the semester, not in the summer (so as to take a break). Finally, the timing of classes at AUC was particularly challenging for students who had to travel to the AUC campus in downtown Cairo from their universities located in the Fifth Settlement and Sixth of October City. 15 IIE Q3- 2015 Report. 16 IIE Q1- 2016 Report. “[The] timing of [the] AUC English course after [the] school day was very hectic and less beneficial, but there were no alternative options or times.” “It was very long. Two or three months would have been OK but I didn’t want to waste a year on the basics of English.” 15 All students in Cohorts 3 and 4 who participated in group discussions at BUE and AASTMT complained about the English language training during the bridge year. According to them, the content was weak, the sessions were long (five days a week from 9:00 am to 4:00 pm), and conversation was lacking. Some students in the BUE nursing program felt their spoken English was deteriorating from lack of practice in the classroom. They explained that, in their classes, content on the data show is usually presented in English but all other explanation is conducted in Arabic. SUMMARY TABLE EQ3: EXTENT TO WHICH STUDENTS SATISFIED WITH ENGLISH LANGUAGE TRAINING Findings 1. Students see English language training as an important component of LOTUS. They believe that improved English language proficiency will increase their chances in the job market. 2. Improved English language proficiency is not directly correlated with GPAs. Tracking the relationship between these two in three universities showed this to be the case. 3. Strong English language proficiency increases students’ chances of studying abroad. 4. Students are not satisfied with the two modes of delivering English language training (i.e., supplementary and bridge year courses). Conclusion 1. Scholarship recipients welcome the opportunity to learn English and recognize the importance of English language proficiency, but are not satisfied with the training methods introduced to date. Recommendation for IIE Action/To be Taken in the Near Future 1. From the students’ perspective, the provider and methods of instruction are key considerations. They recommend instruction through an interactive learning environment using the constructivism learning theory, the presence of native speakers, variations in tasks, and including such activities as English-language movies to make learning the language less tedious. Recommendation for Future Design 2. If the bridge year concept is to be repeated in future higher education scholarship initiatives, include a variety of activities throughout the year in addition to English language instruction. More time spent on soft skills training, introduction to university life and studies, and career counseling and practicums may be beneficial. Consider schedule variations, such as classroom English four days a week and other types of activities for the remaining days. US Study Abroad The goal of the Study Abroad component is for 121 (of 250) students to travel to the US for a semester (or summer session) at a US host university. The purpose is to encourage cross-cultural awareness and engagement. IIE is on track to achieve this target. To date, 65 students from Cohorts 1 and 2 have traveled to the US for a semester or summer term, 45 from Cohort 1 (20 females, 25 males) and 20 from Cohort 2 (11 females, nine males). The remaining students will be selected from Cohorts 3 and 4 (28 out of 50 students in each group). Conclusion: Cross-cultural exposure is the most important benefit of the experience for students. “The study in the US was an opportunity to discover future career path[s].” “It was my first time to do a presentation in English for foreign students. It made me more self-confident. It also was my first experience to do an experiment in the lab on my own.” 16 Findings and Supporting Evidence: Twenty-four of the 155 students surveyed participated in the travel abroad component. What half of them liked the most about the experience was the exposure to US culture. This was most often cited by participants in general, followed by “networking” and “practicing English.” Conclusion: Students said the experience enhanced their self￾confidence and leadership, improved their English, and strengthened their resumes. Findings and Supporting Evidence: Two graduates who traveled abroad emphasized that the opportunity improved their English and helped them get a job. English proficiency was slightly higher among those who participated in the US Study Abroad component than those who did not (see Annex VI, Table VI.12). The correlation between participation in US Study Abroad and employment among graduates was also analyzed. The difference among males does not appear to be significant, but female students who travelled abroad were more likely to be employed (see Annex VI, Tables VI.19).17 Thirteen of the 16 graduates who traveled to the US, rated the experience as “very useful” (4 on a four￾point scale) in developing their careers. In fact, this was the highest ranked of all the LOTUS components. In open-ended questions, students surveyed gave examples of how the experience had enhanced their self-confidence and leadership. These examples can be found in text boxes. Conclusion: Administrative records reflect a gender balance in the US Study Abroad component. Findings and Supporting Evidence: Among the 65 students from Cohorts 1 and 2 who traveled to the United States, 31 were women and 34 were men. This gender balance in the Study Abroad component matches the gender balance found in other LOTUS components. Conclusion: Despite the value of Study Abroad, several negative factors and complications were reported in implementation. Findings and Supporting Evidence: Students complained about a lack of transparency and information about the selection criteria for Study Abroad, especially for Cohort 3. In one university, students accused the committee of favoritism.18 The IIE/New York staff responsible for placing LOTUS students in US host universities explained that, while the LOTUS director in IIE/Egypt makes the final decision on who will travel, other factors affect university placements, such as costs and cost share, start and end dates, and course curricula. Relatively low TOEFL scores may also limit the options available in some fields of study. 17 While the difference by gender on this variable is statistically significant, the number of people in this category (female graduates who travelled abroad) is too small to allow additional analysis. The results could be affected by other factors besides Study Abroad. Also, this statistic measures a relationship but it is not a measure of causality. 18 This impression may reflect occasional assertions by LOTUS staff that poor behavior may affect a student’s chance of studying abroad. The students further explained that they were upset that some were allowed to retake the English exam to travel abroad, while others were not, and that the conditions in the testing rooms varied. “There I studied two management courses, and those courses helped me figure out my career path, which will be a mix of engineering and management.” Another: “After studying a nanotechnology course at X University, I decided to do my graduation project on applications of nanotechnology in the oil industry.” 17 US Study Abroad delayed the graduation dates of a few students, due to the non-transferability of course credits to their home universities. At BUE, each class of engineering students goes through the university as a group and have a set of required classes they all take together. If a student goes abroad, he or she loses a full year and must join the next cohort upon return. For this reason, BUE students travel during the shorter summer session and couple summer courses with an internship to meet the graduation requirement. Many students report that a semester is too short to reap the full benefits of the experience. SUMMARY TABLE EQ3: EXTENT TO WHICH STUDENTS ARE SATISFIED WITH US STUDY ABROAD Findings 1. Students believe the most important benefit of the Study Abroad program is cross-cultural exposure. 2. Students said the experience enhanced their self-confidence and leadership and improved their English and resume. 3. Administrative records show the program is gender balanced. 4. Although the Study Abroad component is considered valuable, several negative factors and complications mar implementation. Conclusions 1. Students who traveled abroad are enthusiastic about the personal and practical benefits of the program — enrichment, improved employability, and greater job options. Recommendation for Future Design 1. The US Study Abroad component is seen as fundamental to the entire program. However, competition for the limited number of opportunities is having negative effects on students who are rejected and on the overall LOTUS community. The evaluation team recommends that future study abroad components be expanded to include all scholarship beneficiaries and support a full year abroad rather than a semester or summer session. Internships Conclusion: LOTUS implementing partners consider internships, career counseling, and career development activities critical tools to enhance student employability. Findings and Supporting Evidence: All LOTUS students are required to complete at least two internships during their university career. Half the survey’s student respondents have completed internships. Most are in Cohorts 1 and 2. Cohort 3 and 4 students are now beginning to identify opportunities. Moreover, according to IIE’s third quarter 2015 report, an equal number of males and females (104 students) have interned, again demonstrating a gender balance in the internship component. The student survey also showed that 72 percent of the internships were relevant to the students’ fields of study. Conclusion: Seventy-nine percent of all internships to date have taken place outside of the students’ home governorates. Findings and Supporting Evidence: Of the 167 internships reported by students in the survey, 95 were in Cairo, 26 in Alexandria, and 11 were part of the Study Abroad experience. Some students explained during the group discussions that it is difficult to find internships in their fields, especially in logistics and biotechnology, and that these fields are almost completely unknown in their home governorates. The survey further revealed that 41 percent of internships were identified through the students’ own networks in Cairo and Alexandria, while the rest were obtained through IIE, their home universities, Nahdet el Mahrousa, and the Internet. 18 Conclusion: IIE confirms that students have faced difficulties in finding and reporting internships. Findings and Supporting Evidence: According to the IIE Monitoring and Evaluation (M&E) officer, students are reluctant to take on internships during their initial academic years because of the demands of their academic work. In the summer, they prefer to travel to their home governorates, so they do not take internships in Cairo or Alexandria even though IIE provides allowances for that purpose. Students say the allowance is generally insufficient to pay for accommodations and expenses during the summer, particularly since most university dorms are closed. Moreover, for the student to get credit for the internship, his or her employer must provide a written statement of compliance, which is often difficult to obtain. Finally, providers of internship contacts, such as Nahdet el Mahrousa, have no way of knowing when a student acquires a placement and, therefore, cannot follow up. Conclusion: Students report little knowledge of or benefit from the career counseling and mentoring component. Findings and Supporting Evidence: Only a few students were able to elaborate on the career counseling and mentoring sessions. This is partly due to the fact that only Cohorts 1 and 2 have taken part in those sessions thus far. However, IIE annual reports state that, in 2014, 32 career self￾assessments, 151 career counseling sessions, and 60 mentoring days took place. In 2015, 10 career self￾assessments, 169 career counseling sessions, and 21 mentoring days were held. Still, students express a desire for more practical assistance in this area. Their answers on the student survey point to a number of ways this assistance can be given: connecting students to recruitment offices, strengthening relations between students and graduates, and organizing awareness sessions about the labor market. Students in Cohorts 3 and 4 suggested expanding this component to include activities during the bridge year when students are making decisions about their fields of study.19 SUMMARY TABLE EQ3: EXTENT TO WHICH STUDENTS ARE SATISFIED WITH INTERNSHIPS Findings 1. LOTUS implementing partners consider internships, career counseling, and career development critical tools to enhance student employability. 2. Seventy-nine percent of all internships to date take place outside of students’ home governorates. 3. IIE reports that students face difficulties in finding and reporting internships. 4. Students report little knowledge of or benefit from the career counseling and mentoring component. Conclusions 1. Both implementers and students agree that internships are important to employability and career development, but issues of coordination have prevented full realization of these benefits. 2. The original intent of using internships to link students to their home governorates has not been accomplished. 3. Implementation of the internship component has been uneven. That is because it is dispersed among partner organizations without a central management and reporting point. It also relies on student initiative and employer sign-off. 4. Students report little knowledge of or benefit from the career counseling and mentoring component. Recommendation for Future Design Career Counseling and Mentoring: 19 See the discussion under Evaluation Question 4. 19 1. If the bridge year concept is to be repeated in future higher education scholarship initiatives, students recommend that career counseling begin during the bridge year to help them select a field of study. Recommendations for IIE Action/To be Taken in the Near Future Management and integration of the internship component: 2. Consider revising the definition of internship. The one-week requirement violates the intent of the experience. 3. Systematize the identification, reporting, and follow-up of internships. IIE should approve the internship up front, regardless of source. This will facilitate follow-up with the employer concerning compliance and with the student concerning the experience. 4. Compile and send a weekly list of internship opportunities to all students. This might require IIE to engage with other entities, such as Am Cham Egypt, AUC career center, AFS, etc. 5. Build a clientele through follow-up with companies to see if they are interested in more internships with LOTUS students in specific fields of study. Career Counseling and Mentoring: 6. Students recommend that career counseling focus should be on information about the labor market, work conditions, types of jobs, etc. in the various fields of study. It could include hands￾on information through job shadowing or a practicum. 7. Counseling should continue throughout the four years of study and address topics that are pertinent to seeking an internship, such as CV writing and interview skills and, later, employment search. 8. With the reduction in the number of universities and students, face-to-face counseling and small group mentoring is preferable to counseling via email and telephone. 9. Topics, such as career/life balance and gender roles, should be addressed. Student Life Conclusion: A large minority of students face challenges in dealing with other students at their universities. Findings and Supporting Evidence: Forty-four percent of the LOTUS students surveyed have encountered challenges interacting with or being accepted by non-LOTUS students. Ten percent say they are “always” challenged, while another 10 percent say they are “often” challenged. The remaining 24 percent report it being a problem “sometimes.” Importantly, more than half of LOTUS students in all universities find that “dealing with other students” is “rarely” a problem. Female students report such problems more often than male students, although no reasons were identified to explain this difference. Twenty-two percent of females and 15 percent of males say they “always” or “often” face this difficulty. In the graduate survey, 64 percent said it was rarely a problem and none saw it as a significant issue. Students in 8 of the 11 group discussions reported early bouts of homesickness. Three graduates recalled being homesick, but were more concerned with challenges in academic advice and instructional strategies. Conclusion: Problems with housing affect student integration into university life. 20 Findings and Supporting Evidence: Housing has been an issue since the beginning. University partners for Cohorts 1 and 2 do not have dormitories as many of their non-LOTUS students live in Cairo or Alexandria or have family in the area. Thus, they rent buildings in areas near campus. University coordinators cited problems with students as residents and with negotiations with landlords, since the universities do not directly manage repairs and upkeep. Housing and security is a central concern of parents, especially regarding their daughters. In nine of the 11 student discussions, the topic of housing elicited negative reactions. Only Cohort 1 and 2 students who are about to graduate seem to have put these issues behind them. According to the student survey, less than half (40 percent) of respondents “rarely” faced housing problems. Twenty-nine percent said they “always” or “often” had such difficulties. Even graduates (18 out of 28 respondents) identified housing as a challenge at least “sometimes” during their university years. Student concerns vary. Some are worried about being located in high-income areas, which affects their access to affordable food. Others noted that the rental buildings are ill-equipped to accommodate students as they have no study or visiting areas. Most rentals in new cities have weak internet connection and, in some buildings, there is no Wi-Fi service. Some students feel they are being observed constantly, lack privacy, and are subject to supervision they see as repressive. In one university, students said they feel isolated from the rest of the university as they are housed in separate buildings and transported in separate buses. Interviews with university coordinators highlighted the fact that university support is dependent upon available resources. In three universities, the coordinator has an assistant who manages logistics, including housing. Certain universities provide students with specific instructions on housing rules, while in another university, female students complain they do not know the rules and face difficulties as a result. Students across the universities consistently said they feel disregarded on the subject of housing. Conclusion: Communication between LOTUS students and university coordinators are a source of tension, mistrust, and student dissatisfaction. Findings and Supporting Evidence: Seventy percent of students surveyed do not address their problems with university coordinators. Rather, they turn to other students or academic advisors.20 They complain about the coordinators’ inefficiency and lack of responsiveness in dealing with their problems. For example, Cohort 3 and 4 students at BUE and AASTMT said their complaints, particularly about housing, were met with warnings that failure to abide by the rules would jeopardize their Study Abroad opportunity. This action was discussed intensively, as students consider this an inappropriate response when dealing with adults and potential leaders. Moreover, students reported sometimes receiving contradictory instructions from partner institutions, and being caught in the middle of miscommunication between their universities and IIE. Students have stated that face-to-face communication is essential and that current visits from IIE are insufficient. Both students and IIE coordinators confirm that most interaction with students is conducted via e-mail or telephone. One coordinator visits her assigned universities once a month, and for Alexandria students, once every other month; the other visits each campus twice a semester. Both coordinators further confirmed that they sometimes communicate problems (including discipline) to 20 See Evaluation Question 1. 21 parents but were unsure as to whether or not this should be done in agreement with the students. Students of both genders explained that calling their parents is inconvenient and asking parents to travel long distances for a meeting with IIE places undue burdens on the parents, unless they are being called for a serious matter, such as illness. SUMMARY TABLE EQ3: EXTENT TO WHICH STUDENTS ARE SATISFIED WITH STUDENT LIFE Findings 1. Forty-four percent of LOTUS students have encountered challenges in interacting with or being accepted by non-LOTUS students in their universities. 2. In their discussions, students said they had problems with homesickness. 3. Twenty-nine percent of currently enrolled students face housing challenges “always” or “often.” 4. Seventy percent of students do not to address their problems with university coordinators, but with other students or academic advisors. Students complain about coordinators’ inefficiency and lack of responsiveness in dealing with their problems. Conclusions 1. Communication among students, IIE, and universities are centralized in IIE and managed through a one-track communication channel focused on the university and IIE coordinators. Limiting students’ communication with IIE and universities to two individuals makes students feel isolated and disregarded. Students are told what to do (and reprimanded), without an effective voice in managing their lives. 2. A large minority of LOTUS students face challenges in dealing with other non-LOTUS students at universities. 3. Housing problems affect student integration into university life. 4. Communication between LOTUS students and coordinators are a source of tension, mistrust, and student dissatisfaction. Recommendations for IIE Action/To be Taken in the Near Future 1. Strengthen links between Cohorts 1 and 2 and Cohorts 3 and 4. Encourage graduates to meet with students socially, on a regular basis, to discuss their problems and serve as mentors. 2. Contract assistant LOTUS coordinators at AASTMT and BUE to be accessible to students and manage program logistics. Engage students in setting LOTUS program policies and eliminate threats and punishments that they consider humiliating. Expand opportunities to impart LIA lessons through interaction and reflection (e.g., retreats, workshops) outside the classroom setting. EVALUATION QUESTION 4 (EQ4): TO WHAT EXTENT DO SCHOLARSHIP RECIPIENTS GRADUATE WITH THE ACADEMIC AND SOFT SKILLS (WORKFORCE PREPAREDNESS) NEEDED TO WORK IN JOBS SUITED TO THEIR ACADEMIC PREPARATION? The LOTUS Program emphasizes human capital development and employability in order to meet Egypt’s labor market demands. Question 4 asks whether LOTUS students and graduates have these skills. Because this midterm performance evaluation is unable to address this issue directly, it examines students’ and graduates’ perceptions of their workforce preparedness. The final section addresses the question of whether LOTUS is producing results consistent with Egypt’s long-term social and economic development objectives, particularly, the effect of targeted fields of study. Broadly, LOTUS students and graduates are confident in their preparation for the workforce. It is uncertain how their training will affect long-term development goals. 22 Current Students’ Perceptions of Workforce Preparedness Conclusion: Students join LOTUS for employment and new opportunities. Findings and Supporting Evidence: Students see the LOTUS Program as an opportunity to improve their employability and to escape the path they would have followed if they had remained in their communities. In female-only discussions, women talked about how LOTUS expanded their options and loosened family control: “I would have been bored…,” “I would have had a harder life…,” “I told my mother, now I am free.” Student comparisons of the employment prospects in the various fields are a factor in choosing careers. Many also cite a strong parental role in deciding what they should study. In all their discussions, students said employment was a part of “life after graduation.” They also mentioned travel abroad, and men said they would delay marriage after graduation. Conclusion: Current students are confident that they will find a good job soon after graduation. Female graduates are less optimistic than male graduates, but there is no information as to why. Findings and Supporting Evidence: Eighty-one percent of the LOTUS students surveyed (83 percent male and 78 percent female) believe they are more likely than their high school peers to get a good job quickly. However, they are less confident about their earnings. Roughly half (56 percent male, 49 percent female) say they are likely to earn more money than their high school peers. Although most are optimistic about their job prospects in spite of scarce opportunities in Egypt, some students in logistics and biotechnology are pessimistic. Conclusion: Current students believe that academic achievement (high GPAs), soft skills training, and internships will contribute to their future success. Findings and Supporting Evidence: Cohort 3 and 4 students (in AASTMT and BUE) are particularly pleased with the soft skills training they had during the bridge year. Soft skills training included resume writing, presentation skills, and time management. Also, training in self-awareness, leadership, and personal interaction helped boost students’ confidence prior to the start of classes. Although mentoring and career counseling were mentioned infrequently in group discussions and scheduling was a major problem for this component, students still assessed their potential utility positively. All student discussion groups identified “building a network” as a benefit of LOTUS (and of Study Abroad), and as important in securing employment. Internships are an avenue to that network. Students’ had varied assessment of the value of internships. Graduates’ Employment Status and Assessment of Workforce Preparedness Conclusion: A majority of LOTUS graduates are employed. Those who are not are in residency, military service, further training, or graduate school. A fourth of recent graduates are unemployed and looking for work. Findings and Supporting Evidence: IIE maintains contact with LOTUS graduates online to document their current employment status. As of April 2016, 83 LOTUS students had graduated (48 females, 35 males) — two in late 2013, 15 in 2014, and 57 in 2015. Of that number, 39 (25 females, 14 males) are employed; 16 are either doing their residency, required military service, or post-graduate studies; 19 (13 23 females, 6 males) are unemployed and looking for work; and nine are missing data. Among the 74 for whom data is available, 53 percent are employed and 26 percent are unemployed.21 The IIE database further shows that most graduates work in Egypt and, specifically, in Cairo (at least 35 out of 39 employed graduates).22 Only two reside outside the country,23 and nine work as teaching assistants.24 Conclusion: Employed LOTUS graduates are relatively satisfied with their current employment and work in areas related to their fields of study. Findings and Supporting Evidence: The online survey measures graduates’ perceptions of their jobs and how the LOTUS Program affected their employability. Thirty-nine out of 83 graduates responded to the survey.25 Fifty-seven percent are employed and 23 percent not employed. In general, recent graduates assess the influence of LOTUS in the same way as current students, except they give greater importance to English language proficiency. Most who are or have been employed (28 graduates) found their first job within six months of graduation; ten were employed in less than a month. Another ten graduates are currently in or applying for post-graduate programs. Four are applying outside of Egypt. Nine (23 percent of respondents) say they are not employed but looking for a job. An additional eight persons (21 percent) are not working or searching for a job due to military service, paid training, graduate studies, and residency. LOTUS graduates found their current jobs through professors (27 percent), personal connections (41 percent), and social media (27 percent). Nearly two-thirds are satisfied or very satisfied with their current jobs. Conclusion: Graduates believe the LOTUS Scholarship Program improved their employability. Findings and Supporting Evidence: Three-fourths had interned as students but only five resulted in full-time employment. Two-thirds agreed, however, that internships provided practical experience about the work environment. Graduates rated English language training and volunteer and community service activities as the most useful among the non-academic components of LOTUS for their careers. Thirteen of the 16 graduates who studied abroad said it was “very useful.” The components ranked lowest were service learning and career planning and counseling. A majority believe the LOTUS Program was important to their chances of getting a job after graduation. Thirty-one percent said it was “very important,” while 55 percent said it was “important.” For career advancement, 28 percent considered it very important and 45 percent “important.” Both graduates and students explained that while LOTUS provides tools to improve employability, success on the job depends on how the person applies the tools. Women did not detect a gender bias in hiring, though they believe it may have a negative effect on their career advancement. 21 Data on graduates come from the IIE Employability Database, the online graduate survey, and a group discussion with four female graduates in Cairo. The latter included all women because none of the men who were invited attended. 22 The IIE database includes the name of the employer but not the location. An examination of the locations of these employers showed that 35 are located in Cairo. Four cases could not be determined, that is, a teacher in a school, a pharmacist in a pharmacy, Vodafone N. Sinai, and an NGO headquartered in Cairo but working nationwide. 23 It is possible, though unknown, that some of those categorized as missing data (n=9) are located outside of Egypt. 24 In Egyptian universities, Teaching Assistant is a career track position usually offered to the top graduates in the faculty. 25 Due to an oversight in administration of the online survey, no demographic information is available for disaggregation. 24 Conclusion: Implementing NGOs and graduates emphasize that the merit-based nature of the LOTUS scholarship increases student chances for employment. Findings and Supporting Evidence: Graduate group discussions and NGO interviews highlighted the importance of the resume and the merit-based nature of the LOTUS scholarship as advantages in getting a job. Graduates tended to find employment quickly. Twenty-six found their first job within six months of graduation. Ten were employed in less than a month. SUMMARY TABLE EQ4: PERCEPTIONS OF WORKFORCE PREPAREDNESS Findings 1. Current LOTUS students are confident that LOTUS will improve their prospects of getting a good job soon after graduation. 2. Current students attribute their anticipated success to academic achievement and to soft skills training. Networking (primarily through internships) is also important. 3. A majority of LOTUS graduates are employed. Those who are unemployed are in residency, military service, further training, or graduate school. A fourth of the graduates are unemployed and looking for work. 4. LOTUS graduates say the LOTUS scholarship and its components have improved their employability. 5. Implementing NGOs and graduates emphasize that the merit-based nature of the LOTUS scholarship increases students’ chances for employment. 6. The majority of LOTUS graduates have been successful in entering the labor market in areas related to their fields of study. A disproportionate number have continued in academia as teaching assistants and/or are pursuing post-graduate studies. They see LOTUS as less important for career advancement. Conclusions 1. Students join LOTUS for employment and new experiences. Employment options are a factor in their decision to join LOTUS and in their selection of fields of study. 2. Students are oriented toward employment and careers. 3. Employed graduates are relatively satisfied with their current employment and work in their fields. Recommendation for IIE Action/To be Taken in the Near Future 1. Encourage participating universities to consider providing support services to recent graduates to assist them in their early job search and career development. Recommendation for Future Design 1. In future designs, consider establishing a financing or grant mechanism to provide loans or grants to assist recent graduates with specific costs associated with job searches; an incentive fund for start-up businesses or professional practices in students’ home governorates; and/or start-up seed money to fund graduate/student community service projects that benefit scholarship recipients’ home governorates and/or for incentivizing start-up businesses or professional practices in home governorates. The mechanism also could be used to establish links to other USAID activities focused on workforce development and/or employment generation. Other Salient Observations Underlying Issues Associated with Field of Study Conclusion: Students are concerned about scarce employment opportunities in Egypt in general, and the even fewer prospects in their home governorates especially in certain fields. These fields include biotechnology, logistics, and nursing (except in private hospitals), despite their being in high-demand growth industries and contributing to Egypt’s economic competitiveness. 25 Findings and Supporting Evidence: In designing LOTUS, USAID/Egypt and IIE emphasized that the fields of study available to students be those “in high demand and critical to both Egypt’s sustained economic growth and the development of an internationally competitive workforce” (IIE Cooperative Agreement Award). The fields of study available for Cohorts 1 and 2 included careers that are widely desired by students and parents in Egypt (engineering and pharmacy) and in demand in the labor market (business, computer science, communications). Recruiters and students indicated that these options were important to accepting the LOTUS scholarship.26 Ninety-five percent of surveyed students selected their own fields of study (i.e., it was one of their three choices). Nine students said it was assigned to them. Four in Cohorts 1 and 2 changed fields after starting university. Students in all discussions addressed the problems they faced getting into an appropriate field of study. Many said they had to negotiate with their parents. IIE even intervened occasionally with parents on behalf of students. Some students chose unfamiliar fields (especially computer science and logistics). Others from previous cohorts discussed how their aspirations changed over the course of the program. They anticipate eventually moving into other fields. The situation changed for Cohorts 3 and 4 when the number of universities was reduced from five to two (i.e., BUE and AASTMT). Moreover, popular five-year fields of study (engineering, dentistry, pharmacy, and physical therapy) were excluded as options because, with the addition of the bridge year, they could not be completed before the project end date, and because IIE identified them as saturated. Doing away with these options affected recruitment because, as mentioned in student discussions and corroborated in interviews with recruiters, high school graduates and their parents (especially) often have clear ideas about what to study. Hence, some students did not apply for LOTUS scholarships. Others sought options elsewhere in medicine. At this point, the relationship between participation in LOTUS and employment is very weak because there are relatively few graduates. The information available is that students from some fields (basically engineering) are finding employment, but it is not clear if this pattern will hold up with a more robust sample or how it will look relative to other fields from which there are still no graduates. Other options (such as, art and design) also disappeared when MSA dropped out and AASTMT joined LOTUS during the Cohort 3 bridge year. Students who were going to enter MSA were transferred to either BUE or AASTMT, where the names and types of careers were different and, in some cases, unknown. Nursing, despite its lack of prestige as a career, was identified as a high demand occupation and incentives were offered in the selection process for students to select this option. These incentives mean that nursing students in Cohort 4 are not permitted to change after enrollment. There is no evidence at this point about how these programmatic changes will affect future employment in the short- or long-terms. The primary effect on LOTUS has been the loss of potential applicants. Current students have suggested including career development and counseling in the bridge year rather than final years to help address uncertainty about the various fields. They are also worried about future employment opportunities in Egypt (and in their home governorates) in such emerging fields as logistics. Nursing students are concerned that they can only work in private hospitals and clinics. Conclusion: Employment data show differences in employability by field of study. Biotechnology graduates have a lower rate of employment than those in other fields. 26 According to IIE and recruiters, the main reason given by students who did not apply for or accept the scholarship when offered was the absence of their preferred career choices, such as medicine and engineering. The latter was later excluded as a study option. 26 Findings and Supporting Evidence: The IIE graduate employability database confirms students’ assessments of the job market. Five of the seven biotechnology graduates are unemployed whereas none of the business graduates are unemployed. Three out of ten engineering graduates are unemployed. SUMMARY TABLE: UNDERLYING ISSUES WITH FIELDS OF STUDY Findings 1. Students are concerned about scarce employment opportunities in Egypt and in their home governorates specifically in the biotechnology, logistics, and nursing (outside of private hospitals) fields. This, despite the fact that these careers are in high-demand growth industries and contribute to Egypt’s economic competitiveness. 2. Employment data show differences in employability by field of study. Biotechnology graduates have a lower rate of employment than those in other fields. Conclusion 1. The emphasis on emerging and cutting-edge fields of study is causing dissatisfaction and concern among students. This focus is inconsistent with long-term objectives to stimulate economic growth in Egypt and in students’ home governorates. There is no data at this point to measure the individual-level impact of the choice of these careers, although preliminary information on new graduates suggests some barriers in the job search. Observations about Gender in the LOTUS Program Conclusion: The gender parity requirement in LOTUS has been an effective method to ensure equal opportunities for young men and women. Gender parity has been achieved for the program as a whole and within each cohort (see Annex VI, Table VI.1). To the extent it was not achieved, it benefitted females. The principal imbalance occurred in Cohort 2, but there is no specific explanation for this result. These results were attained without specific actions to ensure equity. All students, male and female, were selected from a single pool of applicants, the same criteria were applied to both, and gender in and of itself was not a factor in selection. In an interview, IIE suggested that there were more qualified women overall than men among the candidates. An implementing partner commented on the importance of the parity requirement in LOTUS in terms of the perception of legitimacy. One recruitment partner observed that young women in remote areas are interested in participating in LOTUS, despite traditions that girls should stay close to home, that priority should be given to educating sons not daughters, and that girls should dedicate themselves to home and family. The reason is that not all areas outside the cities are equally conservative. New technology has given both females and males much more access to the outside world than in the past. Furthermore, many parents want to improve the lives of their daughters. Of course, there are very conservative families who do consider the possibility of a scholarship and do not attend the informational sessions. Isolated girls may be less likely to learn about or trust the scholarships. To reach such girls, recruiters relied upon trusted religious leaders or older students. They also allayed parents’ concerns and provided young people with the information needed to convince their parents. What does LOTUS mean to students? Both males and females used terms like opportunity and independence and control over their own lives. Females, in particular, described how LOTUS gave them an alternative to what would have been a very predictable and conforming lifestyle. One tentative observation from the quantitative data was that young women who traveled abroad are very successful in the job market. However, random comments in the discussions suggest that parental controls and 27 family ties are constraints on their lives. For example, one woman who wished to go abroad said she had to first find a relative there. Another noted that she can now work in Cairo because her family has moved there. Finally, one female graduate had to leave her job in Cairo because her father insisted she return home. The gender parity achieved in scholarship awards is also observed in all components and activities of the LOTUS Program. Quantitative indicators and group discussions show no substantial gender-specific differences in the experiences of male and female students. This absence and general lack of discussion of gender issues could be interpreted as a sign that males and females do, in fact, interact and compete with each other as equals in the LOTUS context. The evaluation data do not provide further insights on this topic. CHALLENGES Coordination The overriding challenge for the LOTUS Program is coordination and management in scheduling and program content (e.g., LIA and English language components). These problems cause duplication of effort (e.g., similar topics are presented by various partners in LIA presentations) and a lack of knowledge among partners about the LOTUS Program as a whole (e.g., university coordinators know LOTUS students have required non-academic activities but do not know what they are; all partners are involved in internships but do not work together). Centralized decision-making and control of information further contributes to the perception among students and partners that there is a lack of transparency. A second factor is setting priorities and criteria for judging the relative value of the various activities (lectures, workshops, LIA sessions, supplemental English courses, etc.) that students are expected to attend and absorb. Students feel overwhelmed by university and LOTUS demands, the lack of clear priorities, and sporadic scheduling. IIE staff have said they will add unanticipated topics or activities they believe are important (e.g., a current event). Students do not have the information needed to manage their time effectively. For example, IIE policy requires students to give notice three days in advance if they will not attend a particular event (they are allowed two excused absences per year). However, students maintain (and LIA managers confirm) that sometimes they do not have scheduled information to make those decisions. Communication Communication is centralized in IIE and managed through a one-track communication channel between university coordinators and IIE academic coordinators. This control causes misinformation among partners, the potential for manipulation, and dissatisfaction among students. Limiting students’ communication with IIE and the university to two individuals makes them feel isolated and disregarded. Students are told what to do (and reprimanded) without any effective voice in managing their lives. As a result, many, particularly in Cohorts 3 and 4, view the situation in terms of “us against them,” “them” being the coordinators. The recommendation is to strengthen and formalize the involvement of graduates with new students. 28 Small Private Universities The five universities involved with Cohorts 1 and 2 are relatively new (i.e., founded within the past 10 to 15 years). Although the facilities are impressive (particularly, laboratories and equipment), faculty development and student organization and activities are still works in progress. LOTUS students are aware of (and have given voice to) shortcomings where they exist and, in some cases, have taken steps to overcome them (e.g., by enrolling in supplemental online courses, or taking the initiative to start a student union). The hiring of LOTUS graduates as teaching assistants is evidence of this development process. Additionally, none of the five original universities has dormitories and rely on rental property to house non-local students (students from out of town are the exception in the student body as a whole). Housing has been and continues to be a significant challenge for students, parents, and universities. In most cases, the solutions have been fraught with problems (e.g., student feelings of mistreatment and isolation, and extra-ordinary control and property management costs for universities. Study Abroad Challenges in implementation of the Study Abroad component have been outlined in Question 3. First, the most formidable challenge has been the level of English language proficiency among LOTUS students. Even students with the required skills struggle in US universities, particularly in the specialized fields of engineering or biotechnology. Second, the medical fields in LOTUS (i.e., dentistry, pharmacy, and physical therapy) are undergraduate programs in Egypt but graduate programs in the US. Egyptian students, therefore, do not have the qualifications to enroll and succeed in those courses in the US. Third, BUE uses the British system of scheduling and courses, which is not compatible with the US system. In order for LOTUS students at BUE to study abroad they must travel during the summer and enroll in the reduced number and variety of courses offered at US universities during the truncated summer session. These courses are often not related to the students’ fields of study. The requirement that BUE engineering students do internships while in the US further complicates the placement process. Finally, both students and IIE/New York staff involved in placements have commented that the short-term placements for LOTUS students causes coordination problems and do not allow students to fully benefit from the experience. In Conclusion Overall, LOTUS has an impressive achievement record in identifying and guiding highly capable students through quality higher education, skills development, and new opportunities to improve their lives and community. The gender parity requirement is also an effective method to ensure equal opportunities. IIE recognizes and has taken constructive steps to meet emerging challenges. This evaluation provides actionable recommendations to IIE to improve program effectiveness for the remaining period of implementation and to USAID/Egypt to enhance future higher education scholarship programs. REFERENCES For a comprehensive list of references, please see Annex IV, Sources of Information. 29 ANNEXES 30 ANNEX I: EVALUATION STATEMENT OF WORK Leadership Opportunity Transforming University Students (LOTUS) Scholarship Program Midterm Performance Evaluation 1. Activity Background Information A. Activity Identifying Information Award Title: Leadership Opportunity Transforming University Students (LOTUS) Scholarship Activity Cooperative Agreement: 263-A-00-10-00026-00 Total Estimated USAID Amount: $23,735,013 Start date: 2010-05-04 End Date: 2019-9-30 Program Manager: Mary Ishak Evaluation Program Manager: Hanan Abbas Implementing Partner: Institute of International Education – Egypt (IIE) Governorates of Implementation: Nationwide recruitment and selection Universities of implementation: • Ahram Canadian University (ACU) – Sixth of October City • British University in Egypt (BUE) – Shorouk City • Future University in Egypt (FUE) – New Cairo • Modern Sciences & Arts University (MSA) – Sixth of October City • Pharos University in Alexandria (PUA) – Alexandria • Arab Academy for Science, Technology & Maritime Transport (AASTMT), Alexandria Branch. B. Background The Egyptian higher education system continues to be challenged by inequitable access, poor quality, and gross inefficiencies. It is plagued by its poor reputation for producing graduates lacking in expert thinking and complex communication skills much needed in a knowledge based economy. The Government of Egypt and Egyptian policymakers recognize that a serious reform effort is needed to improve the higher education system in order to stay competitive in this global society. In support of the GOE’s efforts to reform their education systems, USAID/Egypt has over the years provided significant support to primary and higher education in Egypt, in the areas of school based reform, institutional support and scholarships. In Egypt, the quality of private education is reputed to be of higher than public education, with the gap decreasing in the secondary stage and then increasing tremendously at the higher education stage. This results in increased gaps between the more economically advantaged youth and their poorer counterparts who cannot afford quality private education. These gaps are manifested in significant differences in graduates’ employment potential for more productive and profitable jobs, which favor the more economically advanced part of the student population. Within this context, USAID/Egypt’s past scholarship programs and this current program aim to 31 address this gap and develop the potential of the economically disadvantaged to contribute to the development of Egypt. As part of this support, on May 4, 2010, USAID/Egypt awarded a cooperative agreement for $9,759,921 to the Institute of International Education (IIE) to implement the New Scholarship Program through September 30, 2016. The program gives Egyptian students an opportunity to attend private universities that would normally be above their families' financial means and creates a network of youth nation-wide who are well-educated and passionate about Egypt. The program initially aimed to award 100 scholarships to provide undergraduate scholarships for promising Egyptian students with high financial need, building on the lessons learned and best practices of previous scholarship programs. Since project inception, three additional cohorts (50 students each) were added to the program, thereby increasing the award’s Total Estimated Cost to $23,735,013 and extending the completion date of the agreement to September 30, 2019. To date, 250 scholarships have been competitively awarded to male and female applicants from each of Egypt's governorates, with a focus on geographic and gender diversity as well as diversity of experiences and desired fields of study. Students are able to choose from an array of study options that are in high demand and critical to both Egypt’s sustained economic growth and the development of an internationally competitive workforce such as biotechnology, physical therapy, engineering, nursing, business administration and many others. The program provides Egypt’s young people with leadership opportunities through academic courses, leadership enrichment programs, career development opportunities, and community-based service. C. Goal The ultimate goal of the LOTUS Scholarship Activity is to support development in Egypt through training a cadre of future leaders. This will be achieved by providing high quality undergraduate education to a limited number of promising Egyptian students with high financial need from all over Egypt. D. Development Hypothesis This activity conforms to the development hypothesis that increased scholarship opportunities for targeted youth will contribute to a better educated workforce that responds to Egypt’s labor market needs. E. Project Description The LOTUS Scholarship Program offers comprehensive scholarship packages to students with outstanding academic and extra-curricular credentials and high financial need to obtain undergraduate degrees from Egyptian private universities in fields of studies that are important to Egypt's development. The program provides Egypt’s young people with scholarship and leadership opportunities through academic courses, leadership enrichment programs, career development opportunities, and community-based service. Since 2010, 250 scholarships have been competitively awarded to male and female applicants from all of Egypt's 27 governorates with a focus on geographic and gender diversity, as well as diversity of experiences and desired fields of study. This program gives Egyptian students an opportunity to attend private universities that would normally be above their families' financial means and creates a network of youth nation￾wide who are well-educated and passionate about Egypt. 32 F. Project Objectives The project objectives include to: • Identify and empower young women and men who have demonstrated academic excellence, leadership, and involvement in their communities; • Give Egyptian students an opportunity to attend private universities that would normally be above their families' financial means; • Develop and nurture the recipients' leadership potential, skills sets, and commitment to community and country, so that they are prepared and equipped to become future leaders and advocates for development in their local communities; • Enhance the recipients' employability and career options; and • Create a network of youth nation-wide who are well educated and passionate about Egypt. G. Main Activities Examples of main activities include: • Enabling students to choose from an array of study options that are in high demand and critical to Egypt's sustained economic growth and the development of an internationally competitive workforce. • Implementing a Leadership in Action Program that engages students in a series of learning opportunities that complement the academic and extracurricular activities available through each university. • Building the capacity of the partner universities through leadership development of key university administrative staff to embed an enriched understanding of leadership principles to institutionalize the Leadership in Action Program. • Making US Study Abroad opportunities available to 121 students out of 250 students to encourage cross-cultural awareness and engagement. • Providing internship opportunities, career counseling and student development services. H. Results to Date Results include: • Four competitive, nationwide outreach, recruitment, and selection cycles (summer 2010, summer 2011, summer 2013, and summer 2014) have resulted in 250 students (112 Males and 138 Females) being selected to receive LOTUS Scholarships to one of six private universities. • 73 graduates from Cohort 1 and 2: 57 Cohort 1 and 16 Cohort 2 (43 Females –30 Males) • 65 students from Cohort 1 and 2 had a US Study Abroad opportunity (semester or summer term): 45 Cohort 1 (20 Females – 25 Males) and 20 Cohort 2 (11 Females – 9 Males) • 104 students had internship opportunities: (52 Females – 52 Males). I. Gender Considerations Throughout the program, LOTUS has aimed to ensure gender equity and geographic distribution targeting students nationwide. In Cohorts one and two, IIE worked on awarding scholarships to a male and a female student from each governorate to allow for an equal gender balance and full 33 country coverage (i.e. select from each of Egypt’s 29 governorates later 27 governorates). IIE tried to strike an equal number of scholarships between males and females. Students would be selected across governorates, and the priority is to support students with high financial need. In Cohorts three and four, the strategy slightly changed as IIE worked with USAID to award LOTUS Scholarships with preference given to the most financially needy applicants from 12 underserved governorates but still in consideration of the need for gender balance and full country coverage (i.e. select from each of Egypt’s 27 Governorates). There was a specific requirement in Cohort 3 to select a minimum of one female and one male student from each of the identified underserved governorates and award at least two male and two female students from both North Sinai and South Sinai. During the research phase for the Cohort 3 proposal, IIE consulted a variety of reports and statistics including the Egypt Human Development Report (2010), the CAPMAS Statistical Yearbook (2013), and the Poverty Assessment Update for the Arab Republic of Egypt (2007), all of which address issues or provide statistics on the criteria for selection of underserved governorates. Underserved governorates are those that have a combination of characteristics such as: low economic and literacy levels, high student drop-out rates, preference to educate males over females, no or few educational institutions, under-represented communities/areas and political instability. Based on these studies and taking into consideration various other factors including political stability (or instability), USAID and other donor activities in governorates, access to higher education institutions and past experience during LOTUS outreach and recruitment, IIE identified the following 12 governorates as underserved governorates for outreach, recruitment and selection purposes for LOTUS Cohort 3 & 4: • Assiut • Aswan • Beheira • Beni Sueif • Fayoum • Marsa Matrouh • Minya • North Sinai • Qena • Sharkiya • Sohag • South Sinai Although the frontier governorates of North Sinai, South Sinai and Marsa Matrouh were not ranked in the Human Development Index, all three governorates face a variety of development challenges including low literacy and school enrollment rates, high drop-out rates for girls, high unemployment and low financial levels. North Sinai has the added challenge of significant political instability as a result of its proximity to Gaza, as well as smuggling and extremism. These governorates have not historically been recipients of donor funding or attention and have limited access to higher education institutions. The four rigorous recruitment, outreach and selection efforts resulted in selecting 250 LOTUS students: 112 males and 138 females. 34 J. Linkage with USAID/Egypt Assistance Objectives The Activity contributes to the Development Objective # 3: Workforce Response to Labor Market Demands Improved and falls under the Intermediate Result # 3.1: Access to Quality Tertiary Education Increased. K. Modifications A number of modifications (MOD) have occurred over the course of the agreement, including: • MOD#1(6/28/2010): The purpose of this modification was to revise program description to include approved branding strategy and marking plan and to change key personnel. • MOD #2(5/24/2011): The purpose of this modification was to revise the reporting and evaluation section and revise the program description. • MOD#3(5/24/2011): The purpose of this modification was to revise the agreement budget by adding a separate budget for an outreach, recruitment, screening, and selection plan nationwide in anticipation of adding Cohort 2 of 50 additional students from 27 governorates subject to fund availability. Increase the total estimated amount by $33,099 from $9,759,921 to $9,793,020. • MOD#4 (8/25/2011): The purpose of this modification was to add a new cohort to this agreement to be named as Cohort II for the period 2011-2016; to revise the agreement budget by adding a separate budget for Cohort II; to increase the total Obligated Amount by $1,000,000 from $9,793,020 to $10,793,020; to increase the Total Obligated Amount by $1,000,000 from $9,793,020 to $10,793,020; and to increase the Total Cost-Sharing Amount by $852,455 from $2,052,812 to $2,905,267. • MOD#5(2/14/2013): The purpose of this modification was to fully fund the balance of the subject cooperative agreement thus raising the obligated amount by $3,463,803 from $10,793,020 to $14,256,823. • MOD#6 (4/9/2013): The purpose of this modification was to add Cohort 3 for the period 2013-2018; to add separate budget for Cohort 3; to increase the total estimated amount from $14,256,823 to $19,050,188; to increase the total obligated amount from $14,256,823 to $19,050,188; and to increase the cost sharing amount from $2,905,267 to $3,491,767. • MOD #7 (4/16/2014): The purpose of this modification was to re-align budget for LOTUS Cohort 1 and 2; to revise the program description for Cohort 1 and 2; to decrease the total obligated amount for Cohort 1 and 2 from $14,256,823 to $14,256,611; to decrease the total estimated amount for Cohort 1 and 2 from $14,256,823 to $14,256,611; and to include Cohort 3 at total estimated amount of $4,793,365. • MOD #8(6/19/2014): The purpose of this modification was to add a new cohort to this agreement to be named as Cohort 4 for the period 2014-2019; to revise the agreement budget by adding a separate budget for Cohort 4; to revise the program description by including Cohort 4; to increase the total estimated amount from $19,049,976 to$23,735,013; to increase the total obligated amount from $19,049,976 to $19,549,976; and to increase the total cost sharing amount from $3,491,767 to $4,089,417. • MOD#9 (3/3/2015): The purpose of this modification was to fully fund the balance of the subject Cooperative Agreement thus raising the obligated amount by $4,185,037 from $19,549,976 to $23,735,013. 35 2. Evaluation Rationale A. Purpose The purpose of this midterm performance of LOTUS is to provide USAID/Egypt with information to help improve the performance of LOTUS and its contribution to USAID/Egypt’s development objectives (i.e.: Workforce response to labor Market demands improved and falls under the IR 3.1: Access to Quality Tertiary Education Increased). The results will provide information critical to understanding the program’s efficacy and relative importance to the higher education portfolio, as well as in making programmatic decisions over the remaining implementation period. B. Audience and Intended Uses The audience for the evaluation will be USAID/Egypt, specifically the Education Team within the Office of Education and Training, the implementing partner (IIE), the LOTUS recipients, participating universities, and USAID/Washington. The evaluation results will be shared with other stakeholders, other donors, and education NGOs. The report should be made accessible to the public via USAID’s Development Experience Clearinghouse (DEC) within three months of report completion. 3. Evaluation Questions This section presents the fundamental questions that the evaluation will answer. The specific evaluation questions, in terms of priority, are: 3.1. To what extent does available evidence suggest that the project is on track to achieve its objectives? 3.2. To what extent has the LOTUS Program enabled recipients to contribute to development, community service, and leadership activities? 3.3. To what extent are scholarship recipients satisfied with the academic related and all the other program components like: Study Abroad, English language training, leadership in action activities, career counseling, housing, university coordinators, etc.? 3.4. To what extent do scholarship recipients graduate with the academic and soft skills (workforce preparedness) needed to work in jobs suited to their academic preparation? 4. Evaluation Design and Methodology A. Evaluation Design This is a midterm performance evaluation and is intended to focus on how LOTUS is implemented, what the Activity has achieved to date, whether expected results were attained according to the Activity design and underlying development hypothesis, and how activities were perceived and valued by stakeholders. 36 The evaluation team will use a mix of quantitative and qualitative data collection and analysis methods to answer the evaluation questions. The evaluation must follow the principles and guidelines for high quality evaluations outlined in the USAID Evaluation Policy (January 2011) (http//www.usaid.gov/evaluation/policy). B. Data Collection Methods The evaluation team should consider a range of possible methods and approaches for collecting and analyzing the information that is required to address the evaluation questions. The evaluation team shall share data collection tools with USAID for review, feedback and/or discussion with sufficient time for USAID’s review before they are applied in the field. The USAID will approve a formal design report. The data collection methodology will include a mix of tools appropriate to answer the evaluation questions. This may include document review, on line and on campus surveys, key informant interviews and focus group discussions. The evaluation team will determine the sample of students that are going to be included in each sample by data collection method out of the 250 beneficiaries (students and alumni), aiming to interview as representative of a cross-section of students as possible. The selection of beneficiaries to be interviewed will be based on the diversity of different universities, academic programs, gender, and geographic location as well as practical considerations for completion of the evaluation. The team will conduct a desk review of all resources sent to them prior to coming to Egypt and during the Team Planning Meeting. Proposed potential data collection sources and methods may include: a. Document review of relevant USAID/Egypt and project documents provided by USAID; b. Key informant interviews with IIE, participating university officials, employers; c. Online survey of alumni and on-campus survey of current beneficiaries; and d. Focus group discussion with alumni and current beneficiaries (disaggregated by sex); These are described in further detail as follows. Document review – The evaluation team will collect and review all relevant project documents from USAID, IIE, and other sources identified during the evaluation. USAID and the implementing partner will provide the evaluation team with relevant soft copies of a package of briefing materials, including: program agreement and amendments, quarterly and annual reports, budget information (as relevant and appropriate to be shared), the Performance Monitoring Plan, and an audit report. The evaluation team should complete as much of the document review prior to arriving in Egypt and during the first week in Egypt. Key Informant Interviews – The evaluation team will conduct key informant interviews with the officials of IIE, USAID, the administration of recipient universities and some of the current student employers. Online and On-campus Surveys – USAID expects the evaluation team to conduct an online survey of LOTUS alumni and on- campus survey for LOTUS current students. The evaluation team will have to determine the sample size of students for the on-line and on-campus surveys. 37 IIE has a data base of LOTUS alumni and LOTUS current students. USAID will make sure that IIE provides the evaluation team with the lists of students and stakeholder contacts. Focus Group discussions – The evaluation team will determine the sample size of students and conduct focus group discussions with LOTUS alumni and current recipients in all 6 universities. The qualitative interviews will provide evidence of “how” the LOTUS Activity works and will support the quantitative data collected through the proposed online and on-campus surveys. A list of interviewees and key stakeholders will be provided by USAID prior to the assignment’s inception. Prior to beginning field work, the evaluation team must submit for USAID review and approval an evaluation design matrix that details the proposed methodology for data collection and analysis. The evaluation team is also required to share data collection tools with the USAID Evaluation Program Manager for review, feedback and/or discussion, with sufficient time for USAID’s review, before they are used in the field. USAID encourages feedback on these proposed methods and is open to the suggestions from the evaluation team on alternative approaches that will provide the highest quality evidence and most effectively answer the evaluation questions. To facilitate analysis, the data will, to the extent feasible, be collected and reported in a way that enables disaggregation across multiple dimensions. These dimensions include, but are not limited to, sex, academic discipline, degree type, institution, and sector of employment. C. Relevant Documentation USAID and the implementing partner will provide the evaluation team with soft copies of a package of briefing materials, including: a. Activity’s agreement and modifications. b. Project’s design documents and modifications c. Activity’s quarterly and annual reports d. Budget information as relevant and appropriate to be shared e. Activity’s Performance Monitoring Plan f. Audit findings of the USAID/Egypt’s LOTUS Activity g. Contact information for key informants h. Current student and alumni database The evaluation team should complete the document review prior to arriving in Egypt. The evaluation team may also request and review additional resources to the extent necessary to perform its work. D. Data Quality Standards The evaluation team must ensure that the data they collect clearly and adequately represents answers to the evaluation questions, is sufficiently precise to present a fair picture of performance, and is at an appropriate level of details. E. Data Analysis Plan Prior to the start of data collection, the evaluation team will develop and present, for USAID/Egypt review and approval, a data analysis plan which will emphasize the unit of analysis; for example, the LOTUS alumni, current LOTUS students, and university officials. The unit of analysis will also be disaggregated by gender, university, and geographic area. The evaluation team will explain how focus group interviews and surveys will be transcribed and analyzed, and 38 how the qualitative data from the focus group discussions and in-depth interviews with the key informants and other stakeholders will be integrated with quantitative data from the different relevant documents to reach conclusions about the effectiveness of the LOTUS Program. F. Data Limitation USAID expects that all issues affecting validity be discussed and documented during evaluation planning. Measures to mitigate these issues will be addressed with all team members and USAID in the implementation phase and detailed in the final report. 5. Evaluation Products A. Deliverables Evaluation Team Planning Meeting. A team planning meeting must be held in Egypt at the onset of the evaluation. This meeting will allow USAID/Egypt to discuss the purpose, expectations, and work plan of the assignment with the evaluation team. In addition, USAID/Egypt and the evaluation team must: • Finalize team members’ roles and responsibilities; • Review and finalize evaluation questions; • Review and finalize the evaluation timeline; • Present and discuss data collection methods, instruments and tools, analysis, and guidelines; • Review and clarify any logistical and administrative procedures for the assignment. A well-written, detailed methodology and data analysis plan (including an evaluation design matrix, data analysis plan, and evaluation work plan) must be prepared by the evaluation team and discussed with USAID during the planning meeting. USAID will provide the evaluation team with a stakeholder analysis that includes an initial list of interviewees, from which the evaluation team can work to create a more comprehensive list. The evaluation team will construct a preliminary interview schedule that includes different stakeholders, and then share with USAID the updated lists of interviewees and schedule as meetings/interviews take place and informants are added to/deleted from the schedule. The finalized list must be sent to USAID no later than five working days after submission of the preliminary interviewees’ schedule. Debriefing with the USAID and Implementing Partner. After five working days of conducting the field work, the evaluation team must present its preliminary findings to USAID/Egypt and to the implementing partner. The debriefing must include a discussion of findings, including project achievements and challenges, as well as preliminary recommendations for the future activity designs and implementation. The team must consider any USAID/Egypt comments, as appropriate, when revising the draft evaluation report. 39 Draft Evaluation Report. After one week of the debriefing presentation and prior to departing Egypt, the evaluation team must submit a draft report of the findings and recommendations to the USAID Evaluation Manager. The written report must clearly describe findings, conclusions and recommendations for future programming. USAID will provide written comments on the draft report within 7 to 10 working days of receiving the document. Final Evaluation Report. The final evaluation report must be submitted within seven working days of receiving USAID’s comments. (See Section V.B., Evaluation Report Requirements). The report must be submitted initially in English followed by an Arabic translation. USAID/Egypt intends to disseminate the evaluation report and expanded executive summary within USAID, the implementing partner and the Ministry of Higher Education. At the time of submission of the final English language report, the survey instruments, interviews, and data sets must be submitted on a flash drive to the Evaluation Program Manager. All data instruments, data sets, presentations, meeting notes, and final report for this evaluation will be presented to USAID/Egypt’s Evaluation Program Manager. All data on the flash drive will be in an unlocked and editable format. The evaluation team must submit an expanded executive summary to accompany the final report. The expanded executive summary must include a background summary on the evaluation purpose and methodology, and an overview of the main data points, limitations, findings, conclusions, and recommendations. The expanded executive summary must be easy to read for wide distribution to local audiences. The expanded executive summary must be submitted in English and Arabic in hard copy and electronically. Payment. In consideration for the work to be performed by the Contractor, the Government intends to pay the Contractor an agreed upon firm fixed price upon the following conditions: (1) 70% of the fixed price upon acceptance of the final evaluation report in English in accordance with Section V.B., Evaluation Report Requirements; and (2) the remaining 30% of the fixed price upon acceptance of the final evaluation report translated into Arabic. B. Evaluation Report Requirements The format for the evaluation report is as follows: 1. Executive Summary: Concisely state the most significant findings and recommendations (2 pp); 2. Table of Contents (1 pp); 3. Introduction: State purpose, audience, and summary of task (1 pp); 4. Background: Provide brief overview of LOTUS project in Egypt, USAID project strategy and activities implemented in response to the problem, brief description of LOTUS, purpose of the evaluation (2 pp); 5. Methodology: Describe evaluation methods, including threats to validity, constraints and gaps (1 pp); 6. Findings/Conclusions/Recommendations: For each evaluation question, state findings, conclusions and recommendations in clearly demarcated sub-sections; also clear distinctions will be made between findings, conclusions, and recommendations (15–20 pp); 7. Challenges: Provide a list of key technical and/or administrative challenges, if any (1–2 pp); 8. References (including bibliographical documentation, meetings, interviews and focus group discussions); 40 9. Annexes: Annexes that document evaluation scope of work, evaluation methods and limitations, copies of the actual data collection tools, documents reviewed, schedules, interview lists and tables—should be concise, relevant and readable. Annexes should also include a disclosure of any conflict of interest by evaluation team members. The entire report must be no longer than 30 pages, single-spaced in Times New Roman font, size 12 type fonts. General evaluation report guidelines include: • The evaluation report must represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not, and why; • The report must include the evaluation Scope of Work as an annex. All modifications, whether in technical requirements, evaluation questions, evaluation team composition, methodology, budget, or timeline must be agreed upon in writing by the AO. • Evaluation methodology must be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides must be included in an annex in the final report; • Limitations to the evaluation must be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparison groups, etc.) and what is being done to mitigate the threats to validity; • Evaluation findings must be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings must be specific, concise and supported by strong quantitative or qualitative evidence; • Sources of information must be properly identified and listed in an annex; • Recommendations must be supported by a specific set of findings; and • Recommendations must be action-oriented – organized according to whether recommendations are short-term or long-term, practical, and specific, with defined responsibility for the action. The final evaluation report in English must be submitted to USAID/Egypt’s Evaluation Program Manager in electronic format (Microsoft Word) as well as five printed and bound copies no later than two working days after the receipt of the acceptance decision. All data and materials are to be surrendered to and will remain the property of USAID. The final evaluation report will be reviewed using the Checklist for Assessing USAID Evaluation Reports (http://www.usaid.gov/results-and-data/informationresources/program evaluations). The final evaluation report must conform to the Criteria to Ensure the Quality of the Evaluation Report found in Appendix I of the USAID Evaluation Policy. The Evaluation Program Manager will determine if the criteria are met. This evaluation will not be considered approved until the Evaluation Program Manager has confirmed, in writing, that the report has met all of the quality criteria. Once the final evaluation report in English is approved, the expanded executive summary must be submitted to USAID/Egypt’s Evaluation Program Manager in electronic format (Microsoft Word) as well as five English and five Arabic printed and bound copies. A full Arabic translation of the final evaluation report must be submitted to USAID/Egypt’s Evaluation Program Manager in electronic format (Microsoft Word) for approval of the translation. The final evaluation report in Arabic should be proof-read by a professional with a 41 technical background in the subject matter to ensure technical terms are appropriately and accurately translated. Once the final evaluation report in Arabic is approved, five printed and bound copies must be submitted to USAID/Egypt’s Evaluation Program Manager. 6. Evaluation Management A. Logistics USAID will provide overall direction to the evaluation team, identify key documents, and assist in facilitating a work plan. USAID will assist in arranging meetings with key stakeholders identified by USAID prior to the initiation of field-work. The evaluation team is responsible for arranging other meetings as identified during the course of this evaluation and advising USAID/Egypt prior to each of those meetings. The evaluation team is also responsible for arranging transportation as needed for site visits in and around Cairo. B. Team Composition and Roles USAID envisions that the evaluation team will be composed of a team leader, two team members and a logistics coordinator, although the bidder can propose alternative staffing. All attempts should be made for the evaluation team to be comprised of male and female members. Per USAID Evaluation Policy, all team members will be required to provide a written disclosure of conflicts of interest. The personnel’s minimum qualifications are as follows: Team Leader: A senior international consultant with minimum 10 years’ experience in conducting evaluations for higher education scholarships programs. S/he should be familiar with evaluating higher education programs, particularly student support programs (i.e. scholarship and fellowship programs) and be knowledgeable about the higher education sector and development needs in Egypt. S/he should also have a minimum 5 years’ experience in leading evaluation teams, interpersonal relations and writing skills. The evaluation team leader will be responsible for designing and implementing the evaluation and for writing the evaluation report. The Team Leader will: a. Finalize and negotiate with USAID/Egypt the evaluation work plan; b. Design the evaluation plan; c. Establish evaluation team roles, responsibilities, and tasks; d. Facilitate the Team Planning Meeting (TPM); e. Ensure that the logistics arrangements in the field are complete; f. Manage team coordination meetings in-country and ensure that team members are working to schedule; g. Coordinate the process of assembling individual input/findings for the evaluation report and finalizing the evaluation report; and h. Lead the preparation and presentation of key evaluation findings and recommendations to USAID/Egypt team prior to departing Egypt. Team Members: Two mid-level local Evaluation Specialists with minimum 5 years’ experience in monitoring and evaluating and designing education projects, with strong writing skills, excellent understanding of the higher education system in Egypt, as well as familiarity with USAID programs. They should also have proven experience in conducting evaluations in Middle Eastern countries, in the areas of employment, vocational training, labor market and gender. They should have experience in drafting high quality reports. They will participate in different evaluation 42 activities and may be assigned specific tasks by the Team Leader as appropriate. These individuals will be responsible for focus group interviews with LOTUS alumni and current students and interviews with IIE and relevant officials and recipient universities. They will also design the online questionnaire and all interview guides and analyzing survey data. Team members will also write specific sections of the report. The Team Members will coordinate in: i. Designing the evaluation plan. j. Developing a data collection plan. k. Conducting field visits, surveys, and interviews. l. Collecting the data. m. Recording and summarizing the data. n. Analyzing the data collected. o. Preparing reports and presentations for discussing the findings. Logistics coordinator: Three (3) to five (5) years’ experience in handling travel related logistics and providing administrative support. The person should be fluent in written and spoken Arabic. The Logistics Coordinator will: p. Provide administrative support to evaluation team members. q. Be responsible for setting up meetings with USAID and stakeholders. C. Period of Performance The evaluation is envisioned to be carried out over a 15-week period, beginning on March 30, 2016 and concluding by June 23, 2016. The evaluation is envisioned to be carried out over a 15-week period, beginning on March 30, 2016 and concluding by June 23, 2016. For purposes of creating a work plan, please consider the following: The data gathering, compilation, and analysis plan must to be conducted within April 17-May 5 because after that period it will be difficult to meet students who will be having their final exams. [END OF RFTOP] 43 ANNEX II: EVALUATION METHODS AND LIMITATIONS The mixed methods approach to data collection and analysis draws upon multiple sources of information to address the evaluation questions. Multiple sources of information are used to corroborate the findings, enrich the analysis, and minimize the effect of biases inherent in any single source. Table II.1 summarizes data sources, data collection methods, sampling or selection approach, and data analysis methods used in the LOTUS evaluation by evaluation question. TABLE II.1: LOTUS MIDTERM EVALUATION METHODS, DATA COLLECTION, AND ANALYSIS Evaluation Question Answer & Evidence Needed Data Information Description Comparison Explanation Data Sources Data Collection Methods Sampling or Selection Approach Data Analysis Methods EQ1: To what extent does available evidence suggest that the project is on track to achieve its objectives? x No x Project relevant documents as listed in Annex IV Students’ GPA and English language performance grades from universities and IP Document review Selected program documents provided by USAID and IP (e.g., cooperative agreement, M&E matrix, periodic reports, etc.) Students’ GPAs and English ITP test score time series data Content analysis, cross tabulations, descriptive statistics, statistical analyses and scatter plots (i.e., p-value, t-test, Fisher Exact Test, Pearson Chi 2, and ANOVA). Graduates and students currently enrolled, IP scholarship administrators and support staff, partnering university coordinators, student support services staff and faculty Key informant group interviews, group interviews, group discussions Total population of scholarship recipients, and purposive sampling of IP and partnering stakeholder organizations Content analysis, coding of qualitative data, tally sheets, cross tabulations 44 Evaluation Question Answer & Evidence Needed Data Information Description Comparison Explanation Data Sources Data Collection Methods Sampling or Selection Approach Data Analysis Methods EQ2: To what extent has the LOTUS Program enabled recipients to contribute to development, community service, and leadership activities? x No x Project relevant documents Document review Purposive sampling Content analysis Coordinators/administrators from IP and partnering organizations Key informant group interviews, group discussions Content analysis, coding, tally sheets, descriptive statistics, cross tabulations University coordinators Key informant group interviews, group interviews Students participating in different community service activities Survey, group discussions EQ3: To what extent are scholarship recipients satisfied with the academic related and all the other program components like Study Abroad, English language training, leadership in action activities, career counseling, housing, university coordinators, etc.? x No x Graduates (employed and unemployed), currently enrolled students Surveys (paper-and￾pencil, online), group discussions with graduates and currently enrolled students Total population of 72 graduates and 178 currently enrolled students; one to two group discussions per university; ten randomly selected students (both single and mixed sex discussion groups) Descriptive statistics, cross tabulations Eligible candidates who declined acceptance of scholarship award, scholarship recipients who have voluntarily withdrawn from the scholarship program, and scholarship beneficiaries who have been terminated from the scholarship program Telephone interviews Eligible candidates who declined acceptance of scholarship award, students placed on academic probation, students who withdrew from scholarship program, and five scholarship beneficiaries terminated from the program Content analysis, coding, tally sheets 45 Evaluation Question Answer & Evidence Needed Data Information Description Comparison Explanation Data Sources Data Collection Methods Sampling or Selection Approach Data Analysis Methods EQ4: To what extent do scholarship recipients graduate with the academic and soft skills (workforce preparedness) needed to work in jobs suited to their academic preparation? x No x Employed and unemployed LOTUS graduates Online survey, group discussions Exhaustive survey targeting all 75 LOTUS graduates (employed and unemployed) Descriptive statistics, cross tabulations, content analysis, coding, tally sheets 46 The LOTUS midterm evaluation utilized three categories of information and data sources — program documents, quantitative data from secondary sources and surveys, and qualitative data from individual and group interviews and group discussions. Each source and its limitations are discussed below. 1. Program Documents A review of program documents preceded development of primary data collection tools to provide contextual and programmatic information about the LOTUS Activity as a whole. The documents, including quarterly and annual reports on implementation and tracking indicators, were provided by USAID/Egypt. Others, such as discrete studies of particular components or activities and definitions of the indicators, were obtained from IIE’s Middle East and North Africa Regional Office. All evaluators reviewed all documents (see Annex IV for list of documents). The documents included in the desk review were culled from a much larger pool of reporting documents for the six years of the LOTUS Program. They oriented the team and provided a sense of the scope and depth of the activity at the start of the evaluation and helped to ground the findings after data collection. They are the basis of the evaluation. However, they are static and do not provide, in and of themselves, sufficient information on process to address the evaluation questions. 2. Quantitative Data Sources The evaluation team used three quantitative datasets in the analysis — secondary datasets on students and component tracking provided by IIE; a paper-and-pencil survey of all currently enrolled students (n=159); and an online survey of LOTUS alumni (n=83) administered via Survey Monkey. The evaluation team designed and administered both surveys. 2.1. Secondary Data Sources The following databases were received from IIE: 1. The internship tracking sheet for Cohorts 1 and 2, and some students in Cohort 3. Information on all LOTUS student internships are reported in an excel file, including period of internship and name of organization at which students interned; (Filename: Internships Tracking - April 19, 2016.xls). Since information is reported by work description rather than by student, it was referenced primarily to identify the geographic distribution of internships (EQ3). 2. Information about leadership activities and career development/counseling sessions provided to the students since 2010, showing for each session, the target group, facilitator, and when/where it was conducted (Filename: Leadership in Action Sessions Tracker.xls). 3. GPA and ITP scores for each student from enrollment through the 2016 fall semester. The file was transferred to be long shape (see below) (Filename: LOTUS Students GPAs and ITPs 2016-04-21 12-38-56 PM.xls). 4. Information about US Study Abroad: who traveled, GPA achieved abroad, and dates of travel. The file includes all students (n=250) regardless of whether they participated in this component (Filename: LOTUS Study Abroad Students 2016-04-21 12-41-31 PM.xls). 5. Information on the employment status of graduates. This file contained data only for students who had graduated before April 20, 2016 (n=83), that is, graduation date, gender, university, home governorate, field of study, employment status, and place of employment as provided by the student (Filename: LOTUS Students Employability Status 4.20.2016.xls). 47 6. Information from the applications of LOTUS students concerning financial scores, home visit scores, and scores on personal characteristics, such as leadership potential, commitment to community service, and team work recorded during the selection process (Filename: USAID cohort 1.2.xls, USAID Cohort 2.xls USAID Cohort 2.xls USAID Cohort 4.xls). Limitations in the Secondary Data Sources In order to compare LOTUS and non-LOTUS student performance, IIE requested the universities to provide data on the average GPAs of the student body by program of study. Unfortunately, these data were either unavailable on such short notice or produced in a format that did not allow for comparison. Alternatively, IIE prepared information on LOTUS students’ class rankings in four of the six universities. Another challenge was the lack of digitized information on the background characteristics of LOTUS applicants and selected students, which would have allowed the team to analyze how household demographic characteristics affect student performance. IIE provided information for Cohorts 1.2, 2, 3, and 4, but the information is not reflected in the analysis because it was received too late in the evaluation process to allow for merging of these data with other files. 2.2. Primary Data Sources: Survey of Currently Enrolled Students Sample selection for the student survey targeted the entire population of current LOTUS students in six universities (n=159). No sampling was done due to the reasonably small number of respondents. The student survey was conducted over a two-week period (April 26 to May 12, 2016) at the six universities in order to reach the maximum number of students. The first round of surveys was administered to 119 students. At the end of the second round of surveys the following week, the total number of completed questionnaires was 155 out of 159 targeted students, a response rate of 97.5 percent. There was no evidence of rejection and no evidence of non-response bias. The questionnaire was available to students in English and/or Arabic. Sixty-five (65) students answered in Arabic and 90 students in English. See Tables II.2 and II.3 for response rates to the student survey and distribution of responses/response rates by university. TABLE II.2: RESPONSE RATE FOR PAPER-AND-PENCIL SURVEY, CURRENTLY ENROLLED STUDENTS Numbers of Students No. Questionnaires Completed 155 Target Population 159 Response Rate 97.5 No. Questionnaires Completed in Arabic 65 No. Questionnaires Completed in English 90 48 TABLE II.3: RESPONSE RATE FOR CURRENTLY ENROLLED STUDENTS BY UNIVERSITY University No. Received in Round One No. Questionnaires Completed Target Population Response Rate (%) AASTMT 31 45 45 100 ACU 2 4 6 66 BUE 51 65 66 98.5 Future 14 15 16 94.8 MSA 13 15 15 100 Pharos 8 11 11 100 2.3. Online Survey of LOTUS Program Graduates The online survey targeted the total population of LOTUS graduates at the time of the evaluation (n=74). No sampling was conducted due to the reasonably small number of targeted respondents. The online survey was administered via Survey Monkey, an online survey development cloud-based software application. The evaluation team developed and tested the instrument prior to posting it online on May 4, 2016. IIE provided a list of graduates and contact information (name, telephone number, e-mail address). Seventy - four graduates27 received both an e-mail from IIE explaining the origin and purpose of the survey and the importance of their collaboration, as well as the survey instrument. Between May 5 and May 18, when the survey was taken offline, a total of 41 responses and 40 valid completed questionnaires were received. The survey was written in English and, in the absence of inquiries about the questions or procedures, the assumption was that neither the content nor mechanics of the process were problematic. TABLE II.4: RESPONSE RATE, ONLINE SURVEY OF LOTUS GRADUATES No. of Questionnaires Completed Target Population Response Rate (%) No. of Questionnaires E-mailed 40 74 54 40 Limitations Encountered with Online Graduate Survey An important limitation in the LOTUS graduate survey was the absence of respondents’ contact and demographic information (i.e., name, home governorate, gender, university, field of study, and cohort). These were eliminated from the questionnaire because of the mistaken assumption that the responses would be linked to e-mail addresses. This omission precludes any disaggregation or comparative analysis among respondents and prevents a merge of survey responses with secondary historical data made available by IIE. A second problem was the high number of missing values in the last ten questions (out of 40); (Filename: Report about missing and N.A_Graduate_students.xls). This may have been due to the fact that the instrument did not allow respondents to return to questions they may have skipped inadvertently by looking ahead to see, for example, how many questions remained. Finally, a potential for non-response bias exists as only 40 out of 74 targeted respondents completed the survey. This bias may be significant in opinion surveys since those with firm 27 Only 74 of the 83 graduates received the e-mailed hyperlink to reply to the survey. One student was terminated from LOTUS prior to graduation and eight students who continued to receive LOTUS financial support were completing their one-year residency in dentistry and/or physical therapy. 49 opinions are more likely to respond than those who feel more ambivalent. At the same time, there is evidence to support the reliability of the survey despite the moderate number of observations. When the employability variable of the online survey is compared with secondary data from IIE (Filename: LOTUS Students Employability Status 4.20.2016.xls), excluding those in residence, they compare fairly well. For example, the employment rate of survey respondents was 60 percent whereas in the IIE dataset it was 63 percent. Similarly, 30 percent responded as not working in the survey compared to 27 percent in the IIE dataset. 3. Qualitative Data Sources The team used two methods to collect qualitative data: (1) group discussions with LOTUS scholarship recipients, and (2) face-to-face individual and group interviews with university administrators, student support staff and faculty, partner NGOs (Nahdet el Mahrousa, AFS-Egypt, and E-ERA), USAID managers, and IIE LOTUS program staff. Interviews with IIE included (1) group interviews with LOTUS program staff, (2) a telephone interview with the IIE/New York Study Abroad staff, and (3) individual interviews with the Leadership in Action team and the academic coordinators. 3.1 Group Discussions Eleven group discussions were held with current LOTUS students from the six universities. They took place once students had completed the paper-and-pencil survey. Initially, a sample was drawn and selected students were invited to participate. In practice, because of the holiday schedule and student time constraints, substitutions of participants were made without reference to the selection criteria, and in BUE and AASTMT a third group discussion was added a week after the first two groups to accommodate students who were away during the first run.28 In addition, the team encountered unexpected enthusiasm on the part of students who wished to join the group discussions, hence, the decision not to refuse their participation. TABLE II.5: DISTRIBUTION OF STUDENT DISCUSSION GROUPS University Gender Cohort Number of Participants AASTMT female 3, 4 7 male 3, 4 6 mixed 3, 4 9 BUE female 3, 4 10 male 2, 3, 4 11 mixed 1, 3, 4 7 ACU male 1 3 FUE mixed 1.2, 2 10 MSA female 1 10 PUA male 1.2, 2 4 PUA male 1.2 4 TOTAL 81 28 Because the first group discussions occurred during the Christian holy week, most of the Christian students were unable to attend, hence, the risk of a non-response bias. 50 The protocol consisted of six broad topics based on the four evaluation questions (see data collection protocols and survey instruments in Annex III). Group discussions ran from an hour (abbreviated due to student time constraints) to two hours and 15 minutes, about an hour and 45 minutes on average. In each, one evaluator directed the discussion while the second took notes on the meeting. The notes were typed and coded for principal themes using an Excel tally sheet for qualitative data analysis. All group discussions were conducted in Arabic,29 and held on campus in a venue provided by the university for this purpose. IIE contacted the students via e-mail informing them of the time, place, and purpose of the group discussions and emphasized the importance of their participation. A twelfth discussion group, not included in the original work plan, was held with LOTUS graduates on a Friday (a weekend day in Egypt) at the IIE office. Again, IIE contacted the graduates, invited them to participate, and offered to pay for their transportation expenses. No IIE staff was present during the discussion. The protocol focused on employability as well as the effect of other LOTUS components, especially the L eadership in Action component, on their lives since graduation (see Annex III for graduates’ group discussion protocol). Four women (three from Cohort 1 and one from Cohort 2) participated in the discussion, which lasted about two hours. 3.2 Key Informant Individual and Group Interviews The evaluation team prepared a separate interview protocol for each key informant category and for each group interview, focusing the questions directly on the informants’ roles within the LOTUS Program. The interviews were constructed to elicit factual information as well as perceptions, opinions, and lessons learned/recommendations. Sixteen individual interviews (some with two persons present) and nine group interviews were held.30 Interviews were designed to last about an hour, although they frequently ran longer, especially the individual interviews. In all cases, key informant interviews were conducted in the offices of the interviewees (see Annex IV for the list of persons consulted). As with the case of the discussions, notes from each interview were typed and coded according to the key themes, utilizing a separate Excel tally sheet for the university interviews and the NGO interviews. 29 The team leader was assisted by a simultaneous translator/interpreter in groups where the team leader was also note taker. 30 Individual interviews were held with university presidents or vice-presidents (from four of the six partner universities), LOTUS university coordinators (6), USAID/Egypt LOTUS AOR (1), recruitment NGOs (2), the IIE leadership in action director (1), and IIE academic coordinators (2). The group interviews were conducted with participants in university faculty training (5), the USAID OET Team (1), Nahdet el Mahrousa (1), IIE LOTUS program staff (1), and IIE/NY Study Abroad team (1). 1 ANNEX III: DATA COLLECTION INSTRUMENTS Fifteen data collection instruments were used to gather data for the LOTUS midterm evaluation. These included two surveys (paper-and-pencil and online), three key informant interview protocols, two telephone interview protocols, one group discussion protocol, and seven group interview protocols. The instruments and targeted subjects are listed in the table below, followed by copies of the instruments. TABLE III.1: DATA COLLECTION TOOLS Type of Instrument Targeted Subjects Paper-and-pencil survey Current students, Egyptian universities, Arabic and English Online survey Graduates and Employed / Unemployed and Other Key informant interview LOTUS Management, IIE LOTUS manager of Leadership-in-Action Key informant interview LOTUS Management, USAID/OET LOTUS program manager Key informant interview University presidents Key informant interview (telephone) Students terminated or dismissed Key informant interview (telephone) Students who voluntarily withdrew Group discussion Current students, Egyptian universities Group interview LOTUS management staff, IIE Group interview LOTUS management staff, IIE/New York office and/or IIE MENA Regional Director Group interview LOTUS management staff, IIE program coordinators Group interview LOTUS management, NGO partners, Nahdet Mahrousa Group interview PARTNER NGOs – Rrecruitment and Selection Group interview USAID/OET program management Group interview Leadership Development Program (LDP) faculty, universities 52 استمارة موافقة مسبقة للمشاركة في الدراسة البحثیة تھدف الدراسة ھذه البحثیة الى إجراء تقییم منتصف المدة لبرنامج المنح ( الدراسیة لوتس) لتوفیر المعلومات اللازمة لھیئة المعونة الامریكیة في مصر من أجل المساھمة في تحسین اداء لوتس خلال الفترة المتبقیة. وقد قامت ھیئة المعونة الامریكیة بمصر بالتعاقد مع ھیئة SIMPLE من أجل ھذا إجراء التقییم، وھم مسئولون عن جمع البیانات من خلال البحث الكمي .والكیفي المشاركة في الدراسة ھذه البحثیة ھو أمر تطوعي، حیث انك أحد المستفیدین أو الأطراف المعنیة ببرنامج المنح ( الدراسیة لوتس). ھذا الاستبیان سوف یستغرق ما یقرب من 35 دقیقة. سوف یتم التعامل بسریة كاملھ مع جمیع الإجابات والمعلومات الواردة بالاستبیان . ونؤكد أن البیانات الدراسة بھذه الواردة سوف تستخدم فقط للأغراض البحثیة ولن یتم عرضھا أو مشاركة أي شخص أو جھة بھا. ونرجو منك التفضل بملىء البیانات التالیة : الاسم : التاریخ : التوقیع : 53 Current Students Paper-and-Pencil Survey مرحبا بكم في ھذا الاستبیان لقد تم اختیارك للمشاركة في ھذا الاستبیان لأنك حصلت على منحة لوتس. ومن أجل جعل المنحة أكثر ملائمة وفعالیة، فقد قامت الوكالة الأمریكیة للتنمیة في مصر بالتعاقد مع ھیئة SIMPLE) خدمات تحسین الأداء، إدارة وتعزیز التعلم والتقییم ) لإجراء تقییم برنامج منحة لوتس. إن مشاركتكم في الدراسة ھذه في غایة الأھمیة. لذلك من المھم أن نسمع من الجمیع حتى نعزز من نتائجنا. تستغرق الإجابة على الأسئلة حوالي 30 دقیقة . نود التأكید على أنھ سیتم الاحتفاظ بإجاباتكم سریة تماما، وسوف یتضمن الدراسة تقریر على نتائج عامة، وغیر شخصیة . نشكرك مقدما على مساعدتك في ھذا الاستبیان المھم 54 البیانات :الشخصیة الاسم............................................. ( النوع ضع دائرة حول الإجابة) 1 - ذكر 2 - أنثى تاریخ المیلاد: الیوم: الشھر: السنھ: المحافظة التابع لھا عند دخول المنحة: ................................................. التخصص في الثانویة العامة ( ضع دائرة حول الإجابة): 1 - علمي 2 - أدبي المجموع في الثانویة العامة % : الجامعة-------------------------------------------------------------------: الكلیة: ---------------------------------------------------------------------- :الدراسة تخصص ------------------------------------------------------------- الفصل ( الدراسي ضع دائرة حول الإجابة): 1 الأول - 2 - الثاني 3 الثالث - 4 الرابع - 5 - الخامس سنة الحصول على المنحة: القسم ( 2): كیفیة الوصول وعملیة الاختیار كیف عرفت عن منحة LOTUS برجاء( ؟ وضع دائرة فقط حول المصدر الأول للمعرفة ) الرقم . 55 الجرائد فیسبوك تویتر أصدقاء المدرسة طالب في منحة LOTUS البحث على الانترنت الجمعیة ( الشریكة AFS للبرامج مصر الثقافیة والتعلیمیة) الجمعیة الشریكة (ERA-E الجمعیة المصریة لمصادر التعلیم) الجمعیة ( الشریكة نھضة )المحروسة مشروع المعھد الدولي للتعلیم ( IIE( الرادیو جھة حكومیة ( مثل وزارة التعلیم، مكتبة عامة، أو المجلس القومي للشباب) عرض تقدیمي عن الفرض في محافظتك أخرى(حدد)........................................ ھل أتممت الخطوات التالیة للالتحاق بمنحة LOTUS ؟ (علم على اختیار واحد بكل سطر): الخطوات نعم (1) لا (2) تقدیم طلب الالتحاق مع المستندات المطلوبة المقابلة الجماعیة الزیارات المنزلیة ھل اخترت تخصصك أم تم تحدیده لك؟ ( ضع دائرة حول الإجابة): اخترتھ تم تحدیده لي ھل اخترت جامعتك أم تم تحدیدھا لك(ضع دائرة حول الإجابة): اخترتھا تم تحدیدھا لي كانت الأفضل لمجال دراستي القسم ( 3): تطویر المستقبل الوظیفي التدریب الاستشارات،( العملي، والملتقى الوظیفي) A ھل التحقت بفرصة تدریب عملي (internship (أو أكثر خلال سنوات دراستك في الجامعة؟ ( ضع دائرة حول الإجابة) 1 - نعم 2 لا - إذا كانت الإجابة نعم، برجاء استكمال الجدول التالي لكل فرص التدریب خلال سنوات دراستك في الجامعة : (اكتب رقم اختیارك بكل فئة) -5B اسم المنظمة او الشركة -5Cأین المكان؟ -1القاھرة 2 اسكندریة- -3أخرى ( حدد: - ----( D-5 إجمالي عدد الأیام E-5 الفصل الدراسي -5F ھل لھا علاقة بمجال دراستك : نعم لا -5G من أین حصلت علیھ: نھضة المحروسة شبكة ( علاقاتي الأسرة، الأساتذة، الأصدقاء) IIE أخرى ( حدد: )-------- 56 A - ھل حضرت أي جلسات تطویر لمستقبلك الوظیفي (session development career (خلال دراستك؟ (ضع دائرة حول الإجابة) نعم لا إذا كانت الإجابة نعم، برجاء استكمال الجدول التالي لكل جلسات تطویر مستقبلك الوظیفي خلال سنوات دراستك في الجامعة: (اكتب اختیارك بكل فئة) -6D- درجة رضاك أو عدم رضاك عن الجلسة؟ ( 4-1 ) 1 غیر= راض على الإطلاق 2 غیر= راض 3= راض 4= راض جدا -6B أذكر الاسم/موضوع -6C الجھة المنظمة الجلسة A ھل حصلت على أي جلسة خاصة التوظیفیة بالإستشارات (counseling career (أثناء وجودك في الجامعة؟ (ضع دائرة حول الإجابة) نعم لا لو كانت الإجابة نعم، برجاء إستكمال الجدول التالي : (اكتب اختیارك بكل فئة) -7D ما مدى رضاك أو عدم رضاك عن الجلسة؟ ( 4-1) 1=راض جدا 2 غیر= راض 3= راض 4= راض جدا -7C متوسط عدد الجلسات الشخصیة في ھذا العام -7B الجھة المنظمة: 1= نھضة المحروسة 2= جامعتك 3= أخرى ( حدد....:) A ھل حضرت أي ملتقى وظیفي؟ (ضع دائرة حول الإجابة) نعم لا 8 - B لو نعم، ھل حصلت من خلالھ على أي علاقات مفیدة لك؟ (ضع دائرة حول الإجابة) نعم لا ما حتى اقتراحاتك یكون مكون تطویر المستقبل الوظیفي مفیدا وأكثر فاعلیة؟ اقتراح ..............................................................................................................:1 ......................................................................................................................... اقتراح ................................................................................................................:2 ............................................................................................................................ 57 القسم (4): أنشطة القيادة برجاء ذكر مواضیع الثلاثة الأكثر تأثیرا، والأقل تأثیرا من مكون Action in Leadership؟ A-الأكثر تأثیرا B- الأقل تأثیرا مع من تتواصل إجتماعیا من الفئات التالیة، والي أي مدى؟ ( علم اختیار على واحد بكل سطر): نادرا ( 1) أحیانا ( 2) غالبا ( 3) دائما ( 4) طلاب LOTUS الطلبة الأخرون الطلبة الآخرون في مجال دراستك مقارنة مع زملائك السابقین بالمدرسة الثانویة، ھل تعتقد كطالب في LOTUS ان لدیك فرصة أكبر أو أقل فیما یلي(علم على اختیار واحد بكل سطر): العبارة فرصة اقل ( 1) نفس الفرصة ( 2) فرصة اكبر ( 3) الحصول على فرصة عمل سریعا أصبح لدیك مھارات قیادیة مؤثرة أصبح لدیك وضع إجتماعي (prestige (أفضل تحسن وضعك المادي القدرة على بدأ عملك الخاص المساھمة لخدمة مجتمعك أخرى (حدد: ---------------) القسم ( 5): الخدمات المجتمعیة :والتطویر A ھل قمت بأي خدمات مجتمعیة (service community (خلال السنة الماضیة؟ نعم لا إذا كانت الإجابة بنعم، ما ھي أنواع الخدمات التي قمت بھا وأین؟ (ضع دائرة حول الإجابة المختارة بكل عمود): -13 B في محافظتي -13C في الجامعة -13D خارج الجامعة لكن في القاھرة أو الأسكندریة تطوع نوادي جامعیة تطوع مشروع تعلیم خدمي اتحاد طلاب مشروع تعلیم خدمي أخرى (حدد: -------) مشروع تعلیم خدمي أخرى (حدد: -------) أخرى (حدد: -------) ما التوصیات التي تقترحھا لجعل مكونات القیادة، والمجتمع والتطویر مؤثرة لك؟ توصیة ( 1)............................................................................................................................... ................ ............................................................................................................................ توصیة ( 2) .............................................................................................................................. ............................................................................................................................................ توصیة ( 3) .............................................................................................................................. ............................................................................................................................................ 58 القسم (6): الحياة الأكاديمية على حسب مجال دراستك وجامعتك، ما مدى ثقتك في التالي ( علم على اختیار واحد بكل سطر): العبارة غیر واثق على الإطلاق ( 1) غیر واثق ( 2) واثق ( 3) واثق جدا ( 4) الحصول على وظیفة في مجال تخصصك الحصول على وظیفة جیدة القبول في برامج الدراسات العلیا البدأ في عملي الخاص أن أكون رائدا اجتماعیا (entrepreneurship social( استكمال عملي التطوعي ما مدى تكرار التحدیات التالیة معك؟ ( علم على اختیار واحد بكل سطر): نوع التحدي نادرا (1) أحیانا (2) غالبا (3) دائما (4) الحاجة للمزید من الدعم الأكادیمي إدارة المحاضرة أسالیب التدریس الأمكانیات التعلیمیة في الحرم الجامعي التكلیفات التعامل مع الطلبة الآخرین السكن اللغة مشاكل صحیة افتقاد المنزل (sickness home( مع من تتواصل لمناقشة التحدیات السابق ذكرھا في السؤال بأعلى؟ ( علم على اختیارك بكل سطر) نعم (1) لا (2) الأصدقاء والزملاء الطلاب الآخرون في LOTUS الأساتذة والمعیدون منسق LOTUSفي جامعتك موظفو شؤون الطلاب الأبوین أو الأقارب لا أحد أخرى (حدد--------------:) 59 في تقییم من 4-1 ، 1 ھو الأقل و 4 ھو الأعلى، ما مدى رضاك أو عدم رضاك عن الخدمات التالیة : (علم على اختیار واحد بكل سطر): الخدمات غیر راض جدا ( 1) غیر راض (2) راض (3) راض جدا (4) الدعم الأكادیمي الذي تحصل علیھ من الجامعة خدمات الدعم الطلابي (مثل الخدمات الصحیة، الأندیة والأنشطة) في الجامعة؟ التوازن بین المتطلبات الأكادیمیة، أنشطة LOTUS الأخرى وحیاتك الاجتماعیة القسم (7): تعليم اللغة الانجليزي من فضلك حدد درجة إتقانك للغة : الإنجلیزیة القراءة و الكتابة و التحدث و قبل خلال الجامعة: (ضع الرقم المختار بكل فئة) الوقت القراءة الكتابة التحدث غیر متقن ( 1) إلى حد ما ( 2) متقن ( 3) متقن جدا ( 4) غیر متقن ( 1) إلى حد ما ( 2) متقن ( 3) متقن جدا ( 4) غیر متقن ( 1) إلى حد ما ( 2) متقن ( 3) متقن جدا ( 4) عند بدایة المنحة الآن بعد ما بدأت الدراسة في الجامعة، ھل حصلت على دورات لغة انجلیزیة خارج منحة LOTUS ؟ (ضع دائرة حول الإجابة) نعم لا قیم مدى كفاءة المكونات التالیة في تدریب اللغة الانجلیزیة؟ ( علم على اختیار واحد بكل سطر): غیر مؤثرة على الإطلاق غیر مؤثرة مؤثرة مؤثرة جدا لم أشارك السنة الانتقالیة year Bridge program برنامج صیفي مكثف للغة الانجلیزیة Intensive summer English program منھج إنجلیزي إضافي أخرى (حدد: ------------) ما التوصیات التي تقترحھا لتطویر خدمات مكون اللغة الانجلیزیة؟ توصیة ( 1)............................................................................................................................... ............... ............................................................................................................................. توصیة ( 2) .............................................................................................................................. ................ ............................................................................................................................ القسم (8): الدراسة بالولايات المتحدة الأمريكية ھل شاركت في مكون الدراسة في الولایات المتحدة الأمریكیة؟ (ضع دائرة حول الإجابة) نعم لا إن كانت الإجابة بلا برجاء الانتقال للقسم ( 9) سؤال 28 60 كم امتدت فترة إقامتك في المتحدة الولایات الأمریكیة؟ فصل دراسي واحد دراسة صیفیة أخرى ( حدد: -----------------------) ما ھو أكثر شيء أعجبك في التجربة؟ ( اختار أھم ثلاثة أشیاء من القائمة التالیة وأكتبھا) أكثر ما أعجبك قائمة الاختیارات التعرف على الثقافة الأمریكیة الاختیار الأول---------------------------: الاختیار الثاني--------------------------: الاختیار :الثالث -------------------------- ممارسة اللغة الإنجلیزیة البرنامج الأكادیمي توسیع دائرة المعارف (Networking( الأنشطة الطلابیة زیارة المزارات السیاحیة الإمكانیات الجامعیة أخرى (حدد: -------------------------) ما ھو أكثر شيء تود تغییره في التجربة؟ ( اختار أھم ثلاثة أشیاء من القائمة التالیة وأكتبھا) التغیرات المقترحة قائمة الاختیارات اختیار الجامعة الاختیار الأول---------------------------: الاختیار الثاني--------------------------: الاختیار :الثالث -------------------------- البرنامج الأكادیمي مدة الإقامة عملیة الاختیار الترتیبات الإداریة والفیزا الأنشطة الطلابیة أخرى (حدد: -------------------------) أعطي مثالا من خلال تجربتك الشخصیة عن مدى تأثیر مكون الدراسة بالولایات المتحدة الأمریكیة على حیاتك الشخصیة أوالعملیة أو مدى تأثیر ذلك على رؤیتك للأمور المختلفة. كن محددا. ...................................................................................................................................................................... ...................................................................................................................................................................... ....................................................... القسم (9): التوصيات ھل توصي أشقائك أو شقیقاتك أو أصدقاءك المقربین على الإلتحاق بمنحة LOTUS ؟ نعم لا لا أعرف لماذا؟ برجاء الشرح : ...................................................................................................................................................................... ..................................................................................................................................................................... شكرا وقتك، على برجاء التأكد من الإجابة على كل الأسئلة  61 Consent Form Student Survey The purpose of this research is to conduct a midterm evaluation for LOTUS Scholarship Program to provide USAID/Egypt with information to help improve the scholarship program’s performance in the remaining implementation period. The evaluation team from the SIMPLE project has been contracted by the USAID/Egypt Mission to conduct this evaluation and they are responsible for data collection through quantitative and qualitative surveys. We kindly ask you to participate voluntarily in this research project because you are one of the beneficiaries/stakeholders of the higher education scholarship program. This survey form is estimated to take you approximately thirty-five minutes. To protect your confidentiality, reports and datasets will not contain information that will personally identify you. The results of this study will be used ONLY for research purposes. Please provide us the information below to show you agree to participate. Name: Date: Signature: 62 Welcome to the Survey. You have been chosen to participate in this survey because you received a LOTUS Scholarship. To help make its programs more relevant and effective, USAID/Egypt has contracted our firm, Services to Improve Performance Management, Enhance Learning and Evaluation (SIMPLE) to conduct an evaluation of the Leadership Opportunity Transforming University Students Scholarship Program (LOTUS). Your response is very important. We are giving this questionnaire to LOTUS recipients and it is important that we hear from everyone in order for our results to be meaningful. Answering the questions will take about 30 minutes. All your answers will be kept strictly confidential. Your personal identifiable information will not be shared with anyone and reports will not identify individual responses. Thank you in advance for your assistance with this important survey. 63 *Please be sure to answer each question Introductory Information Number: Your Name: Gender: (Circle one answer only) Male 2. Female Year of birth: Home Governorate:-------------------------------------------------- Specialty in Thanawya: Science 2. Art Grade in Thanaweya in %: % University:------------------------------------------------------------------- Faculty:------------------------------------------------------------------------ Field of Study:---------------------------------------------------------------- Academic Year (Circle the year) 1. First 2. Second 3. Third 4. Fourth 5. Fifth Year received the scholarship: Section 2: Outreach and Selection Process How did you find out about the LOTUS Program? (Circle one primary source) Newspaper Facebook Twitter Friends School LOTUS Scholarship student Web search AFS E-ERA Nahdet Mahrousa IIE project Radio Government Institution (such as the Ministry of Education, Public Library, or the National Council for Youth) Presentation about the opportunity in your governorate Other (Specify, ---------------------------------------------------------------) 64 To become a LOTUS student, did you complete the following steps? (Please check the correct answer) Steps YES (1) NO (2) a. Application with supporting documents b. Group Interview c. Home Visit Did you choose your field of study or was it assigned to you? (Circle your response) I chose it It was assigned to me Did you choose your university or was it assigned to you? (Circle your response) I chose it It was assigned to me It was the best for my field of study Section 3: Career development (Career Counseling, Internships, and Job Fairs) A. Have you had one or more internships while you have been in the university? (Circle your response) Yes No If YES, please complete the following table for all internships you had during your university years B￾Organization/ Company Name C-Location 1- Cairo 2 - Alex 3 - Other (Specify:______) D- Total number of days E￾Academic Year31 F-Relevance to your field of study 1- Yes 2 No G- How did you get the internship? 1- Nahdet Mahrousa 2- My network (family, professors, friends) 3- IIE (Institute of International Education) 4- Other (specify: __________________) 31 If the internships are done in summer, then reference the previous academic year 65 A- Did you attend any career development sessions during your university years? (Circle your response) Yes No If YES, please complete the following table, for all career development sessions you had during your university years B- List the name/topic of each session C-Organizing entity 1= Nahdet Mahrousa 2= Your University 3= Other (Specify: _______________) D-How satisfied or dissatisfied were you with the session? (1 to 4) 1=Very dissatisfied 2= Dissatisfied 3= Satisfied 4= Very satisfied A- Are you receiving any career counselling sessions while you are in the university? (Circle your response) Yes No If YES, fill out the following table: B- Organizing entity 1 = Nahdet Mahrousa 2 = Your university 3 = Other (Specify: _____________) C- Average number of face￾to-face sessions this year D- How satisfied or dissatisfied were you with the session? 1=very dissatisfied 2= dissatisfied 3= satisfied 4= very satisfied A- Did you attend any job fairs? (Circle your response) Yes No B- If yes, did they result in any useful contacts for you? (Circle your response) Yes No 66 What do you recommend to make the career development component more effective? Recommendation 1:------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------ Recommendation 2:------------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------- Section 4: Leadership Activities List the three most influential and least influential topics from the Leadership in Action curriculum? Most influential 1- 2- 3- Least influential 1- 2- 3- How often do you interact socially with: (Please check the correct answer in each line) Rarely (1) Sometimes (2) Often (3) Always (4) a-LOTUS students b-Other students c-Other students in your field of study Thinking about your high school peers, as a LOTUS student, are you more or less likely to………: (check the appropriate box on each line) Statement Less likely (1) As likely (2) More likely (3) Most likely (4) A- Get a good job quickly B-Have effective leadership skills C- Gain more prestige D- Gain more money E- Start your own business F- Contribute to your community G- Other (Specify:____________) 67 Section 5: Community Service and Development A- Have you done any community service during the past year? (Circle your response) Yes No If yes, what kind of community services have you done and where? (Check all that apply) B- In my home governorate C- On campus D- Off campus but in Cairo or Alexandria Volunteering University Clubs Volunteering Service Learning Project Student Union Service Learning Project Other (Specify: ______________) Service Learning Project Other (Specify: ______________) Other (Specify: ______________) What recommendations do you have to make the leadership, community service and development component more effective for you? Recommendation 1:------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------ Recommendation 2:------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------ Recommendation 3:------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------ Section 6: Academic Life Based on your field of study and university program, how confident are you about: (Check one box in each line) Statement Not at all confident (1) Not confident (2) Confident (3) Very confident (4) A- Getting a job in my field of study. B- Getting a good job. C- Getting accepted in a post graduate program D- Starting my own business E- Doing social entrepreneurship F- Continuing to do volunteer work 68 How frequently do you face the following challenges? (Check one box for each line) Type of challenge Rarely (1) Sometimes (2) Often (3) Always (4) Need more academic advice Classroom management Instructional strategies Campus learning facilities Assignments Dealing with other students Housing Language Health problems Home sickness Who do you contact to discuss the above challenges in the question above? (Check one box in each line) Yes No Friends and peers Other students in the LOTUS Program Faculty and Teacher Assistants LOTUS program coordinator at your university Student Affairs staff Parents and family No one Other (Specify: ---------------------) How satisfied or dissatisfied are you with each of the following services: (Check one box in each line) Service Very dissatisfied (1) Dissatisfied (2) Satisfied (3) Very Satisfied (4) The academic support you are receiving from the university Student support services (such as health services, clubs and activities) at the university? The balance among academics, other LOTUS activities, and your social life. 69 Section 7: English Language Training In your opinion how do you rate your English language proficiency in reading, writing, and speaking: (Put one number in each box) Time Reading Writing Speaking Not proficient 1 Somewhat proficient 2 Proficient 3 Very proficient 4 Not proficient 1 Somewhat proficient 2 Proficient 3 Very proficient 4 Not proficient 1 Somewhat proficient 2 Proficient 3 Very proficient 4 When you started the program Now After starting university, did you take additional English courses outside the LOTUS Program, at your own expense? (Circle your response) Yes No Rate the level of effectiveness of the following types of the LOTUS English language training? (Check one box in each line) Type of training Not at all effective Not effective Effective Very effective Did not participate Bridge year program Intensive summer English program Tutoring/Supplemental English Program Other (Specify: -------------- ------) What recommendations do you have to improve the English language training? Recommendation 1:------------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------- Recommendation 2:------------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------- 70 Section 8: Study Abroad Did you participate in the Study Abroad component? (Circle your response) Yes No (If NO, go to Section 9, question 28). How long did you stay in the US? One semester Summer session Other (Specify: ------------------------------------) What did you like the most about the experience? (Circle your response) (Select the 3 most important from the list below) What you liked the most? List of Options First:---------------------------------------- Second:------------------------------------ Third:--------------------------------------- Exposure to US culture Practicing English Academic program Networking University campus activities Sightseeing US university facilities Other (Specify: ---------------------------) What would you change about the experience abroad? (Select the 3 most important from the list below) Your suggested changes List of options First:---------------------------------------- Second:------------------------------------ Third:--------------------------------------- University choice Academic program Length of stay Selection process Administrative and visa arrangements Student life activities Other (Specify: -------------------------------) Give an example from your own experience about how the Study Abroad component affected your personal or professional life or gave you a different perspective. Be specific. ------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------- 71 Section 9: Recommendations Would you recommend the LOTUS Program to your siblings or close friends? (Circle your response) Yes No Don’t know Why, please elaborate: --------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------- Thank you for your time. Please make sure that you responded to all the questions.  72 Current Student Group Discussion Protocol بروتوكول مجموعات النقاش مقدمة : أھلا بیكو في مجموعة النقاش، لقد تم اختیارك للمشاركة في مجموعة النقاش لأنك حصلت على منحة لوتس. وعلشان المنحة تبقي أكثر ملائمة وفعالیة، قامت الوكالة الأمریكیة للتنمیة في مصر بالتعاقد مع ھیئةSIMPLE ( خدمات تحسین الأداء، إدارة وتعزیز التعلم والتقییم ( لتقییم منحة لوتس . مشاركتكم في الدراسة ھذه في غایة الأھمیة ومھم جدا نسمع كل الأراء علشان نعزز من نتائجنا. مافیش حاجة صح او غلط الموضوع بیعتمد علي وجھة نظركوا . احنا بنتكلم مع طلبة الجامعات التانیة . وحیتم دمج اجابتكم مع نتائج الاستبیان والمقابلات علشان نحصل علي توصیات لمنح تانیة في المستقبل . نحب نأكد ان سیتم الاحتفاظ بسریة إجاباتكم حیتضمن تقریر مجموعة النقاش على نتائج عامة، ومن غیر اسامي . نشكركم علي مساھمتكم الھامة. اسئلة عامة ایھ اھم مزایا إنك تكون طالب في منحة لوتس؟ او لما اقول منحة لوتس ایھ اھم حاجة بتخطر علي بالكوا؟ تفتكروا كنتم حتعملوا ایھ/حتكونوا فین بعد الثانویة عامة لو ما اتقبلتوش في المنحة؟ فكروا في اول اوریانتیشن) الجلسة التحضیریة (البرنامج عملھ .ھل اھلكم) حضرك (بشكل كامل للانتقال للحیاة الجامعیة؟ طب ایھ اقتراحاتكم عن حاجات) نشاطات (تانیة تتزود؟ الحياة الاكاديمية إزاي إختارتو التخصص بتاعكم؟ طب ايه تقيم تجربتكم في التخصص ده لحد دلوقتي؟) الضغوط، صعوبه الدراسة، انجليزي، طرق التدريس ، امكانيات الحرم الجامعي،التكليفاتassignments ، المناخ الاكاديمي، علاقته بالحياة العملية( القیادة، تنمیة المجتمع والتنمیة ازاي تقیم نشاط برنامج القیادة العملیة وأجزائھ المختلفة Leadership in Action نمي مھاراتك القیادیة (التدريب علي التواصل، حل المشكلات، ادارة النزاع، مشاريع خدمة المجتمع) ھل طبقت اي من اللي المھارات اكتسبتھا من التدریبات في حیاتك العملیة او في ارض الواقع ؟ ازاي؟ (ھل كان فيه تدريبات مالقتش ليھافايده عملية ؟ ) احكي لنا موقف حصلك في حیاتك الجامعیة واتصرفت فیھ كقائد؟ ایھ نظرة اھلك اللي والناس في مجتمعك لیك دلوقتي كطالب منحة لوتس؟ في رأیك ایھ الاجزاء في مكون خدمة المجتمع والتنمیة اللي ما تطرقش لیھا البرنامج بشكل كافي؟ كطالب منحة لوتس، ما مدي مشاركتك او اندماجك في تخطیط أو تعدیل نشاطات خدمة المجتمع او أیمكونات اخري؟ ھل تعتقد ان رأیك في تقییم التدریبات بیعمل فرق؟ ( تقییم التدریبات، المقابلات مع ال IIE وخصوصا مع مسئولة التقییم ) الدراسة في المتحدة الولایات الأمریكیة إیھ القیمة المضافة للدراسة في الخارج) التعرف علي الثقافة الأمریكیة ،البرنامج الأكادیمي،التشبیك، نشاطات الطلبة( ھل الاوریانتیشن) الجلسة التحضیریة (حضرتك بشكل كافي للتجربة؟ ایھ اللي تحب تغیره في تجربتك ؟) الجامعة، المواد، الفترة/المدة، طریقة الاختیار، الترتیبات الاداریة والفیزا، نشاطات الطلبة( للناس اللي ماسفرتش :ایھ اضافات) اختیارات او بدائل (ادارة البرنامج كان ممكن تقدمھالك؟ التحدیات ایھ انواع التحدیات اللي واجھتك في المنحة والجامعة؟) الدعم الاكادیمي، السكن، الصحة، الانجلیزي، الدعم المعنوي، تكوین صداقات مع طلبة من خارج من المنحة( ایھ اللي ممكن تخلیھ او تغیره في المنحة علشان الطلبة الجدیدة؟ الحیاة بعد التخرج ایھ خططك الشخصیة والعملیة بعد التخرج؟ فین المجتمع من الخطط دیھ؟ في رأیك ایھ اھمیة منحة لوتس لتحسین فرصتك في الحصول عل عمل بعد التخرج؟ في رأیك ایھ اھمیة منحة لوتس لتحسین فرصتك في الحصول للارتقاء في مستقبلك المھني؟ ایھ مصدر المال لمنحة لوتس؟ الفلوس جت منین؟ 73 Welcome to the Group Discussion You have been chosen to participate in this group discussion because you received a LOTUS Scholarship. To help make its programs more relevant and effective, USAID/Egypt has contracted our firm, Services to Improve Performance Management, Enhance Learning and Evaluation (SIMPLE) to conduct an evaluation of the Leadership Opportunity Transforming University Students Scholarship Program (LOTUS). We have prepared a series of discussion questions. It is very important that everyone participate. These questions do not have right or wrong answers; it is all about your perceptions. We are talking to LOTUS students in the other partner universities. Your answers will be combined with the results from the surveys and other interviews to provide guidance and recommendations for scholarship programs in the future. Your opinions will be kept strictly confidential. We will report on the discussion of the group as a whole and we will not identify individual responses. Thank you in advance for your assistance with this important discussion. General questions about the LOTUS Program What are the main benefits of being a LOTUS student? Or When I say “LOTUS”, what is the most important thing about it? What would you have done after high school if you had not joined the LOTUS Scholarship? Thinking about the orientation session at the beginning of the program, did it fully prepare you for your transition to the university life? Do you have recommendations for other activities to be included? Questions about the academic program How did you choose your field of study? How do you assess your experience in this field of study so far? (probes: pressure, difficulty of courses, teaching theories and methodologies, facilities and labs, intellectual climate, assignments, life￾related…) Questions about leadership, community services and development How have the various activities in the Leadership in Action component enhanced your leadership skills? (e.g., training in communication, problem solving, conflict management, service learning project, etc.) Have you applied any of the skills you have learned in the trainings and workshops? (Are there any that you have not found useful?) Can you tell us about a situation during your time in the university where you acted as a leader? How do your parents and others in the community perceive you now that you are in the LOTUS Program? Another aspect of the LOTUS is the community service and development activities; what aspects have not been fully addressed in these areas? 74 As a LOTUS student, to what extent have you been engaged and/or involved in designing and/or modifying the community services activities, or other activities? (e.g., feedback of training assessments is taken into consideration, meeting the IIE M&E specialist or any other staff) or do you think your feedback makes a difference? Study Abroad What is the added value of the Study Abroad program? (i.e. Exposure to the US culture, Academic program, Networking, University campus activities.) Did the pre-departure orientation fully prepare you for your US experience? What would you change about the experience abroad? (i.e. University choice, Academic program, Length of stay, Selection process, Administrative and visa arrangements, Student life) For those who didn’t get a chance to travel, what additional support or options could the program have provided you? Challenges What kind of challenges have you been facing in your scholarship and university? (academic support, housing, health, English, emotional support, relationships with non-LOTUS peers) What would you keep or change in the LOTUS Program for future cohort? Life after graduation What are your plans on the professional and personal levels after graduation? How does the community fit into your future plans? In your opinion, how important is the LOTUS Scholarship to improving your chances of getting a job once you graduate? In your opinion, how important is the LOTUS Scholarship to helping you advance more quickly in a job once you graduate? Do you know the source of the money for the LOTUS Scholarship? 75 Graduate Online Survey Welcome to the Survey You have been chosen to participate in this survey because you received a LOTUS Scholarship. To help make its programs more relevant and effective, USAID/Egypt has contracted our firm, Services to Improve Performance Management, Enhance Learning and Evaluation (SIMPLE) to conduct an evaluation of the Leadership Opportunity Transforming University Students (LOTUS) Scholarship Program. Your answers will provide guidance and recommendations for scholarships programs in the future. Your response is very important. It is important that we hear from everyone who received the questionnaire in order for our results to be meaningful. Answering the questions will take about 10 minutes. Please be assured that your answers will be kept strictly confidential. We will report on the findings of the survey for the group as a whole and no individual responses will be identified in the reports. 76 Online Survey - Students Graduate (Employed / Unemployed & Other) Introductory Information: Name: Marital Status (Circle one response) Not married (includes never married, divorced, widowed) Married Survey questions: Are you currently: (EQ1+EQ4) (Circle only one of the following) Wage/salary worker? self-employed? employer? unpaid worker for the family? unemployed but looking for a job? not working and not looking for a job? military? Other: specify ---- (if answer in Q1 from 4-8, skip question 2) If your answer in Question 1 is either 1, 2 or 3, how did you get your current job? (EQ1+EQ4) (Circle only one answer) From my internship Through the university or a professor Job Fairs Assistance of the LOTUS Program Personal connection (friend, family) Newspaper listing Social Media Recruitment Agencies Starting a business Other (Specify: _____________) 77 After graduation, how long did it take you to find your first job (excluding military time)? (EQ4) (Circle only one of the following) Less than one month From 1-3 months From 3-6 months From 6-12 months More than 12 moths Never employed Is your current job related to your field of study in the university? (EQ1) Yes No Not currently employed or self-employed Are you currently in any post graduate program? (EQ1+EQ4) Yes No No, but applying 5a. If YES, where? (EQ1+EQ4) Egypt Abroad (Specify country: ___________________) How satisfied or dissatisfied are you with your current job? (EQ1+EQ4) 1=Very dissatisfied 2=Dissatisfied 3=Satisfied 4=Very satisfied ____ Not currently employed or self-employed Did you have any internships during your time in the university? (EQ3+EQ4) Yes No (Skip to Q8) 7a: If YES, did any of your internships lead you directly to a permanent job after graduation? (EQ3+EQ4) Yes No 7b: If YES, do you think the internship gave you useful practical experience about the work environment? (EQ3+EQ4) Yes No 78 How useful have the following LOTUS program components been for you in developing your career? (EQ3+EQ4) Component 1=Not at all useful 2=Slightly useful 3=Useful 4=Very useful English language training Study Abroad (Skip if not applicable) Annual student-led workshops Service learning projects Nahdet el Mahrousa enrichment activities Retreats Career mentoring and counseling Career-planning training sessions and workshops Self-assessments Internships Volunteer and community service activities Other university activities and clubs Please list the three topics from the Leadership in Action curriculum that have influenced you the most and the three topics that have influenced you the least. (EQ2+EQ3+EQ4) Most influential topics Least influential topics 79 Now that you have graduated, do you maintain contact with…? (EQ1) (Check one box on each line) 1=Rarely 2=Sometimes 3=Often 4=Always Other LOTUS alumni or students Alumni from your field of study Other alumni from your university High school friends Have you done any community service during the past year? (EQ1+EQ2) Yes No If YES: 11a. Have you done any community service in the past year in your home governorate? (EQ1+EQ2) Yes No What challenges did you face as a student during your years in the university? (EQ1+EQ3) (Check the box on each line that reflects how often you faced that challenge.) Type of challenge 1=Rarely 2=Sometimes 3=Often 4=Always Need for more academic advice Classroom management Problems with instructional strategies Campus learning facilities Assignments Dealing with other students Housing Language Health Home sickness 80 Please rate your English language proficiency in reading, writing and speaking before, during, and after university. (Please put the appropriate number in each box) (EQ1+EQ3+EQ4) Time Reading Writing Speaking Not proficient 1 Somewhat proficient 2 Proficient 3 Very proficient 4 Not proficient 1 Somewhat proficient 2 Proficient 3 Very proficient 4 Not proficient 1 Somewhat proficient 2 Proficient 3 Very proficient 4 When you entered the university When you graduated from the university Now Would you recommend the LOTUS Program to your siblings or close friends? (EQ1+EQ3) Yes No I don’t know Please rate how important or unimportant was the LOTUS Scholarship to improving your chances of getting a job once you graduated? 1= Very unimportant 2= Somewhat unimportant 3=Important 4=Very Important How important or unimportant was the LOTUS Scholarship to helping you advance more quickly in a job once you graduated? 1=Very unimportant 2=Somewhat unimportant 3=Important 4=Very Important May we contact your employer to assess the skills that are important for new employees in your field? (EQ1+EQ4) 1. No 2. Yes Employer ____________ Contact __________________ Supervisor ___________ Contact ___________________ 81 PROTOCOL FOR TELEPHONE INTERVIEW WITH TERMINATED OR DISMISSED STUDENTS I am contacting you today because you received a LOTUS Scholarship. I also am aware that you left the program before graduation. To help make its programs more relevant and effective, USAID/Egypt has contracted our firm, Services to Improve Performance Management, Enhance Learning and Evaluation (SIMPLE) to conduct an evaluation of the LOTUS Scholarship Program. The purpose of the evaluation is to provide guidance and recommendations for future scholarships programs. Your observations are very important to this evaluation to give us a full understanding of the LOTUS experience. I have a few short questions that may take about 10 minutes to discuss. Please be assured that your answers will be kept strictly confidential. We will report on the findings of the evaluation for the group as a whole and no individual names or responses will be identified in the reports. QUESTIONS Where are you now? What did you do after you left the LOTUS Program? (work + university information) (EQ4) How long were you in the LOTUS Program? (reference point for the interview) [If studying] What are your plans on the professional and personal levels after graduation? (EQ4) What kind of additional support could the LOTUS Program have provided you to prevent you from leaving the program? Did you receive any warnings and additional assistance before you were dismissed? What was missing? (EQ1, EQ3) Are you friends with any LOTUS students/graduates? How do you perceive them after their years in the private university? Do you think they have changed? (EQ1+EQ2+EQ4) Do you still maintain contact with any IIE staff? (EQ1) Are there any other comments you would like to add about your experience with the LOTUS Program? Thank you for taking the time to talk with me. 82 PROTOCOL FOR TELEPHONE INTERVIEW WITH VOLUNTARILY WITHDRAWN STUDENTS I am contacting you today because you received a LOTUS Scholarship. I also am aware that you left the program before graduation. To help make its programs more relevant and effective, USAID/Egypt has contracted our group, Services to Improve Performance Management, Enhance Learning and Evaluation (SIMPLE) to conduct an evaluation of the LOTUS Scholarship Program. The purpose of the evaluation is to provide guidance and recommendations for future scholarships programs. As a person who received the scholarship but then decided to leave the program, your observations are very important to give us a full understanding of the LOTUS experience. I have a few short questions that may take about 10 minutes to discuss. Please be assured that your answers will be kept strictly confidential. We will report on the findings of the evaluation for the group as a whole and no individual names or responses will be identified. QUESTIONS Why did you decide to leave the LOTUS Scholarship program? At what point (academic year) did you withdraw? Do you still think you made the right decision for you? (EQ1+EQ3) What was the main reason that you decided to leave? What could the LOTUS Program have provided to prevent you from withdrawing? What was missing? (EQ1, EQ3) Where are you now? What did you do after you left the LOTUS Program? (work + university information) (EQ4) [If studying] What are your plans on the professional and personal levels after graduation? (EQ4) [If graduated] In your opinion, could the LOTUS Scholarship have improved your chances to get a job/get promoted quickly after graduation? [If currently employed] Are you working in your field of study? (EQ4) Can you give an example of an achievement or personal success story during your university years? (EQ2 - comparison to LOTUS students in community service and leadership) Are you friends with any LOTUS students/graduates? How do you perceive them after their years in the private university? Do you think they have changed? (EQ1+EQ2+EQ4) Do you still maintain contacts with IIE staff? How? (EQ1) Are there any other comments you would like to add about your experience with the LOTUS Program? Thank you for taking the time to talk with me. 83 GROUP INTERVIEW PROTOCOL FOR THE INSTITUTE OF INTERNATIONAL EDUCATION (IIE) The following topics related to program implementation will be discussed with IIE LOTUS staff. The Evaluation team would like to meet with the LOTUS director and the coordinators with the universities, as well as the M&E director who we met earlier. Other individuals may be brought in as appropriate, such as those involved in recruitment and selection, and individuals managing the Study Abroad selection, placement, orientation and return activities. The session with IIE also will include a telephone interview with the IIE Office in New York, including the person responsible for the US-side of the Study Abroad component. If available, the IIE MENA regional director also may be interviewed. In the introductory meeting between the evaluation team and IIE, she indicated that she would not be involved in the evaluation. General questions about the LOTUS implementation process and pre-university components: What were the lessons learned from the LOTUS recruitment experience? What was your outreach strategy? And how did you receive the applications? (We encountered minimal reference to AFS and E-ERA by students.) Differences by gender? Differences by governorate? Assessment and criteria of eligibility in terms of financial need? How did the recruitment process (and selection criteria) change over the four cohorts? In what ways, if any, has IIE applied these lessons in other programs in Egypt (LSP)? In other countries? What were the lessons learned from the LOTUS selection process? Were identical criteria applied across cohorts, governorates, and gender? How were males and females selected? For example, were they in separate pools or the same pool? What was the first criterion applied to the applications? What was the principal factor? What was the final deciding factor, if selecting between two equal candidates? What was the value of the interview for IIE? How many people and who participated in the panel? How many students were in each interview group? Were all final candidates interviewed? What proportion of the students selected, declined the scholarship? Why did they decline? Have you observed any differences among the final scholarship recipients in the four cohorts? In the last meeting you described the process for the decision on the student’s field of study. In discussion groups, some students said they were not offered any of their three preferences. Also, based on conversations with students, the process for entering nursing seems to have had some unique factors? How and why was nursing different? Was the orientation session the same across all universities? What was the agenda? Who was responsible for the session and who participated? 84 What was the nature of the interaction with the parents in this session? Did all students come accompanied by parents? What differences did you see for male and female students? Please describe the content and purpose of the contract signed by the parents at the orientation session? Did both students and parents sign? What have been the lessons learned and/or changes you have made in the orientation program over time? Thinking about the orientation as well as subsequent activities and trainings? To what extent were students involved in the planning and selection of topics? Questions about the academic program and the on-campus experience The selection of the partner universities is described in documents included in the desk review. Were your experiences with all partners equally beneficial? The Arab Academy was added more recently and, as a not-for-profit institution under the umbrella of the Arab League, it involves a different model than the other partners. Please discuss the rationale for partnering with the Arab Academy. Training to achieve proficiency in English is an objective of the LOTUS Program and an important factor in the students’ academic achievement. You have used various approaches to language training, with varying degrees of success. What is your assessment of the bridge year approach and its implementation, particularly in comparison to the other techniques LOTUS has used? Many students in the group discussions said they (and their parents) were not satisfied with giving up an entire year to English language study, and they also were dissatisfied with the training itself. The Leadership and Capacity Building component for faculty and staff was implemented and received differently across the universities. What were the IIE lessons learned from this component? What, if any, changes do you envision for the content and coverage of the sessions in the future? Questions about LOTUS after graduation, and the effectiveness and impact of this approach to achieve the overall program purpose Employability is a key objective for the scholarship program. Besides the indicators included in the monitoring plan, in what terms (qualitative as well as quantitative) do you assess the effectiveness of the LOTUS Program in this area? Given the current job market in Egypt (and the acknowledged value of contacts and networks), what do you see as the barriers to their success in the labor market? Do you see any differences for female and male graduates? LOTUS does not include any guidance or assistance to graduates for job search. Please explain how students will find jobs. Many of the students we met said they hope/plan to go abroad to work and/or study. What is your reaction to this outcome in light of the LOTUS objective to create a network of leaders committed to the development of Egypt? What can you say about gender differences in this regard? The underlying hypothesis for awarding scholarships to students from all governorates and remote communities is that these students would return to their home governorates as leaders to strengthen the human capital base in these areas and contribute to economic and social development. Is this 85 realistic, given the lack of opportunities in the LOTUS career areas? Have you seen any evidence to date of this presumed “spread effect?” How do you assess the importance, success, and impact of the Study Abroad component to date? (employability, leadership, English, commitment to development in Egypt, personal development.) To what extent do you think it creates two tiers of LOTUS graduates in terms of employability and leadership? Overall, in terms of contribution to employability and leadership, where do you place the various components in order of priority? Specifically, how do you assess the value and impact of the community service requirements in general and in terms of ties to home governorates? To date – what is the most important lesson learned for IIE in the LOTUS project and to what extent has it affected your implementation of the LSP and other programs in Egypt? Would you suggest any other actions by (a) MOHESR (b) USAID that could increase the effectiveness of LOTUS? 86 GROUP INTERVIEW PROTOCOL FOR THE INSTITUTE OF INTERNATIONAL EDUCATION (IIE) Questions for the IIE New York office and/or the IIE MENA Regional Director The following topics related to program implementation will be discussed with IIE LOTUS staff. The Evaluation team would like to meet with the LOTUS director and the coordinators with the universities, as well as the M&E director who we met earlier. Other individuals may be brought in as appropriate, such as those involved in recruitment and selection, and individuals managing the Study Abroad selection, placement, orientation and return activities. The session with IIE also will include a telephone interview with the IIE Office in New York, including the person responsible for the US-side of the Study Abroad component. If available, the IIE MENA regional director also may be interviewed. In the introductory meeting between the evaluation team and IIE, she indicated that she would not be involved in the evaluation. Specific questions about implementation of the Study Abroad component of LOTUS: What is the role of the NY office in identification of U.S. universities to receive LOTUS students? What characteristics or qualifications are you seeking? What is the role of IIE in getting the students into the specific universities (i.e., fees, support services, visas, etc.)? Do you have direct contact with the students at any point? Which expertise/areas are the most in-need in study abroad programs in recent years? What obstacles/challenges have you encountered in these placements? How, if at all, has the placement of LOTUS students differed from the placements of other study abroad programs managed by IIE? Do you monitor the students’ progress during their semester in the U.S.? Do other challenges exist in the study abroad programs such as recognition of credits or credentials, cultural adjustments, any unanticipated needs, etc.? Is the NY office involved in the return of the Study Abroad students to Egypt in terms of logistics and/or emotional/social considerations? Broad questions about LOTUS and IIE regional programming: IIE has been involved in Egypt and the MENA region for many years. In what ways does the LOTUS Program contribute to/fit into the broad IIE organizational goal, purpose, and strategy for education in this region? What lessons has IIE drawn from this experience to date relative to other IIE programs in Egypt and the region? The LOTUS Scholarship Program and the IIE implementation plan were designed before the events of 2011. In your mind, how did the changes in Egypt during and after the events of 2011 affect the implementation, results, and impact of LOTUS? 87 How, if at all, would you have modified your approach if you could have foreseen the changes that occurred? Would you suggest any other actions by (a) MOHESR (b) USAID that could increase the effectiveness of LOTUS? 88 GROUP INTERVIEW PROTOCOL FOR THE INSTITUTE OF INTERNATIONAL EDUCATION (IIE) Questions for IIE Program Coordinators The following topics related to program implementation will be discussed with IIE LOTUS staff. The Evaluation team would like to meet with the LOTUS director and the coordinators with the universities, as well as the M&E director who we met earlier. Other individuals may be brought in as appropriate, such as those involved in recruitment and selection, and individuals managing the Study Abroad selection, placement, orientation and return activities. The session with IIE also will include a telephone interview with the IIE Office in New York, including the person responsible for the US-side of the Study Abroad component. If available, the IIE MENA regional director also may be interviewed. In the introductory meeting between the evaluation team and IIE, she indicated that she would not be involved in the evaluation. QUESTIONS What is the primary responsibility of the IIE program coordinator on all levels and the relationship with students on campus? What training or other specific preparation does the program coordination receive from IIE for this position? What training or other resources does the program coordinator have to identify and deal with emotionally troubled students? What is the relationship between the IIE program coordinator and the university program coordinator? How do their roles differ and intersect? If a student has a problem, who does he/she contact first? What is the next step? How often does the IIE program coordinator visit each campus? What is the primary means of communication between IIE and the students, and between the IIE program coordinator and the students? On average, with what frequency does the program coordinator interact with each student? What mechanisms does the LOTUS Program coordinator have to monitor the students’ well-being or how they are doing in the program? At what point does the IIE coordinator become involved when a student is having a problem? Please describe the types of actions that may be taken to support students through difficulties. What are the limitations of this support? What is the role of the IIE program coordinator, if any, to students’ participation in other LOTUS program components (e.g., internships, English program, mentoring and career development, community service)? What is the process and nature of interaction with IIE and the NGO partners, especially NM? (and between the program coordinators and the NGOs?) In terms of the university LOTUS coordinator – did LOTUS have any role in identifying or training these individuals? Does the IIE program coordinator interact with the students’ academic advisors? Based on your observations, what do you think are the biggest challenges for LOTUS students? Do you think LOTUS students face different or more challenges than non-LOTUS students? Examples and ways to deal with them? What gender differences can you identify in this regard? 89 All students change and mature during their time in the university. What changes have you observed in the LOTUS students (male and female) as they advanced? How does this affect your interaction with them? Are you still in contact with any of the LOTUS graduates? How? Would you suggest any other actions by (a) MOHESR (b) USAID that could increase the effectiveness of LOTUS? If you had the chance to change anything with regards to the implementation approach to make it better, what would you change? What are the lessons learned from the LOTUS experience? 90 INTERVIEW PROTOCOL FOR THE INSTITUTE OF INTERNATIONAL EDUCATION (IIE) Manager of Leadership in Action Activities The following topics related to program implementation will be discussed with IIE LOTUS staff. The Evaluation team would like to meet with the LOTUS director and the coordinators with the universities, as well as the M&E director who we met earlier. Other individuals may be brought in as appropriate, such as those involved in recruitment and selection, and individuals managing the Study Abroad selection, placement, orientation and return activities. Questions Leadership in Action: (EQ2) What activities are included under the Leadership-in-Action component? At what point/year in the student’s time in the university do these various activities occur? What, if any, changes have you made in this scheduling since 2010? What was the process for developing the LIA curriculum for LOTUS? How (if at all) has it changed since the program began in 2010? Did you seek input and/or feedback from the students? Have any changes resulted from their input? What are the core concepts that you seek to communicate to the students? Why? To what extent have you adapted the curriculum to accommodate gender differences? What differences, if any, have you observed in the response of the young women and men to the training? Who are the trainers? (Who has face-to-face contact with the students?) Have you observed any differences in the response of students across different cohorts? Different governorates or regions? How have you responded to these differences? Please describe the logistics of the training delivery. Which activities are held on the campuses? Where are the others held? What is the process for scheduling? Some students reported long gaps between sessions, inconvenient times (e.g., during exams, in the evening), and last minute notification – with penalties for missing the activities. Why has this occurred? Internships (EQ2, EQ4) What is the process for matching the students to internships? Is it the same for male and female students? What differences have you encountered across fields of study? Have you encountered any specific problems associated with gender? Do you try to set up internships in the student’s home governorate? What are the main characteristics of a good internship? What is the average length of the internship? Does LOTUS monitor, follow-up, or independently verify placements after the students start working or when they finish? What action do you take if a student is dissatisfied with the internship – or – if an employer is dissatisfied with a student? [I think internships are self-reported. If so, there are questions about accuracy, validity, etc.] In your opinion, how does the internship affect the student’s employability? Career Development – Training, Counseling, Mentoring (EQ2, EQ3, EQ4) Is the career development training the same as the leadership training? Please describe the career development training. What are you trying to communicate to the students? How does it change as the students advance in the university? 91 How did you go about developing the curriculum? How often do you do a needs assessment or follow-up assessment with the students? How do you know whether the curriculum is producing results? To what extent have you incorporated student feedback? How, if at all, do you take gender differences into account? Please describe the self-assessment process. How do you use this in your training and counseling? What is mentoring and counseling? How often do you meet with the students? Who meets with the students? Do you collaborate with the university student support services or career development centers? How do you monitor a student’s employability as an outcome of these activities? What shortcomings or needed modifications have you identified in these components? What modifications are you planning for the future? Do you interact with any of the personnel of the universities (besides the LOTUS coordinator) on any of these activities? (e.g., counseling centers, academic advisors). What types of problems have you encountered? Other activities, including the big picture sessions, the field trips, the retreats/camps, and the annual student lead leadership conferences (EQ2, EQ3) What is the process for deciding what activities to schedule when? What to keep or drop? How has the content and programming changed since 2010? How do these contribute to the LOTUS Program and the development of leadership, employability, and personal growth of the LOTUS students? What kind of monitoring and/or evaluation have you done for these various components both in terms of students’ reaction to them and in terms of their results or impacts? How do you assess their value in comparison to other LOTUS components? If you had to assign priorities to any of these activities, and decide what to keep, add, or drop – how would you do it? Did the students collaborate in the identification, planning, and/or implementation of these activities? Did they provide feedback? Do you see any direct relationship between the component for capacity building of faculty and staff and these activities for students? Is there any connection between the leadership training for the two groups, for example? 92 GROUP INTERVIEW PROTOCOL FOR PARTNER NGO NAHDET EL MAHROUSA The purpose of this meeting is to discuss your role in the USAID LOTUS Scholarship Program. To help make its programs more relevant and effective, USAID/Egypt has contracted our group, Services to Improve Performance Management, Enhance Learning and Evaluation (SIMPLE) to conduct an independent, mid-term evaluation of the LOTUS Program. USAID will utilize the results of this evaluation to strengthen the current program in the remaining three years and to inform the design of future scholarship activities. In addition to the university and NGO partner LOTUS program personnel, we are interviewing IIE and USAID managers, and we are surveying current LOTUS students and graduates. Some of these students and graduates also are participating in group discussions with the evaluators. Please be assured that your answers will be kept strictly confidential. We will report on the findings of the evaluation for the group as a whole and no individual names or responses will be identified. Your organization has been responsible for several key components of the LOTUS Programs and your perceptions and observations are very important to capture a complete picture of the students’ experience. Questions Turning first to the Leadership-in-Action training: (EQ2) What was the process for developing the LIA curriculum for LOTUS? How (if at all) has it changed since the program began in 2010? Did you seek input and/or feedback from the students? Have any changes resulted from their input? What are the core concepts that you seek to communicate to the students? Why? To what extent have you adapted the curriculum to accommodate gender differences? What differences, if any, have you observed in the response of the young women and men to the training? Have you observed any differences in the response of students across different cohorts? Different governorates or regions? How have you responded to these differences? Please describe the logistics of the training delivery. Do the students go to NM or do you go to the universities? Some students reported long gaps between sessions. Why has this occurred? Career Development – Internships (EQ2, EQ4) At the beginning of the program, NM encountered problems in setting up internships for the students. What actions did you take to resolve these problems? What is the process now for matching the students to internships? Is it the same for male and female students? Have you encountered any specific problems associated with gender? Do you try to set up internships in the student’s home governorate? What are the main characteristics of a good internship? What is the average length of the internship? Does NM monitor or follow-up on placements after the students start working? What action do you take if a student is dissatisfied with the internship? In your opinion, how does the internship affect the student’s employability? Career Development – Training, Counseling, Mentoring (EQ2, EQ3, EQ4) 93 Please describe the career development training. What are you trying to communicate to the students? How does it change as the students advance in the university? How often do you do a needs assessment or follow-up assessment with the students? To what extent have you incorporated student feedback? How, if at all, do you take gender differences into account? Please describe the self-assessment process. How do you use this in your training and counseling? What is mentoring and counseling? How often do you meet with the students? Who meets with the students? Do you collaborate with the university student support services or career development centers? How do you monitor a student’s employability as an outcome of this component? What shortcomings or needed modifications have you identified in these components? What modifications are you planning for the future? How do you work with IIE and the universities as you develop and implement your activities? What types of problems have you encountered? Are there other NM activities under LOTUS, such as enrichment programs, trips, retreats and annual conferences? (EQ2, EQ3) How do these contribute to the LOTUS Program and the development of leadership, employability, and personal growth of the LOTUS students? How do you assess their value in comparison to other LOTUS components? Did the students collaborate in the identification, planning, and/or implementation of these activities? Did they provide feedback? What are your observations about the LOTUS students compared to other students in their universities in terms of: (EQ1) Strengths and challenges in leadership and employability? and Adaptation to the university community and interactions with peers and faculty? What differences have you observed between the young women and men in these terms? What differences have you seen among the cohorts? What recommendations do you have for the current LOTUS program, and for future USAID scholarship programs? What are the principal barriers or obstacles that you have encountered in implementing your program? How has NM overcome these obstacles -or – what recommendations do you have to remove them? How do you describe the impact of the NM programs on the LOTUS students in terms of personal development, leadership, commitment to sustainable development, and responsible citizenry? [Does the LOTUS Program contribute to the NM organizational goals?] Have you replicated any of these activities for other groups of students or youth? 94 GROUP INTERVIEW PROTOCOL FOR PARTNER NGOs RECRUITMENT AND SELECTION The purpose of this meeting is to discuss your role in the USAID LOTUS Scholarship Program. To help make its programs more relevant and effective, USAID/Egypt has contracted our group, Services to Improve Performance Management, Enhance Learning and Evaluation (SIMPLE) to conduct an independent, mid-term evaluation of the LOTUS Program. USAID will utilize the results of this evaluation to inform the design of future scholarship activities. In addition to the university and NGO partner LOTUS program personnel, we are interviewing IIE and USAID managers, and we are surveying current LOTUS students and graduates. Some of these students and graduates also are participating in group discussions with the evaluators. Please be assured that your answers will be kept strictly confidential. We will report on the findings of the evaluation for the group as a whole and no individual names or responses will be identified. Your perceptions and observations are important to capture a complete picture of the LOTUS Program implementation. We appreciate your willingness to meet with us even though your role in the program ended several years ago. QUESTIONS (EQ1 and objectives 1-3) Please describe your recruitment process for the LOTUS Scholarship program. What was the first step in the governorate? Did your approach differ by governorate or region? In what ways? Why or why not? Did your approach to recruitment differ for young men and women? What gender differences, if any, did you encounter in terms getting the message to them, convincing them to apply, interactions with their parents, or other differences? Did this vary by governorate or region? What were the most successful/least successful methods of recruitment? How did this differ by governorate? How did your methods change across the four cohorts? Did you encounter any unique challenges in reaching and selecting students in particularly remote areas and those with strong financial needs? What methods did you use to overcome these challenges? What was the process for receiving applications? Did you assist students in understanding and fulfilling the application requirements? Did you allow any flexibility in the process? (e.g., due dates, second chance to get all the required documents) Why or why not? What proportion of the applications you received met the minimal requirements to continue into the selection process? What was the major shortcoming? Was there any difference by gender? How did you define and measure financial need? What criteria did you use? What was the process for verifying that the information submitted with the application was correct? Moving to the selection process: What was the first criterion applied to narrow down the pool of candidates? Did you evaluate the male and female candidates separately? 95 Please explain the interview process? How many people were in a group? How did you form the groups? How many people were on the interview panels? What exactly did you learn from these interviews? If two candidates were very similar at the end of the process, what was the final deciding factor? The home visit was the last step in the selection process? Did you reject candidates on the basis of the home visit? Can you identify any lessons that you learned in the recruitment and selection of candidates specific to the LOTUS Scholarship? Did you modify your procedures in any way over the four cohorts based on these lessons? Are you now applying any of these methods in any other programs? Please describe the counseling that candidates received in choosing a field of study and a university? Who did the counseling? What involvement did the candidate’s parents have in these decisions? In your opinion, how successful were you in identifying and enrolling students that match the profile to be successful in the LOTUS Program? Are there additional factors that should be considered in the future? What do you see as the defining characteristic of the LOTUS Program? What recommendations do you have to improve the recruitment and selection process? 96 INTERVIEW PROTOCOL FOR USAID/OET LOTUS PROGRAM MANAGER I would like to begin by talking about the enthusiasm for the LOTUS Program among the people directly involved in it, its increasing recognition as a good scholarship program (according to the recruiters), the enthusiasm of the current and graduated students for recommending it in their communities and with their friends, and the affirmation of the faculty of the fact that many of these students stand out on campus because they are high quality students. And then move to some of the questions that have come up in our discussions and in our key informant and group interviews. 1. Discussion of the shift in the LOTUS Program between Cohorts 1 and 2, and Cohorts 3 and 4. The changes we have observed are: - a shift in the scope of the recruitment; - a decrease in the number of universities included; - the introduction of the bridge year program; - the elimination of five-year fields of study (e.g., engineering and dentistry; - the elimination of the Nahdet el Mahrousa activities; and, - the addition of the service learning projects. - Modification in the cost-share among the partner universities e.g. BUE were cost sharing with 30% and decreased the percentage to 25% for Cohort 3 and 15% to Cohort 4. - Provision of internships’ allowance Why were these changes made? Do they all fit together into a new design or were there different reasons for each of them/ Did analysis of the monitoring data, or any special studies or evaluations contribute to these decisions? Are these changes in any way related to the development of the new LSP? 2. What was the rationale for the selection of the original five universities? i.e. their strengths at the selection time. They tend to be new institutions with varying reputations in term so quality of education. - Some LOTUS students have reported that they have experiences harassment or resentment from other students because they are there and getting good marks, and their parents don’t have to pay anything. - Some LOTUS students have commented that the education at a particular school is inferior compared to other universities, especially the most prominent public universities. - An empirical question that has been raised (that we will not be able to answer) is whether this selection affects the competitiveness of the LOTUS graduates on the job market in Egypt? - The universities have told us that they like the LOTUS Program because it has brought them high quality students that have challenged the other students in the university. 3. Another question related to employability is the basis for the selection of the target career areas. To what extent do you see this as compatible with the objective to have the graduates return to their home governorates to serve as leaders and to stimulate economic growth. - In some of the newer career areas (e.g., logistics) students do not even see opportunities in Egypt right now. They say they will need to travel abroad to find employment in their field. 97 - Do you think LOTUS could/should assist graduates in finding a job or creating other opportunities for themselves? e.g. one suggestion was to provide training and financial support for entrepreneurship and business start-ups, or for developing and getting funding for projects. - How does nursing fit into this set of careers? Nursing as a professional degree program is new in Egypt and the graduates may not be eligible to work in public institutions? 4. We have heard a lot about the English language component from students and to a lesser extent from staff, and the emphasis on English is one of the differences between public and private universities. What was the basis for the decisions and shifts among the various providers – especially BUE, Amideast and AUC. Was there any monitoring or evaluation of the programs and their results? - Students tend to blame the courses and instructors for deficiencies and program managers tend to blame the students. Were objective standards used in designing the changes? 5. To what extent have you and others from USAID had direct contact with the universities and/or the students? There is a high level of recognition among students and implementing organizations that this is USAID funded. And yet, there has been no sense that USAID is involved in any way except funding. Is this intentional? - [We could mention the questions (and suspicions) reported, especially from parents and others in the communities about why the US is giving all this money to their children. Is there another motive besides education and leadership?] 98 1- GROUP INTERVIEW PROTOCOL FOR USAID/OET PROGRAM MANAGMENT The questions directed to the USAID LOTUS Program Manager focus on positioning this project within the strategy of USAID and follow-on programs. This discussion is intended to increase the evaluation team’s understanding of the LOTUS objectives within the broader USAID and GOE context. The team also will use this interview to get background on the background and timeline of LOTUS from the USAID perspective. Questions 1. The LOTUS Scholarship Program began in 2010, just before the events of 2011. How, if at all, were the design and subsequent modifications affected by those events and the changes since then? 2. USAID/Egypt has had activities in higher education for many years. In your mind, what sets LOTUS apart or distinguishes it from past USAID higher education scholarship activities and others that are operating now? 3. How does LOTUS complement/compare with other scholarship funds? 4. The LOTUS Program gives a high priority to gender parity in awarding scholarships. What has been the reaction to this requirement from students, universities, or government? What gender￾related challenges have you encountered in implementing this requirement? 5. The LOTUS Scholarship Program is directed to private universities. The Arab Academy, which was added in 2014, is different from the original five private universities in that it is a non￾governmental, not-for-profit institution under the umbrella of the Arab League, focused on the Arab region as a whole rather than Egypt. What is the rationale for inclusion of the Arab Academy? 6. The development hypothesis for the LOTUS Scholarship Program links the nationwide selection and education of young talented leaders to nationwide development and growth as these leaders return to their home governorates to be agents of change. A broad spectrum of individuals we have interviewed have questioned this hypothesis. Could you discuss the dynamic underlying this expected impact? Do you believe the hypothesis is accurate or is there another hypothesis to the LOTUS Program? 7. How do you assess your relationship with MOHESR in the context of LOTUS program? 99 INTERVIEW PROTOCOL FOR UNIVERSITY PRESIDENT The purpose of this meeting is to discuss the experience of [name of university] as a partner in the USAID LOTUS Scholarship Program. To help make its programs more relevant and effective, USAID/Egypt has contracted our group, Services to Improve Performance Management, Enhance Learning and Evaluation (SIMPLE) to conduct an independent, mid-term evaluation of the LOTUS Program. USAID and IIE will utilize the results of this evaluation both to enhance the program’s effectiveness during the remaining three years and to inform the design of future scholarship activities. In addition to university and LOTUS program personnel, we are surveying current LOTUS students and graduates. Some of these students and graduates also are participating in group discussions with the evaluators. We also are interviewing IIE and USAID managers, and the LOTUS NGO partners involved in selection, and in leadership training and community service. Questions Why is [name of the university] participating in the LOTUS Program? What are the benefits to the university? [EQ 1] What has the university learned from the experience? [lessons learned; EQ 1] What are the problems that the university has encountered with LOTUS? For example: with program management/funding, faculty and staff, or students? [EQ 1; EQ 4] What program components have been most challenging for meeting the LOTUS Program objectives with these students? Examples: English language, academic weaknesses, social adjustments. [EQ 1} After LOTUS ends, do you anticipate that [your university] will continue some type of scholarship program focused on exceptional high school students from outlying governorates who lack the financial resources to attend this university? Why or why not? Do you think these students stand out among the student population as a result of the additional training and activities they receive? [sustainability; long term impact] How do the members of [the university] Board of Directors assess the LOTUS Program to date? [sustainability; long term impact] The underlying purpose of the LOTUS Scholarship Program is to contribute to a better educated workforce that responds to Egypt’s labor market needs. How do you assess the direct benefits of this approach for the Egyptian economy and society, for the private sector, for the public sector? [sustainability – creating a network; development hypothesis and LOTUS purpose] What recommendations do you have for the remaining years of the LOTUS Scholarship Program and for new scholarship programs in the future? 100 GROUP INTERVIEW PROTOCOL FOR FACULTY AND STAFF TRAINED IN LEADERSHIP DEVELOPMENT The purpose of this meeting is to discuss your role in the USAID LOTUS program in [name of university]. To help make its programs more relevant and effective, USAID/Egypt has contracted our group, Services to Improve Performance Management, Enhance Learning and Evaluation (SIMPLE) to conduct an independent, mid-term evaluation of the LOTUS Program. USAID will utilize the results of this evaluation both to enhance the program’s effectiveness during the remaining three years and to inform the design of future scholarship activities. Your observations are important to the evaluation because you interact with and support the LOTUS students academically and on campus. In addition to university and LOTUS program personnel, we are surveying current LOTUS students and graduates. Some of these students and graduates also are participating in group discussions with the evaluators. We also are interviewing IIE and USAID managers, and the LOTUS NGO partners involved in selection, and in leadership training and community service. Please be assured that your answers will be kept strictly confidential. We will report on the findings of the evaluation for the group as a whole and no individual names or responses will be identified. Questions Please describe your role and activities in the LOTUS Program. What kind of interaction do you have with the individual students? How often do you interact with them? [EQ1 and EQ3] The Leadership Development Training for the university is an innovative component of the LOTUS Scholarship Program. (EQ1) How many training sessions did you attend? What were the topics of these sessions? Were you invited to attend the training or did you take the initiative to join them? How do you assess the training? What aspects of the training did you find useful? What aspects were less useful for you? Can you give an example of how you have applied the training lessons? Have you applied any of the training lessons in your work with the LOTUS students and program? How do you assess the training methods and materials? Would you like to participate in more sessions? On what topics? How do you assess the value of this type of training for the university as an institution? Do you recommend that it should be offered to more faculty and staff? What recommendations do you have for IIE to strengthen or expand the program? Based on your experience with the students in the LOTUS Program, what are the major challenges that these students face in [this university]? (e.g., academic program, housing, homesickness, assignments, LOTUS activities, interactions in the university community.) (EQ1, EQ3) One area of particular interest for the evaluation is English language training, since many of the LOTUS students were behind their peers in this area when they entered the university. How do you assess their progress in this area? In your opinion, is English proficiency more of a problem for LOTUS 101 students than for others in the university? Do you have suggestions on how to improve the English program? What aspects of the training seem to have been most effective? [EQ1] From your point of view, do you think the LOTUS students stand out in the university community as leaders and active community members? (EQ1, EQ2) Do you think the LOTUS activities distinguish the LOTUS students as job seekers when they graduate? Do you see any differences between the young men and women in terms of employability? (EQ1, EQ4) Do you maintain contact with any graduates of the LOTUS Program at [this university]? (EQ1) What recommendations do you have to make the LOTUS Program more effective at [this university]? (EQ1, EQ3) 102 ANNEX IV: SOURCES OF INFORMATION AND LISTS OF PARTICIPANTS This annex includes: (1) A list of all project-related documents reviewed for this evaluation; (2) A list of persons interviewed; and (3) Data collection sites and activities. I. PROJECT DOCUMENTS IIE Annual Progress Report – FY2014 (October 1, 2013-September 30, 2014) IIE Annual Progress Report – FY2015 (October 1, 2014-September 30, 2015) IG Audit of USAID/Egypt’s New Scholarship Program, May 29, 2014 LOTUS M&E Matrix June 14, 2015 Revised Indicator Tracking Sheet FY2015 Q3 LOTUS Implementing Partner (IIE) Implementation Data Collection Tools New Scholarship Program Cooperative Agreement 263-A-00-10-00026-00 New Scholarship Program Request for Applications (RFA) 05 Nov 09 Modification 9 to New Scholarship Program Cooperative Agreement LOTUS Scholarship Program Annual Work Plan (October 1, 2014-September 30, 2015) Revised December 14, 2014 LOTUS Scholarship Program Quarterly Report: FY 2014, Quarter 4 LOTUS Scholarship Program Quarterly Report: FY 2015, Quarter 1 LOTUS Scholarship Program Quarterly Report: FY 2015, Quarter 2 LOTUS Scholarship Program Quarterly Report: FY 2015, Quarter 3 LOTUS Scholarship Program Quarterly Report: FY 2015, Quarter 4 LOTUS Scholarship Program Quarterly Report: FY 2016, Quarter 1 LOTUS Scholarship Program Comparison Group Midterm Evaluation Report 2014 LOTUS Final M&E Procedure Manual, June 26, 2014 Egypt Labor Market Assessment, USAID-IBTCI September 29, 2015 LOTUS Third Cohort Program Description, USAID, nd LOTUS M&E Indicator Definition All Cohorts 2015 Ragui Assaad and Caroline Kraft., Youth Transitions in Egypt: School, Work, and Family Formation in an Era of Changing Opportunities, October 2014 103 II. LIST OF PERSONS INTERVIEWED Date Title of Persons Interviewed Institution May 4, 2016 President AASTMT May 4, 2016 Head of International Relations and Agreements, Associate Professor, Communication and Electronic Engineering, Faculty of Engineering and Technology Coordinator/LOTUS AASTMT May 4, 2016 Head of the Public Relations Department AASTMT May 5, 2016 Vice-President for International Relations & Quality Assurance, Professor of Pharmaceutics PUA May 5, 2016 LOTUS Coordinator PUA May 5, 2016 LOTUS Assistant Coordinator PUA May 8, 2016 LOTUS Coordinator MSA May 8, 2016 LOTUS Assistant Coordinator MSA May 10, 2016 Professor of Structural Engineering, Dean of Engineering Coordinator/LOTUS BUE May 10, 2016 Associate Professor, Mechanical Engineering, Faculty of Engineering, Director, Center for Community & Consulting Services (CCCS) BUE May 10, 2016 Assistant Coordinator/LOTUS BUE May 10, 2016 Vice President for Research and Postgraduate Studies BUE May 10, 2016 Associate Professor, International Business, Acting Vice Dean, Teaching and Learning, Faculty of Business, Economics & Political Science May 11, 2016 Dean of Engineering, LOTUS Coordinator FUE May 11, 2016 LOTUS Assistant Coordinator FUE May 12, 2016 President ACU May 12, 2016 LOTUS Coordinator ACU May 14, 2016 Program Coordinator E-ERA May 14, 2016 LOTUS Coordinator E-ERA May 14, 2016 National Director AFS May 14, 2016 HEI-LSP Program Manager AFS Note: The above list includes only KII and/or GI session participants. Group discussions involving students are not included in this list as they were granted confidentiality for their testimonies. 104 III. SITE VISITS AND DATA COLLECTION ACTIVITIES The evaluation team visited each of the six university campuses twice. During the first visit, on April 26 and 27, 2016, the evaluators administered the student survey and conducted group discussions. The second visit took place during weeks of May 1 and May 8. The team conducted interviews with university presidents, LOTUS university coordinators, and faculty trained by IIE. They collected additional student surveys and held group discussions at AASTMT and BUE. Four of the universities are located in the Cairo governorate. • Ahram Canadian University (ACU) in Sixth of October City • Modern Sciences and Art University (MSA), Sixth of October City • Future University of Egypt (FUE), New Cairo • British University in Egypt (BUE), El Sherouk City Two of the universities are located in Alexandria. • Pharos University in Alexandria (PUA), Qism Sidi Gabir • Arab Academy of Science, Technology, & Marine Transport (AASTMT), Alexandria LOTUS Currently Enrolled Students (Only), Two-Day Data Collection Schedule University Name Time Slot Duration Session Type & Team Names Day 1 Day2 26-Apr 27-Apr AASTMT/Alexandria 45 currently enrolled students 2 paper-and-pencil based surveys 2 ten-person group discussions (one mixed gender and one female only) 9:00AM – 11:00PM Session Survey 1 GD 1 Team Amaal Refaat Virginia Lambert Ola Hussein Mahmoud Kamel Ah d Gb 11:00AM – 1:00PM Session Survey 2 GD 2 Team Amaal Refaat Ahmed Gabr Virginia Lambert Ola Hussein Mahmoud Kamel Ah d Gb 1:00PM – 3:00PM Session (Survey 3) (FGD 3) Team Amaal Refaat Ahmed Gabr Virginia Lambert Ola Hussein Mahmoud Kamel Ahmed Gabr 105 Pharos/Alexandria 11 currently enrolled students 4 graduated* and doing external residencies (Cairo) 1 paper - and-pencil based survey 1 ten-person group discussion (one mixed gender) 9:00AM – 11:00PM Session Survey 1 Team Ola Hussein Ahmed Gabr 11:00AM – 1:00PM Session GD 1 Team Virginia Lambert Ola Hussein Mahmoud Kamel 1:00PM – 3:00PM Session (Survey 2) Team Ola Hussein Future/New Cairo 16 currently enrolled students 3 graduates* doing residency at FUE 1 paper-and-pencil based survey 1 ten-person group discussion (one mixed gender) 9:00AM – 11:00PM Session Survey 1 Team Farouk Salah Sherine Saber 11:00AM – 1:00PM Session GD 1 Team Farouk Salah Sherine Saber 1:00PM – 3:00PM Session (Survey 2) Team Farouk Salah Sherine Saber MSA/6th October 15 currently enrolled students 2 graduated* and doing external residencies (Cairo) 1 paper - and - pencil based survey 1 ten-person group discussion (one mixed gender) 9:00AM – 11:00PM Session Survey 1 Team Osama Radwan 11:00AM – 1:00PM 1:00PM – 3:00PM Session FGD1 Team Youmna Khalil Doaa Abdel-Aal 106 Ahram Canadian/ 6th October 6 currently enrolled students 1 paper-and-pencil based survey 1 ten-person group discussion (one mixed gender) 9:00 –10:30AM Session Survey1 Team Youmna Khalil Doaa Abdel- Aal 10:45AM – 12:30PM Session GD 1 Team Youmna Khalil Doaa Abdel- Aal 1:00PM – 3:00PM Session Team British University/ Shourouk 66 currently enrolled students 2 paper-and-pencil based surveys 2 ten-person group discussions (one mixed gender and one female only) 9:00– 10:30AM Session Survey 1 GD 1 Team Wael Karim Abdel Youmna Khalil Doaa Abdel-Aal 10:45AM – 12:30PM Session Survey 2 GD 2 Team Wael Karim Abdel Youmna Khalil Doaa Abdel-Aal 1:00PM – 3:00PM Session (Survey 3) (GD 3) Team Wael Karim Abdel Youmna Khalil Doaa Abdel-Aal Notes: Parentheses represent alternative time frames for completing the required number of sessions. * All graduated students who continue as residents within a program of study supported by LOTUS funding are considered currently enrolled students. As such, they were required to complete the paper-and-pencil survey at the participating university at a time most convenient to their schedules. 107 Meetings with Key Stakeholder Organizations Stakeholder Entity Name Time Frame Duration Session Type Targeted Participants Team Members AASTMT/Alexandria President’s Office 10:00 – 10:45 AM Key Informant group Interview President, AASTMT University - Virginia Lambert - Ola Hosny - Mahmoud Kamel - Ahmed Gabr CONFIRMED 11:00AM – 1:00PM Group Interview 1 LOTUS University Coordinator and LOTUS Student Support Services Personnel - Virginia Lambert - Ola Hosny - Mahmoud Kamel - Ahmed Gabr Dorm Building D (same as last meetings) ADVISING STUDENTS 1:00PM – 2:00PM Student Make-Up Meeting and GROUP DISCUSSION Meeting All students who missed April 26 Meeting session - Ahmed Gabr 2:00PM – 3:30PM GROUP DISCUSSION 1 All students who missed April 27 GROUP DISCUSSION session - Virginia Lambert - Ola Hosny - Mahmoud Kamel - Ahmed Gabr 108 Thu, 5 May 2016 Pharos/Alexandria Administration Building, 3rd floor 9:00AM – 10:30AM Key Informant group Interview Vice President, PUA - Virginia Lambert - Ola Hosny - Mahmoud Kamel - Ahmed Gabr Finance and Admin. Sciences Building, Room D019 – Ground Floor 10:30AM – 12:30PM Group Interview 1 LOTUS University Coordinator and LOTUS Student Support Services Personnel - Virginia Lambert - Ola Hosny - Mahmoud Kamel - Ahmed Gabr Finance and Admin. Sciences Building, Room D019 – Ground Floor 12:30PM – 2:00PM Group Interview 2 University staff and/or faculty who participated in the LOTUS Leadership Capacity Building for Partner Universities - Virginia Lambert - Ola Hosny - Mahmoud Kamel - Ahmed Gabr Finance and Admin. Sciences Building, Room D019 – Ground Floor ADVISING STUDENTS 2:00PM – 3:00PM Student Make-Up Meeting and GROUP DISCUSSION Meeting All students who missed April 26 Meeting session - Ahmed Gabr 3:00PM – 4:30:00 PM GROUP DISCUSSION 1 All students who missed April 27 GROUP DISCUSSION session - Virginia Lambert - Ola Hosny - Mahmoud Kamel - Ahmed Gabr Sun, 8 May 2016 MSA/6th October PRESIDENT UNAVAILABLE DUE TO VALIDATION UNIVERSITY MEETINGS 9:00AM - 10:30AM Key Informant Interview President, Modern Sciences and Arts University - Virginia Lambert - Ola Hosny - Mahmoud Kamel - Ahmed Gabr 109 TRYING TO CONFIRM THIS MEETING BUT MANY OF THE FACULTY HAVE LEFT 10:30AM – 12:30PM G Ii 1 University staff and/or faculty who participated in the LOTUS Leadership Capacity Building for Partner Universities activity - Virginia Lambert - Ola Hosny - Mahmoud Kamel - Ahmed Gabr Confirmed 12:30PM – 2:00PM Group Interview 2 LOTUS University Coordinator and LOTUS Student Support Services Personnel - Virginia Lambert - Ola Hosny - Mahmoud Kamel - Ahmed Gabr Confirmed Room E111 2:00PM – 3:00PM Student Make-Up Meeting and GROUP DISCUSSION Meeting - All students who missed April 26 Meeting session - Ahmed Gabr 3:00PM – 4:30PM GROUP DISCUSSION 1 All students who missed April 27 GROUP DISCUSSION session - Virginia Lambert - Ola Hosny - Mahmoud Kamel - Ahmed Gabr Thursday, 12 May 2016 Ahram Canadian/6th October CONFIRMED 9:00AM – 10:30AM Key Informant group Interview President, Ahram Canadian University - Virginia Lambert - Ola Hosny - Mahmoud Kamel - Ahmed Gabr 110 Ahram Canadian/6th October CONFIRMED 10:30AM – 12:30PM Group Interview 1 LOTUS University Coordinator and LOTUS Student Support Services Personnel - Virginia Lambert - Ola Hosny - Mahmoud Kamel - Ahmed Gabr Ahram Canadian/6th October CONFIRMED 12:30PM – 2:00PM Group Interview 2 University staff and/or faculty who participated in the LOTUS Leadership Capacity Building for Partner Universities Activity - Virginia Lambert - Ola Hosny - Mahmoud Kamel - Ahmed Gabr Ahram Canadian/6th October CONFIRMED 2:00PM – 3:00PM Student Make-Up Meeting and GROUP DISCUSSION Meeting All students who missed April 26 Meeting session Ahmed Gabr 3:00PM – 4:30PM GROUP DISCUSSION1 All students who missed April 27 GROUP DISCUSSION session - Virginia Lambert - Ola Hosny - Mahmoud Kamel - Ahmed Gabr Tue, 10 May 2016 British University/El Shurouk City CONFIRMED 9:00 – 10:30AM Key Informant group Interview President, British University in Egypt - Virginia Lambert - Ola Hosny - Mahmoud Kamel - Ahmed Gabr 111 British University/El Shurouk City CONFIRMED Deans office, Faculty of Engineering, room 117 10:45AM – 12:00PM Group Interview 1 LOTUS University Coordinator and LOTUS Student Support Services Personnel - Virginia Lambert - Ola Hosny - Mahmoud Kamel - Ahmed Gabr British University/El Shurouk City 12:00PM – 1:30PM Group Interview 2 University staff and/or faculty who participated in the LOTUS Leadership Capacity Building for Partner Universities Activity - Virginia Lambert CONFIRMED Faculty of Engineering A, Room 118 - Ola Hosny - Mahmoud Kamel - Ahmed Gabr Capacity Building for Partner Universities Activity British University/El Shurouk City CONFIRMED Faculty of Engineering A, Room 118 1:30PM– 2:30PM Student Make-Up Meeting and GROUP DISCUSSION Meeting All students who missed April 26 Meeting session Ahmed Gabr 2:30PM– 4::00 PM GROUP DISCUSSION 1 All students who missed April 27 GROUP DISCUSSION session - Virginia Lambert - Ola Hosny - Mahmoud Kamel Wed, 11 May 2016 Future University/New Cairo PRESIDENT OUT OF COUNTRY 9:00AM –10:30AM Key Informant group Interview President, Future University in Egypt - Virginia Lambert - Ola Hosny - Mahmoud Kamel - Ahmed Gabr British University/El Shurouk City CONFIRMED 10:30AM – 12:00PM Group Interview 1 University LOTUS Coordinator and LOTUS Student Support Services Personnel - Ola Hosny - Ahmed Gabr - Virginia Lambert - Mahmoud Kamel 112 British University/El Shurouk City CONFIRMED 12:00PM – 1:30PM Group Interview 2 University staff and/or faculty who participated in the LOTUS Leadership Capacity Building for Partner Universities Activity - Ola Hosny - Ahmed Gabr - Virginia Lambert - Mahmoud Kamel British University/El Shurouk City CONFIRMED 1:30PM￾2:30PM Student Make-Up Meeting and GROUP DISCUSSION 113 9:00AM - –10:30AM Meeting All students who missed April 26 Meeting session Ahmed Gabr 2:30PM – 4:00 PM GROUP DISCUSSION 1 All students who missed April 27 GROUP DISCUSSION session - Virginia Lambert - Ola Hosny - Mahmoud Kamel - Ahmed Gabr Sun,12 May, 2016 Nahdet El Mahrousa 2:00PM – 4:00PM Group Interview Nahdet El Mahrousa LOTUS Project Team Sat, 14 May 2016 Egyptian Society for Intercultural Exchange (AFS Egypt) Dokki Egyptian Educational Resource Association (E-ERA) 9:00AM – 11:00AM Key Informant Interview AFS Egypt LOTUS Project Team - Virginia Lambert - Ola Hosny 1:00PM– 3:00PM CONFIRMED Group Interview E-ERA LOTUS Project Team - Virginia Lambert - Ola Hosny Tue, 17 May, 2016 Institute of International Education – Middle East and North Africa Regional Office 1:00PM – 4:00PM Group Interview IIE-MENA LOTUS Project Team IIE MENA Director - Virginia Lambert - Ola Hosny - (Mahmoud Kamel) - Ahmed Gabr Thu, 19 May, 2016 USAID Mission, Maady 1:00PM – 3:00PM - Virginia Lambert - Ola Hosny - Ahmed Gabr USAID LOTUS Management Team (OET) 114 ANNEX V: DISCLOSURE OF ANY CONFLICTS OF INTEREST The Disclosure of any Conflict of Interest can be found on file with the COR. 115 ANNEX VI: DESCRIPTIVE STATISTICS AND STATISTICAL ANALYSES Part I: General Descriptive Statistics of LOTUS Cohorts TABLE VI.1: GENERAL CHARACTERISTICS OF SCHOLARSHIP RECIPIENTS DISAGGREGATED BY COHORT COHORT 1 1.2 2 3 4 TOTAL Total no. students 44 56 50 50 50 250 No. of females 25 26 30 28 28 137 No. of governorates represented Note 1 26 28 20 23 __ No. of fields of study Note 1 16 12 13 15 __ Financial need (% strong need or greater*) Note 1 68 58 78 Note 2 __ Avg. Thanaweya Cumulative Amma (%) Note 1 89.26 93.98 89.14 Note 2 __ Source: IIE Demographic Data on Students by Cohort *Financial need was coded into five categories based on monthly income, assets, and size of household: (1) exceptional financial need, (2) very strong financial need, (3) strong financial need, (4) financial need, and (5) cannot afford private university tuition but need is marginal compared to other applicants. Note 1: Data for Cohort 1 are not included in IIE database, Demographic Data on Students by Cohort. File begins with Cohort 1.2 Note 2: Incomplete listing of Cohort 4 students in IIE database, Demographic Data on Students by Cohort. TABLE VI.2: NUMBER OF STUDENTS ENROLLED IN EACH UNIVERSITY BY COHORT COHORT UNIVERSITY 1 1.2 2 3 4 TOTAL AASTMT __ __ __ 21 26 47 ACU 9 14 7 __ __ 30 BUE 14 7 9 29 24 83 FUE 6 6 16 __ __ 28 MSA 7 19 11 __ __ 37 PUA 8 10 7 __ __ 25 TOTAL 44 56 50 50 50 250 Sources: IIE Demographic Data of Students by Cohort; FY2014 and FY2015 Annual Reports. TABLE VI.3: NUMBER OF LOTUS SCHOLARSHIP RECIPIENTS IN COHORTS 1, 2, AND 3 BY GPA SCORES Cohorts 1 and 2 Cohort 3 GPA Fall 2014 Spring 2015 Fall 2014 Spring 2015 4.00 2 3 0 1 3.50-3.99 8 10 21 26 3.00-3.49 17 13 24 7 2.50-2.99 9 13 5 13 2.00-2.49 5 3 0 3 2.00 or less 0 4 0 0 Sources: FY 2015 Annual Report, page 9, n=41 and n=46 (Cohorts 1 and 2); FY 2015 Annual Report, page 16, n=50 (Cohort 3). 116 PART II: Percent of IIE LOTUS Targets Achieved Objective 1: Two hundred and fifty financially disadvantaged female and male youths from 27 governorates are enrolled in undergraduate programs of study in private Egyptian universities. IIE has met 100 percent of its enrollment targets for LOTUS. LOTUS has successfully enrolled 250 financially disadvantaged female and male youths from all 27 governorates in undergraduate programs of study in private Egyptian universities. A total of 100 students were enrolled in Cohort 1 and 50 students in each of the remaining cohorts (2, 3, and 4). See Table VI.4 below. TABLE VI.4: DISTRIBUTION OF FINANCIALLY DISADVANTAGED YOUTH ENROLLED IN LOTUS UNDERGRADUATE STUDIES BY COHORT Targets % of Target Achieved Year 5 Cumulative Targets Year 5 Cumulative Actuals No. Females No. Males 1d: No. of Cohort 1 scholarships awarded/ students enrolled 100 100 100 51 49 1e: No. of Cohort 2 scholarships awarded/ students enrolled 100 50 50 30 20 1.1c: No. of Cohort 3 scholarships awarded/ students enrolled 100 50 50 28 22 1.2c: No. of Cohort 4 scholarships awarded/ students enrolled 100 50 50 28 22 Overall Objective 1: Enrollments 100 250 250 137 113 Source: LOTUS M&E Matrix FY2015, dated 27 December 2016, reported by IIE on cumulative actuals through June 2016. Objective 2: At least 90 percent of LOTUS graduates are empowered to assume future leadership roles and civic responsibilities. Numerical targets for this objective are the numbers of leadership-related training sessions, skill sessions, specialized meetings, workshops, enrichment sessions, and cultural events held, as well as student participation in NM Namaa Summer Program (Cohorts 1 and 2) and Leadership Camp (Cohorts 3 & 4). Although IIE is achieving its targets in these areas, these activities themselves do not indicate how much scholarship beneficiaries are acquiring the skills necessary to assume future leadership roles and civic responsibilities. It is dubious to conclude that exposure equals empowerment. Objective 3: At least 90 percent of LOTUS graduates are proficient in the English language. IIE adopted the ITP test as the tool for measuring students’ English language proficiency and has established a standard score of ≥550. Targets for the number of scholarship recipients achieving this standard vary by cohort, as targets are set as IIE “best estimates based on baseline ITP scores and expected progression (90 percent of the targeted 90 percent graduation rate).” The LOTUS M&E Matrix of December 27, 2015, which indicates actual cumulative performance through June 2016, reports that 33 percent of Cohort 1 (21 students) and 40 percent of Cohort 2 (10 students) achieved the targeted level of English language proficiency prior to graduation. At the time of this report, Year 5 targets, which were to be established in September 2015, were not reported for Cohorts 3 and 4. 117 Objective 4: At least 90 percent of Cohorts 3 and 4 students complete the bridge year program successfully. The bridge year program was introduced by IIE in response to the low level of English language proficiency of Cohorts 1 and 2 students. Only thirty-three graduates (22 percent) achieved an ITP score equal to or greater than 550. In the bridge year program students learned only English and computer skills for a full academic year before enrolling in classes in their chosen fields of study. The LOTUS M&E Matrix (December 27, 2015) reports that Cohorts 3 and 4 exceeded their end-of-program targets. One hundred students from both cohorts (and 111 percent of the target) passed the English exam (the target was 90 students); and 97 students of a targeted 90 students (107 percent) passed the computer skills final exam. Objective 5: LOTUS students are placed in Study Abroad program opportunities in the US. According to the December 27, 2015 LOTUS M&E Matrix, IIE has achieved 100 percent of the Cohort 3 and 4 targets. LOTUS has achieved numeric targets of implementation objective 5 as set for both Cohorts 1 and 2. Data extracted from the M&E matrix indicate a 100 percent achievement for targets for both Cohorts 1 and 2; this is equivalent to 45 and 20 students, respectively. The performance of Cohorts 3 and 4 will be reported following the summer after the students’ second academic year (i.e., FY2016 and FY2017). Objective 6: At least 90 percent of LOTUS students demonstrate enhanced employability. Numerical targets for this objective relate to students’ participation in mentoring days, career self￾assessments and counseling sessions, internships, and job interviews. IIE is demonstrating progress toward completion of its stated targets, although these activities do not necessarily measure the extent to which scholarship recipients demonstrate greater employability. Objective 7: At least 100 staff and faculty from Egyptian partner universities receive capacity building training. The original estimate of the number of staff and faculty to be trained is 600. Given IIE’s recognized scheduling challenges, this number was revised downward to 500 and represents an average of 100 university staff and/or faculty trained per partner university for the five original universities. Although IIE does not track the number of staff per university trained, it reports a 69 percent achievement of the number of person days of training. Objective 8: At least two youths from the majority of Egypt's governorates are provided access to quality higher education (Cohorts 1-4). The average percent of governorates with at least one student enrolled was 95.25 percent and 100 percent of underserved governorates were represented (see Table VI.5 below). 118 TABLE VI.5: NUMBER OF GOVERNORATES WITH ACCESS TO QUALITY HIGHER EDUCATION WITH AT LEAST ONE STUDENT ENROLLED IN LOTUS SCHOLARSHIP INITIATIVE (COHORTS 1-4) Targets % of Target Achieved Year 5 Cumulative Target Year 5 Cumulative Actual Objective 8: At least two youths from the majority of Egypt's governorates are provided access to quality higher education (Cohort 1) 8b: No. of governorates with at least two students enrolled (one male, one female) 93 29* 27 Objective 8.1: At least one youth from the majority of Egypt's governorates is provided access to quality higher education (Cohort 2) 8.1b: No. of governorates with at least one student enrolled 96 27 26 Objective 8.2: At least one youth from the majority of Egypt's governorates with preference to the 12 underserved governorates are provided access to quality higher education (Cohort 3) 8.2a: No. of underserved governorates represented 100 12 12 8.2b: No. of governorates with at least one student enrolled 74 27 20 Objective 8.3: At least one youth from the majority of Egypt's governorates with preference to the 12 underserved governorates is provided access to quality higher education (Cohort 4) 8.3a: No. of underserved governorates represented 92 12 11 8.3b: No. of governorates with at least one student enrolled 80 27 23 Note: * The number of governorates in Egypt was 29 at the time of reporting on Cohort 1. This number declined to 27 at the time of reporting on Cohort 2 as select governorates were merged. Objective 9: At least 125 girls/women are provided with access to quality higher education. IIE has consistently exceeded the female enrollment targets for Cohorts 1 to 4. The average rate of enrollment of women (all cohorts) is 111 percent. One hundred thirty-seven scholarship recipients are female, which is equivalent to 54.8 percent of the 250 total awards (see Table VI.6). TABLE VI.6: FEMALE ENROLLMENT BY COHORT Targets % of Target Achieved Year 5 Cumulative Targets Year 5 Cumulative Actuals 9b: Number of Cohort 1 girls/women enrolled in undergraduate degree 104 50 52 9d: Number of Cohort 2 girls/women enrolled in undergraduate degree 116 25 29 9g: Number of Cohort 3 girls/women enrolled in undergraduate degree 112 25 28 9j: Number of Cohort 4 girls/women enrolled in undergraduate degree 112 25 28 Total * 125 137 Note: * Average enrollment is 111 percent. 119 Part III: Statistical Analyses of Student Data Impact of English Language Proficiency on Academic Performance The team studied the impact of improved English language scores on GPA. The correlation coefficient between ITP scores and GPA for current students and graduates over time were calculated for both in all universities combined. Table VI.7 shows that, despite a positive correlation, it is not significant (p-value was high).32 The positive relation is slightly higher among graduates. This is confirmed by the correlation within each cohort. The positive correlation is significant among students in the first cohort; most graduates were in that cohort. The relation is not significant in Cohorts 2 and 4, and negatively significant in Cohort 3. There is no clear explanation for the latter except that the few number of data points is being affected by the few cases of poor performance. When examined by university, the positive and significant correlation is clear within AASTMT, BUE, and MSA, while in Al-Ahram and Pharos, the correlation is positive but not significant. In Future University, the correlation is negative but not significant. Reviewing Figures VI.1 and VI. 2, it is clear that some students suffered fluctuations in their GPA scores in Future University, which might have caused the negative correlation. The team analyzed LOTUS students’ ITP and GPA scores over a fifteen semester period for each university. Figure VI.1 shows that average ITP scores improved over time for all cohorts and across all universities. There were minor reductions in scores in five of the six universities, but they do not affect the overall general trend. Average GPA scores fluctuated over time in five of the six universities (Figure VI.2), but they cannot be linked to improvement in English language proficiency scores. 32 The p-value is defined as the probability, under the assumption of the hypotheses being tested, of obtaining a result equal to or more extreme than what was actually observed. The smaller the p-value, the larger the significance because it tells the investigator that the hypothesis under consideration may not adequately explain the observation. For typical analysis, using the standard α = 0.05 cutoff, the null hypothesis (“null hypothesis" usually refers to a general statement or default position that there is no relationship between two measured phenomena) is rejected when p < .05 and not rejected when p > .05. 120 TABLE VI.7: CORRELATION COEFFICIENTS BETWEEN GPA AND ITP SCORES BY STATUS, COHORT, AND UNIVERSITIES Correlation of GPA and ITP Correlation Coefficient and Number of Data p-Value Status Currently Enrolled 0.0238 217 0.7274 Graduated 0.1031 217 0.1301 Cohort Cohort 1 0.1452 239 0.0247** Cohort 2 -0.0556 95 0.5923 Cohort 3 -0.2707 50 0.0573* Cohort 4 0.2062 50 0.1509 (based on one GPA score) University AASTMT 0.3442 45 0.0206** Al-Ahram Canadian University 0.1856 67 0.1326 British University 0.1818 114 0.0529* Future University -0.1852 63 0.1463 Modern Sciences and Arts University 0.3092 86 0.0038*** Pharos University in Alexandria 0.1251 59 0.3452 Note: *p-value <0.1; i.e., statistically significant at level 0.1, less than one in ten chance of being wrong. **p-value <0.05; i.e., statistically significant at level 0.05, less than one in twenty chance of being wrong. ***p-value <0.01; i.e., statistically significant at level 0.01, less than one in one hundred chance of being wrong. 121 FIGURE VI.1: MEAN ITP TEST SCORES OVER TIME BY UNIVERSITY AND COHORT FIGURE VI.2: MEAN GPA TEST SCORES OVER TIME BY UNIVERSITY AND COHORT Comparative Analysis of the Impact of Bridge Year versus English Intensive Courses on English Language Proficiency The relative change in English ITP scores after the first year of instruction is calculated for all graduates and current students from the four cohorts. This allowed the evaluation team to compare results of the bridge year courses (Cohorts 3 and 4) with those of the English Intensive courses (Cohorts 1 and 2). The relative change is defined by the difference between the second and first test scores achieved during the first enrollment year. A t-test is used to compare the mean of the relative change in the English ITP scores between the two cohorts. The mean of the relative change is combined for Cohorts 1 and 2 and Cohorts 3 and 4 and compared. As Table VI.8 shows, the mean change in English ITP scores is higher after the first 122 year of English language training for Cohorts 1 and 2 (who took English Intensive Courses) than for Cohorts 3 and 4 (who took bridge year courses). A t-test shows the difference to be statistically significant with a p-value of 0.09. TABLE VI.8: MEAN CHANGE IN ENGLISH ITP SCORES AFTER FIRST YEAR OF INSTRUCTION BY COHORT Groups Number of Observations Mean Standard Error 95% Confidence Interval Cohorts 1 and 2 135 0.0699 0.0091 0.0520 0.0879 Cohorts 3 and 4 100 0.0491 0.0077 0.0339 0.0644 Comparing the relative change of the combined English ITP score of Cohorts 1 and 2 with that of Cohorts 3 and 4, it can be seen that the relative change of Cohort 1 and 2 students is significantly higher than that of Cohort 3 and 4 students (p-value 0.0012). This means that students who took the intensive English courses had greater English language proficiency than those who took bridge year courses. TABLE VI.9: MEAN CHANGE IN ENGLISH TEST SCORES AFTER FIRST YEAR OF INSTRUCTION IN COHORTS 2 AND 3 Groups Number of Observations Mean Standard Error 95% Confidence Interval Cohort 2 43 0.1381 0.0148 0.1083 0.1679 Cohort 3 50 0.0779 0.0108 0.0563 0.0996 Comparing the relative change in English scores between Cohort 3 and 4 students, all of whom had bridge year instruction in English, a significant difference is observed in favor of Cohort 3 (see Table VI.10, p-Value 0.0001). TABLE VI.10: MEAN CHANGE IN ENGLISH TEST SCORES AFTER FIRST YEAR OF INSTRUCTION IN COHORTS 3 AND 4 Groups Number of Observations Mean Standard Error 95% Confidence Interval Cohort 3 50 0.0779 0.0108 0.0563 0.0996 Cohort 4 50 0.0204 0.0095 0.0013 0.0394 A comparison between AASTMT and BUE, since only they have students from Cohorts 3 and 4, shows no significant difference in the relative change in English scores between the two universities (see Table VI.11, p-value 0.9). TABLE VI.11: MEAN CHANGE IN ENGLISH TEST SCORES AFTER FIRST YEAR OF INSTRUCTION IN AASTMT AND BUE Groups Number of Observations Mean Standard Error 95% Confidence Interval AASTMT 45 0.0502 0.0113 0.0274 0.0729 BUE 55 0.0483 0.0106 0.0270 0.0696 123 Impact of US Study Abroad on English Language Proficiency The impact that travelling abroad has on English language proficiency can be measured by calculating the relative change between ITP scores before and after travel. The team studied Cohorts 1 and 2 since they are the only ones who have had opportunities to study abroad. A t-test determines whether the difference between the means of the two groups is significant. Improvement in English is slightly higher among those who travelled abroad than those who did not, with a significance level of 0.1 (p-value 0.089) (Table VI.12). This finding applies very clearly to students in Cohort 1 (p-value 0.0075) (Table VI.13), but the difference among students in Cohort 2 is reversed (not significant with a p-value of 0.79) (Table VI.14). This suggests that some students who did not study abroad still studied hard and improved their English test scores. TABLE VI.12: MEAN CHANGE IN ENGLISH TEST SCORES BY TRAVEL OR NON-TRAVEL, US STUDY ABROAD Group Number of Observations Mean Standard Error 95% Confidence Interval Didn't Travel 78 0.1170 0.0100 0.0970 0.1370 Did Travel 64 0.1367 0.0106 0.1157 0.1579 TABLE VI.13: MEAN CHANGE IN ENGLISH TEST SCORES FOR COHORT 1 BY US STUDY ABROAD STATUS Group Number of Observations Mean Standard Error 95% Confidence Interval Didn't Travel 52 0.0981 0.0115 0.0751 0.1212 Did Travel 44 0.1390 0.0117 0.1153 0.1627 TABLE VI.14: MEAN CHANGE IN ENGLISH TEST SCORES FOR COHORT 2 BY US STUDY ABROAD STATUS Group Number of Observations Mean Standard Error 95% Confidence Interval Didn’t Travel 26 0.1547 0.0175 0.1186 0.1908 Did Travel 20 0.1319 0.0222 0.0853 0.1785 A comparison of the average change in English proficiency, by university , 33 for those who travelled/did not travel shows that, for all u niversities except FUE, English language improvement is higher among those who travelled than those who did not (Table VI.15). The difference in improvement is significant at ACU (p-value 0.005) and MSA (p-value 0.079), but not at BUE or PUA. Future University shows an unexpected difference in favor of those who did not travel. 33 AASTMT is not included as it only has students from Cohorts 3 and 4. 124 TABLE VI.15: MEAN CHANGE IN ENGLISH TEST SCORES BY UNIVERSITY AND US STUDY ABROAD STATUS University Didn’t Travel Did Travel Total ACU*** 0.0945 0.1952 0.1431 0.1104 0.0851 0.1099 15 14 29 BUE 0.0706 0.0895 0.0837 0.0518 0.0879 0.0780 8 18 26 FUE 0.1519 0.0917 0.1385 0.0940 0.0549 0.0896 21 6 27 MSA* 0.1195 0.1555 0.1325 0.0794 0.0556 0.0730 23 13 36 PUA 0.1095 0.1413 0.1267 0.0704 0.0745 0.0729 11 13 24 Notes: Figures (rounded) in each box correspond to mean, standard deviation, and number of observations, in that order. * P-value <0.1 ** P-value<0.01 *** P-value<0.001 Impact of US Study Abroad on Employability In this analysis, the correlation between US Study Abroad and graduate employment is used to examine whether more graduates who studied abroad are currently employed. Those in residency in their chosen fields of study (e.g., dentists, physical therapists, currently enrolled graduate students, and males in the army) as well as those with no known current status are both excluded from the analysis. Both the Chi 2 test and Fisher exact test, which is performed when the number of observations is low, as is the case here, are used in this analysis. Table VI.16 shows that a higher percentage of employed graduated students studied abroad, but most of the unemployed did not. The p-values for both the Chi 2 test and Fisher exact test is 0.015 (see Table VI.17). The difference does not appear to be significant for males (Table VI.18) but, among females, there is a positive relation (Table VI.19). Female graduated students who travelled abroad are more likely than male graduates to enter the labor market. 125 TABLE VI.16: LOTUS GRADUATED STUDENTS BY EMPLOYABILITY STATUS AS REPORTED BY IIE Current Status Didn’t Travel Did Travel Total Unemployed 13 4 17 Employed 16 23 39 Graduate Student 0 2 2 Resident 7 2 9 Military Service 1 3 4 Undetermined Status by IIE 9 3 12 Total 46 37 83 TABLE VI.17 GRADUATED STUDENTS (MALES AND FEMALES) BY EMPLOYMENT AND TRAVELED ABROAD STATUS Unemployed Employed Total No. Didn't Travel 13 (44.8%) 16 (55.2%) 29 (100%) No. Did Travel 4 (14.8%) 23 (85.2%) 27 (100%) Total 17 39 56 Pearson Chi 2 p-Value 0.015 Fisher Exact test p-Value 0.015 TABLE VI.18: GRADUATED STUDENTS (MALES ONLY) BY EMPLOYMENT AND TRAVELED ABROAD STATUS Not employed Employed Total No. Didn't Travel 4 (40%) 6 (60%) 10 (100%) No. Did Travel 3 (27.3%) 8 (72.7%) 11 (100%) Total 7 14 21 Pearson Chi 2 p-Value 0.54 Fisher Exact test p-Value 0.44 TABLE VI.19: GRADUATED STUDENTS (FEMALES ONLY) BY EMPLOYMENT AND TRAVELED ABROAD STATUS Not employed Employed Total No. Didn't Travel 9 (47.4%) 10 (52.6%) 19 (100%) No. Did Travel 1 (6.25%) 15 (93.75%) 16 (100%) Total 10 25 35 Pearson Chi 2 p-Value 0.007 Fisher Exact Test p-Value 0.01 126 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523