November 6, 2017 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Virginia Lambert, Ashraf Bakr, Doaa Mohammed, Ola Hussein Hosny, Tamer Said, Youmna Khalil, and May Gadallah under The QED Group, LLC Egypt SIMPLE project. Final Performance Evaluation of the Egypt STEM Schools Project (ESSP) Graduated and currently enrolled science, technology, engineering, and mathematics (STEM) students. Photo courtesy of USAID/Egypt. i FINAL PERFORMANCE EVALUATION OF THE EGYPT STEM SCHOOLS PROJECT (ESSP) November 6, 2017 AID-263-I-15-00001/AID-263-TO-17-00003 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ii ACKNOWLEDGEMENTS The evaluation team for the Egypt STEM Schools Project (ESSP) Final Performance Evaluation and the USAID/Egypt Services to Improve Performance Management, Enhance Learning and Evaluation (SIMPLE) activity, which was charged with carrying out the evaluation, wish to acknowledge and thank the numerous individuals and organizations who contributed to our knowledge and understanding of this ambitious activity. First, we would like to recognize the invaluable support of World Learning and the ESSP implementing team for their support. They were willing to meet with the evaluators as a group and individually, arrange school visits, and schedule stakeholder interviews. The evaluators also appreciate the collaboration of World Learning in responding graciously and completely to numerous requests for information from the project database. Likewise, the evaluators would like to thank the representatives of the technical sub-grantee organizations who were generous in taking time from their crowded in￾country schedules to meet. Their insights and explanations were crucial to understanding the school activities. The partnership between the ESSP and the Ministry of Education is a defining characteristic of the STEM school program. The members of the Ministry Central STEM Unit and their organizations provided invaluable input by explaining their experience with the ESSP, their plans and expectations for the future, and their commitment to the STEM school initiative. The evaluation team’s visits to the nine STEM schools and conversations with the staff, teachers, and students were engaging and enriching. SIMPLE and the evaluation team credit the success of these visits, in large part, to the timely intervention of the Ministry of Education Central STEM Unit and the World Learning team in explaining the purpose and importance of the meetings. The evaluators were well received in all the schools, the meeting schedules were in place, and the participants in the discussions were generous with their time and efforts to respond meaningfully to the evaluation questions. The evaluation team also wishes to thank the teachers, students, and graduates who not only met with us in￾person, but also took the time to complete and return the online questionnaires. The support and guidance for the evaluation team from USAID was invaluable. We would like to thank especially Hala El Serafy, the ESSP Activity Manager who met with the evaluation team to explain the background and rationale for the activity in the context of USAID support to basic education in Egypt, and provided innumerable project documents. The team also thanks Seba Auda, the SIMPLE COR and Evaluation Manager who offered guidance and direction for the team and helped smooth the inevitable scheduling complications. SIMPLE and the evaluation team also wish to thank The North South Consultants Exchange (NSCE), which generously hosted the team in its offices and provided logistical support (and coffee breaks) throughout the evaluation. The team particularly appreciates the flexibility of the NSCE staff in responding to requests for meetings outside of regular business hours. The evaluation would not have been possible without the constant attention and response from the SIMPLE staff, Sherine Saber (Evaluation Manager) and especially Richard Gaeta (SIMPLE Senior Evaluation Specialist), who supported the team without pause throughout the evaluation process. Finally, the evaluation team wishes to thank the many students, teachers, school administrators, and others who made this evaluation a learning experience and a pleasure to hear their experiences and dreams for the future. iii TABLE OF CONTENTS ACRONYMS................................................................................................................................. IV BIOGRAPHIES.............................................................................................................................. V EXECUTIVE SUMMARY........................................................................................................... VIII INTRODUCTION..........................................................................................................................1 BACKGROUND.............................................................................................................................1 METHODS AND LIMITATIONS .................................................................................................4 QUALITATIVE DATA.................................................................................................................................................. 5 QUANTITATIVE DATA.............................................................................................................................................. 5 LIMITATIONS ................................................................................................................................................................ 6 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS ..................................................7 EVALUATION QUESTION 1: TO WHAT EXTENT HAS THE ESSPACHIEVED THE TERMS AND CONDITIONS (INCLUDING THE PLANNED OBJECTIVES) OF ITS COOPERATIVE AGREEMENT SINCE ITS START?........................................................................................................................................................ 7 EVALUATION QUESTION 2: WHAT ARE THE MOST SIGNIFICANT FACTORS THAT SUPPORT OR CONSTRAIN THE IMPLEMENTING PARTNERS’ ABILITY TO ACHIEVE THE PROJECT’S DESIRED OUTCOMES?............................................................................................................................................. 13 EVALUATION QUESTION 3: WHAT FACTORS HAVE FACILITATED AND/OR IMPEDED STUDENTS’ IMPROVED CAPABILITIES IN THE ESSP EDUCATION SYSTEM?........................................ 17 EVALUATION QUESTION 4: WHAT ARE THE ACTIONS AND POLICIES THAT NEED TO BE TAKEN BY USAID AND MOE TO EXPAND ESSP EDUCATION TO ADDITIONAL GOVERNORATES?..................................................................................................................................................... 23 EVALUATION QUESTION 5: TO WHAT EXTENT HAS THE ESSP SUCCEEDED TO ACHIEVE GENDER BALANCE?................................................................................................................................................. 30 OVERALL CONCLUSIONS AND RECOMMENDATIONS ...................................................32 ANNEX I: EVALUATION SCOPE OF WORK .................................................................................................... 34 ANNEX II: EVALUATION METHODS AND LIMITATIONS........................................................................... 47 ANNEX III: DATA COLLECTION INSTRUMENTS .......................................................................................... 53 ANNEX IV: DATA COLLECTION SCHEDULE............................................................................................... 106 ANNEX V: DATA TABLES AND GRAPHS ....................................................................................................... 108 A. SECONDARY QUANTITATIVE/QUALITATIVE DATA.......................................................................... 108 B. PRIMARY QUANTITATIVE/QUALITATIVE DATA (ONLINE SURVEYS) ........................................... 119 ANNEX VI: BIBLIOGRAPHY ................................................................................................................................ 130 ANNEX VII: ESSP END-OF-PROJECT PERFORMANCE EVALUATION TIMELINE, JANUARY￾OCTOBER 2017 ....................................................................................................................................................... 133 ANNEX VIII: DISCLOSURE OF ANY CONFLICTS OF INTEREST…………………………………..138 iv ACRONYMS 21PSTEM 21st Century Partnership for STEM ACT University Readiness Test (US) AOR Agreement Officer Representative BOT Board of Trustees CA Cooperative Agreement CCIMD Center for Curriculum and Instructional Material Development COR Contracting Officer’s Representative COS Classroom Observation Scale ECASE Education Consortium for the Advancement of STEM in Egypt ESSP Egypt STEM Schools Project GAEB General Authority for Educational Buildings ISEF International Science and Engineering Fair MOE Ministry of Education MSI Management Systems International NCEEE National Center for Educational Evaluation and Examination NCERD National Center for Education Research and Development PAT Professional Academy for Teachers PPP Public Private Partnership SIMPLE Services to Improve Performance Management, Enhance Learning and Evaluating SPSS Statistical Package for the Social Sciences STEM Science, Technology, Engineering, and Mathematics STEP Scholarships and Training for Egyptian Professionals TDC Technology Development Center TFI The Franklin Institute TIES Teaching Institute for Excellence in STEM TIMSS Trends in Mathematics and Science Study URT University Readiness Test (Egypt) USAID United States Agency for International Development v BIOGRAPHIES Virginia Lambert is a senior development professional, trained in sociology, with expertise in evaluation, gender analysis, and monitoring systems and indicators. She has worked in all regions of the world, with a particular focus on Latin America, and is fluent in Spanish. She is a capable technical writer and editor. Her primary sectoral experience has been in economic growth, rural development, workforce development, and education. Her work in Egypt began in 2001, and has included a variety of short-term assignments during the past 15 years, including workforce development activities, gender analyses, and evaluations. Since 2015, she has led three evaluations for the USAID Office of Health and Education, including two with the USAID/Egypt SIMPLE project. Doaa Abdelaal is specialized in monitoring and evaluation, research, and gender analysis and training. She participated in the midterm evaluation of the LOTUS project and the end of term evaluation of the LEAD project, both of which were funded by USAID/Egypt. From 2013-2015, she was the Voters’ Education Officer at the Strengthening of Democratic Process in Egypt (a UNDP-funded project). There, she designed and implemented a national campaign on voter education targeting rural women and disenfranchised youth (which included data collection and analysis), and developed a media campaign that consisted of many products, such as a radio soap opera and TV spots. She worked for three years (2010-2013) as MENA Coordinator of the International Knowledge Network of Women in Politics (iknow Politics), then as a gender consultant with several organizations working in Egypt and regionally in the Middle East and Africa. Through her work with the WELDD project, she trained and facilitated workshops in English and Arabic on women’s leadership and gender equality. She is currently a gender consultant for the ISMAP project (funded by JICA). Ms. Abdelaal has an MA in political science from Cairo University. Ashraf Bakr Al-Shareef is the Results-Based Management (RBM) Advisor to Global Affairs, Canada (GAC; formerly CIDA), and is an independent international consultant. He was Co-Director of the M&E Division of the USAID-funded Egypt Education Reform Program (ERP). He was the primary author of the M&E Program of the Egyptian Ministry of Education’s Strategic Plan for Pre-University Education Reform. Subsequently, he led the Ministry’s efforts in the development, piloting, and institutionalization of a results-based planning, monitoring, evaluation, and reporting system for early childhood education (ECE) in Egypt. Dr. Al-Shareef has conducted and led teams in many impartial evaluation and survey activities employing both quantitative and qualitative research techniques in Egypt and across the Middle East and North Africa (MENA) region. His recent work includes an evaluation of a USAID-funded administrative reform program in Iraq. He holds degrees from the University of Cairo (MSc) and the University of Liverpool (UK) (MA and PhD in Social and Environmental Studies). May Gadalla is an Assistant Professor at Cairo University, Faculty of Economics and Political Science. She holds a PhD in Biostatistics from University of California, Los Angeles, and a Master’s degree in Statistics from Cairo University. She is also a research associate at the Economic Research Forum (ERF Middle East), Cairo, Egypt. She has 25 years of experience in the field of data analysis. She was a statistical consultant for the World Bank, UNFPA, ERF, Social Research Centre (SRC American University), Population Council (Middle East), and USAID; Deputy Director at the Center of Economics and Financial Studies, Cairo University, Egypt; Head of the Social Justice Observatory at the Social Contract Centre; and Senior Program Officer at the Population Council (Middle East). She has led teams in conducting and supervising national surveys, and conducted capacity building workshops in monitoring and evaluation, impact evaluation, advanced statistical tools, and survey methodologies. She has extensive experience with in-depth data analysis of different research themes (labor market, education, vi poverty and child poverty, gender studies, impact evaluation, nutrition, and other topics focused on poverty and youth-related policies), in addition to midterm and end-of-term project evaluations. Her research deals with quantitative and mixed method analysis with a focus on gender perspectives, and her work has been published in academic papers and national reports. Ola Hosny is an Educational Research and Development Specialist. She has over 18 years of experience in conceptualizing, managing, and evaluating development projects in the area of education, development, poverty, gender, and youth. Her work demonstrates advanced application of different research methods and strategies. She is skilled in examining policies and practices in education development in local and global contexts. She is also capable of synthesizing information from various sources, views, research and opinions to produce well-crafted, thoughtful, data-driven, analytical research that acknowledges the influence and power of cultural, political, economic, and social issues. She works on effecting system￾level change through sustainable development. Ms. Hosny has conducted multiple regional and national evaluations in such areas as youth employment, women empowerment, scholarship programs, entrepreneurship, and nutrition for UNIVERSALIA/Canada, NIRAS/Denmark, Drosos/Switzerland, International Rescue Committee/USA, USAID/Egypt, and WFP/Egypt. She also has extensive experience in Upper Egypt governorates, building NGO capacity at the governorate level and CDA capacity at the village level. She has experience with USAID-funded intervention projects, and experience working with donors (e.g., IDRC, CIDA, DFID, UNFPA, UNICEF). Ms. Hosny is also a part-time instructor at the Professional Educator Diploma of the American University in Cairo (AUC). She holds a Master’s degree in International and Comparative Education from AUC. Sally Ibrahim is a freelance statistician as well as Statistical Analysis Trainer. She has participated in different research projects and studies as a statistician, assistant statistician, and data entry officer. She has worked with international agencies (e.g., UNIDO, JICA, World Bank, and USAID), and has more than eight years of experience managing, coordinating, and scheduling the work of data entry teams to ensure high quality data entry and tabulation. She also has experience in social and economic development, and has participated in different baseline surveys and midterm and end-of-term evaluations. She has more than 10 years of experience in statistical data entry, preparation, and analysis as well as quality control. She is currently completing her Master’s degree in Statistics at Cairo University, and teaches statistics part-time in pursuit of CFA certification. Youmna Khalil is a development practitioner and M&E specialist with over 15 years of experience. Her expertise includes education, higher education, youth, leadership development, financial education, economic empowerment for women, rural communities, health, housing rehabilitation, microfinance, vocational training, and crafts development. She has extensive knowledge in managing, monitoring, assessing, and evaluating projects using different quantitative and qualitative methodologies. She has also worked with different donors (USAID, CIDA, GIZ, Social Fund for Development, Aga Khan Development Network, Embassy of Finland, and the US State Department). Tamer Said is an educator specialized in research related to cognitive and developmental psychology and brain science. He received training as a master’s candidate in the Mind, Brain, and Education (MBE) Program at Harvard Graduate School of Education. Prior to that, he obtained an MA in International and Comparative Education and an MSc. in Biotechnology from AUC. Since 2013, he has been teaching at the AUC Professional Educator Diploma (PED) Program. He provides training to teachers in Teaching Adolescents Learners (TAL) and STEM education focused on strategies, assessment, learning theories, and inquiry based learning. His course instruction aims to link student brain and psychological development to actual classroom practice. On the higher education level, he designs and delivers training workshops for faculty members at Zewail University for Science and Technology on such topics as course design, psychological development, active learning, and assessment. He also assists faculty vii members in designing project-based learning courses for science students. He has published papers on promoting problem-based learning in science classrooms, applying inquiry based learning in laboratories, and designing active learning environments that support cognitive and psychological development in both grade school and the higher educational setting. viii EXECUTIVE SUMMARY The USAID Egypt STEM Schools Project (ESSP) supports the Ministry of Education (MOE) initiative to upgrade the quality of Egyptian education in science and mathematics through the establishment of a network of public secondary schools specialized in STEM subjects (science, technology, engineering, and mathematics). The ESSP began August 28, 2012, and is scheduled to end August 27, 2017. Total USAID funding for the activity is US$29,965,648. The ESSP is implemented by World Learning and a consortium of three US technical organizations specialized in STEM education.1 This End-of-Project Performance Evaluation was carried out by the USAID/Egypt Services to Improve Performance, Enhance Learning and Evaluation (SIMPLE) activity with a team of six evaluators, between March and May 2017. BACKGROUND The development hypothesis of the ESSP is, “If targeted technical assistance and capacity building is provided to the MOE to establish a rigorous academic foundation for introducing and sustaining a successful and functional STEM prototype, the potential to stimulate innovation and quality improvement across the education system will be accelerated.” The five ESSP objectives are directed to this same process of strengthening the foundation for a continuing process of innovation and change. The ESSP objectives are to: 1. Increase student interest, participation, and achievement in science and mathematics with a special effort geared to underrepresented groups such as girls and economically marginalized students. 2. Strengthen the STEM school local initiative through developing an effective model of specialized high schools focusing on science, technology, and mathematics for gifted students. 3. Build the capacity of a highly-qualified cadre of STEM professionals and provide opportunities for training and sustained, intellectually rigorous professional training. 4. Strengthen MOE capacity at the systems and policy level to sustain and replicate these model schools. 5. Support the MOE in upgrading science and mathematics curriculum standards, student assessment and teacher preparation for the mainstream. The evaluation purpose has two parts: (1) To review, analyze, and evaluate the achievement of the objectives; and (2) To provide recommendations and lessons learned to ensure that the ESSP activities contribute to the continuation of the change process initiated by the ministry. The achievements of the ESSP are measured, in part, by whether the process of innovation and change continues. The five evaluation questions are: 1. To what extent has the ESSP Program achieved the terms and conditions (including the planned objectives) of its cooperative agreement since its start? 2. What are the most significant factors that support or constrain the implementing partners’ ability to achieve the project desired outcomes? 3. What factors have facilitated and/or impeded students’ improved capabilities in the ESSP2 1 The three technical institutions are 21st Century Partnership for STEM (21PSTEM), Teaching Institute for Excellence in STEM (TIES), and The Franklin Institute (TFI). 2 When the Cooperative Agreement was signed, the activity was called the Education Consortium for the Advancement of STEM in Egypt (ECASE). Subsequently, it was named the Egypt STEM Schools Project (ESSP). ESSP replaces the ECASE label. ix education system? 4. What are the actions and policies that need to be taken by USAID and MOE to expand ESSP education to additional governorates? 5. To what extent has the ESSP succeeded to achieve gender balance? Two underlying characteristics of the ESSP are important in framing the analysis of the five evaluation questions. First, the ESSP is part of an ambitious ongoing process of innovation. As a process of innovation, the plan and specific outputs are not completely charted out at the beginning, and the end point, described as a “dream” or “vision,” is not fully detailed. Success is not measured at one point in time. Rather, it is measured by the continuation and maturation of the process. The evaluation assesses the achievements to date, seeks out the points of vulnerability, and highlights critical issues to be confronted for the process to continue. Second, implementation of the ESSP has been interwoven with the MOE STEM initiative. The method of implementation has been highly participatory, with a strong emphasis on building capacity for the MOE initiative to continue. It is important to note that the STEM school model is Egyptian, and is owned by the MOE and other Egyptian stakeholders. One of the main tasks of the ESSP has been to provide a platform for the stakeholders to work together. METHODOLOGY The mixed methods approach to data collection and analysis provided quantitative and qualitative data for a wide spectrum of stakeholders. Qualitative data were collected in the schools and in Cairo. Three sub-teams of two evaluators and a classroom observer each conducted three-day visits to three of the nine schools for interviews and discussions with students, teachers, administrators, staff, parents in the Boards of Directors (BOTs), and representatives of the local STEM units. In Cairo, the team interviewed all members of the Central STEM Unit, USAID project officials, and multiple representatives of the implementing organizations (World Learning, TIES, TFI, and 21PSTEM.) The team also held two group discussions (six participants each) with STEM school graduates. The richness of this qualitative data lies in the subjective assessment of the STEM school model and components from multiple positions within the system, and, particularly, from the students who are the principal beneficiaries. The team supplemented the qualitative findings with quantitative data collected in three anonymous online surveys with students, teachers, and graduates. Questionnaires were sent to all students, teachers, and graduates, with response rates of 67% for students, 85% for teachers, and 22% for graduates. A third source of data was 54 classroom observations (six per school), using the tool developed by the ESSP for STEM (i.e., the Classroom Observation Scale or COS), and conducted by trained MOE observers during the school visits. Finally, the team used quantitative data from published secondary sources, the ESSP monitoring database, and ESSP reports and documentation to contextualize qualitative findings and to document project deliverables. Anonymous individual student academic scores (grade point averages), by grade and gender, were provided by the nine school principals. Analysis of the quantitative data focused primarily on comparisons by gender and school (the two original schools and the seven 2015 schools). Limitations to the analysis, due to the methodology, were minor. For the qualitative data collection, the principal constraint, especially in the schools, was the scheduled time. Discussion groups frequently pushed beyond the one-hour limit. Limitations for the quantitative data centered on the use of online, rather than paper-and-pencil surveys. Although the response rates were adequate, the gaps, especially for graduates, probably introduced unknown response bias. The most serious limitation in the online surveys for teachers and students, was the incidence of missing responses, which increased exponentially x toward the end of the questionnaires. This suggests problems of poor Internet quality in the STEM schools. KEY CONCLUSIONS The achievements of the MOE STEM School initiative are concrete, visible, and positive (i.e., fully functioning schools that demonstrate the success of the educational model in generating high levels of achievement among students and graduates). Implementation has occurred through a collaborative and participatory process, with a conscious plan to ensure that the model and development process are fully Egyptian, and integrated into the MOE institutional structure. The strong sense of commitment and ownership voiced by stakeholders at all levels, in the schools and in the ministry, attest to the impact of this approach. Evaluation Question 1: To what extent has the ESSP Program achieved the terms and conditions (including the planned objectives) of its cooperative agreement since its start? The ESSP demonstrated continuous collaboration and support for the MOE STEM school initiative, registering significant achievements in terms of the five original ESSP Objectives. For example: • For Objective 1, the expansion of the STEM school model, the growth in the number of students, and student success academically and internationally affirm the achievement of the objective to increase student interest, participation, and achievement in science and mathematics. • For Objective 3, the ESSP invested heavily in the creation of a cadre of STEM professionals, and a sustained professional development system is in process. Training of teachers and administrators has been a central contribution of the ESSP to the MOE initiative. The training programs have been fully documented and certified, and master trainers are in place. The ESSP also successfully implemented a system for coaching and mentoring. • For Objective 4, the ESSP has enhanced the capacity of NCEEE, CCIMD, and NCERD3 to replicate and revise the STEM curriculum and assessment processes. Key individuals within each organization have been trained and participate in the Central STEM Unit. At the same time, through this process, some aspects of the original objectives were modified, delayed, or dropped. For example: • For Objective 1, the planned outreach to preparatory schools was based on a local rather than a national model, and was replaced by a national science activity in grade 7. This year, the MOE Science Counselor piloted a national program to introduce preparatory students to the STEM approach to science. Also, Egyptian law requires neutrality in school admission. No direct action was taken to encourage underrepresented groups to apply to the STEM schools, although the absence of school fees, including room and board, removes a potential barrier for economically marginalized students. • Objective 2 anticipated that each STEM school would be tailored to its local environment, but, to date, the model, curriculum, assessment, and capstone themes are uniform across schools. • Under Objective 3, the professional development system is not yet fully institutionalized. Courses and curriculum have been certified by PAT (Professional Academy for Teachers). Master trainers have been identified but not yet certified, and the management and financing mechanisms are still under development. 3 National Center for Educational Evaluation and Examination (NCEEE), Center for Curriculum and Instructional Material Development (CCIMD), and National Center for Education Research and Development (NCERD). xi • Likewise, the institutional capacity building for NCEEE, CCIMD, and NCERD under Objective 4, at present, is dependent upon the highly-trained individual representatives to the STEM Unit nominated by each organization. The remaining task is to build on the skills of these individuals to internalize the STEM methods in the organizations as a whole. • For Objective 5, to date, the application of the successful practices in STEM school curriculum standards, student assessment, and teacher training in the mainstream secondary school science and mathematics programs has been minimal. Expansion has occurred through the establishment of new STEM schools. Lateral integration of STEM practices that have demonstrated effective learning results has not yet been developed. Evaluation Question 2: What are the most significant factors that support or constrain the implementing partners’ ability to achieve the project desired outcomes? The ESSP is linked to an ongoing MOE initiative. Factors in the external environment and MOE supported the project outcomes, but also placed constraints on the direction and pace of change (refer to the supporting and constraining factors below). The task of the implementing partners has been to nourish a self-perpetuating process of innovation and adaptation to build on the support and manage the constraints. Supporting Factors: Growing attention to the importance of science and technology for development in Egypt helped open the door for an innovative model in secondary education based on the STEM model. Examples include the emphasis on math and science in the MOE 2007 strategic plan for pre-university education, the mandate for USAID support voiced in the 2009 Obama speech at Cairo University, and the early positive media coverage of STEM student achievements. The composition of the ESSP team and their approach to implementation also were positive factors in the achievement of sustainable outcomes. The balance of skills and experience within the team (i.e., the combination of experience in the MOE and strong external expertise) successfully managed the development process through adaptation to changing circumstances, while continuing the technical innovation. The participatory approach, incremental successes, and the sense of being a part of “something new” have contributed to strong local ownership. This is reflected in the Central STEM Unit, the local STEM units, administrators, teachers, students, and school BOTs. Constraining Factors: The unanticipated increase in the number of new STEM schools in 2015, from two to nine, a decision external to the ESSP, required the reallocation of resources, and caused disruptions in operations, problems in filling teaching positions, and issues in recruitment and retention of students because of incomplete facilities, such as dormitories. A politically charged environment caused intermittent disruptions and delays in the planned implementation schedule. Schools and students were distracted by the general unrest in the country. At one point, US technical experts were barred from travel, and USAID went through about six months of recalibration of normal activities at the beginning of FY2014. The MOE had seven ministers during the project period. A third external constraint is the centralization of the mechanisms to form public-private partnerships (PPP). This has restricted schools’ access to financial resources from potential local partners for local schools. xii Evaluation Question 3: What factors have facilitated and/or impeded students’ improved capabilities in the STEM education system? In the student and graduate surveys, students express a strong overall positive assessment of their experience in the STEM schools. Ninety-four percent (94%) of the graduates (n=104) and 79% of the current students (n=1,423) said they would recommend the STEM schools to others. In group discussions and the survey, they also identified aspects of the model that, in their eyes, are problematic and points for improvement going forward. ● The capstone project is at the center of the STEM education model in terms of both pedagogy and student assessment. The capstone project, and its various stages such as the journals, account for 60% of the student’s grade in years 10 and 11, and 20% in the final year. Students recognize and praise the capstone project as a multi-dimensional learning tool that engages them in project-based/inquiry-based self-learning. It is cited as a reason to apply to the STEM school. At the same time, students and some teachers suggest a need to refine some aspects of the capstone experience to address excessive time investments and distraction from other aspects of their education. ● In the student discussion groups in all the schools, the capstone was cited as both a reason for entering the STEM school, and an important feature that distinguishes STEM education from mainstream education. Students value the opportunity to solve problems and take pride in their solutions. In some groups, primarily for grades 11 and 12, students commented on the importance of grounding the capstone themes in Egypt’s Grand Challenges, since it strengthens their awareness and understanding of the potential for practical application of their projects. Many said they would like to have the opportunity to share their projects more broadly in a public arena. ● Teachers, capstone leaders, and students all agreed that applying the Engineering Design Process (EDP) to students’ work on the capstone project, particularly in grade 12, often advances students’ research skills and knowledge beyond that of the teachers. Students value the opportunity to access expertise in the education and research community outside the school to expand their knowledge. The ease of access to these resources is cited by students, teachers, and administrators as a differentiating factor among STEM schools. For example, access to university facilities is particularly difficult for students in Luxor and Hurghada. ● Other positive factors associated with the capstone experience include the strong value they attach to team work. Students also recognize the importance of the capstone experience in building their presentation skills and ability to express themselves in English. ● While students lauded the capstone project experience, they also identified negative aspects of the experience. The positive and negative factors emerged from the same student and teacher discussion groups. Both students and teachers cited instances of misalignment across semesters between the curriculum/learning outcomes and the capstone themes. This caused frustration and duplication. ● Students invest significant non-classroom time in project preparations; most also expend personal funds (up to 800 LE) on materials for the projects. Given the time requirements for the capstone, students complain about having to attend class sessions as well. They contend that the time would be better spent on the capstone project, where more learning takes place. Students also are dissatisfied with the time required for capstones and classes, and the lack of time and resources for activities such as sports, music, and art. “We’re not just scientists.” ● Students and teachers in all the schools discussed problems and dissatisfaction with the assessment of the capstone projects. Students complain about the weight given to the capstone, relative to the required time and commitment (60% in grades 10 and 11, and 20% in grade 12). xiii Students and teachers also observe inconsistencies in the assessment standards across schools. Finally, in all the schools, teachers and students are dissatisfied and highly critical of the entire journal process, including questions, assessment rubrics and process, time requirement, and fairness. The centralization of the journal process and timing across schools are intended to ensure “fairness,” but no one in the schools is defending this exercise. ● Student reaction to the laboratories has two sides: (a) Laboratory facilities in the schools, especially the FabLabs, support student academic achievement, and are a critical distinguishing feature of the STEM model. However, (b) students face difficulties in fully utilizing the laboratories, especially the FabLabs, because of gaps in materials, supplies, and timely maintenance, as well as the issue of availability outside class time. • Access to international competitions, which is magnified in the STEM schools, contributes to the development of globally competitive scholars. ● The boarding school model and dormitories contribute to students’ personal development, school commitment, and to the STEM school learning objectives. At the same time, students note a lack of trained personnel in the dorms to deal with their personal issues and medical problems, and they are dissatisfied with the accommodations. ● In all 34 student group discussions, students expressed the opinion that some teachers have shortcomings in STEM school qualifications, particularly in terms of capacity to teach in English, use of inquiry-/problem-based methodology, and in some cases, technical skills. While students’ critique of teachers is expected, the fact that all student discussion groups also recommended that teachers’ access to training be increased affirms the value of that training from the students’ point of view. ● Students identified multiple factors in the school management system that, from their point of view, detract from their school experience and performance. These included lack of a uniform Student Code of Conduct reflective of the boarding school situation, extended class sessions (80 minutes each) and strict attendance policies, and constraints on their activities and time outside of class. No information is available for objective measures of the effect of these factors on student learning or achievement. ● Concerns about future education and career prospects for STEM school graduates, due to the perception among parents and students that university admission policies are unfavorable for STEM graduates, and that scholarship opportunities are limited, are causing students to drop out before graduation. (This point is elaborated under Evaluation Question 4.) Evaluation Question 4: What are the actions and policies that need to be taken by USAID and the MOE to expand STEM education to additional governorates? In response to this question, the evaluation draws on the analysis for the entire study to specify five critical issues that require action and policy development by USAID and the MOE in order for the STEM school initiative to move forward and expand. Potential actions and policies to respond to these issues, which were identified by stakeholders in interviews and group discussions, are listed in the report. Based on these lists, the evaluation makes five recommendations, which are actionable and within the manageable interest of USAID and/or the MOE. xiv Issue 1: Uncertainty about post-graduation status, opportunities, and implied risk poses an immediate threat to the continued viability of the STEM school model as an alternative to mainstream secondary education for parents and students. The clearest indicator of this issue is the dropout rate among students, especially girls, at the end of grade 11. The dropout rate for male students after grade 11 was 15% (n = 116) in 2016; the dropout rate for female students after grade 11 was 39% (n = 64) in 2016. Interviews and group discussions in the nine schools that will have grade 12 next year attest to the likelihood that the pattern will be replicated widely. In general, the issue is stated as a choice between a guaranteed and an unknown future. The uncertainty derives from a lack of information about universities, admission requirements, career options, scholarship opportunities, and application procedures. Moreover, the effect of STEM school assessment processes on all these factors is contrasted with students’ and parents’ knowledge and experience with the process for admission through the mainstream schools and Thanaweya Amma. In the discussion groups, graduates said they would have benefitted from more mentoring in the STEM schools, and also from more guidance and social support about future plans. Current students said they rely on word-of-mouth from other students, especially those ahead of them in the school, for information about scholarships and other opportunities. Issue 2: The number and capacity of available teachers is not sufficient to adequately staff additional STEM schools with qualified teachers. The application and assessment processes for identifying and hiring teachers for the STEM schools is complete and formalized within the Professional Academy for Teachers (PAT). However, too few applicants meet the required qualifications to fill the available positions. Further, in the surveys, students and graduates identified teacher quality as a key impediment to learning. In interviews, three factors were identified that contribute to the shortfall: (1) teachers generally prefer to work in their home governorates; (2) the pool of qualified teachers varies by region, especially in terms of English proficiency, but also in STEM subjects; and (3) teacher turnover. A total of 141 teachers trained by the ESSP are no longer teaching in the STEM schools. This number is equivalent to 40.7% of the current number of teachers in the nine STEM schools. Recent departures, which are small in the newest schools (e.g., less than five per school), generally are explained by teachers and administrators in terms of the factors specific to the STEM schools or the individuals — excessively long hours and heavy work load, school location, insufficient financial incentives or job benefits, and lack of a defined career path. Some also left because they could not adjust to the new pedagogy or because of poor performance.4 Issue 3: The current pace of expansion of STEM schools presents a risk for the quality and sustainability of the STEM school model and initiative. In key informant interviews and discussion groups with stakeholders across the board, the consensus response to the question about future expansion was that new construction should pause or slow down. In some cases, concerns focused on “dilution” of the model; others applaud the objective of broadening access to the model, but point to the risks for quality and sustainability, because of human resource constraints and operating costs. Other stakeholders also question the criteria for school locations. Not all governorates offer ready access to universities, research centers, or industry for the extra-curricular aspect of the STEM model. Issue 4: The decision-making structure and lines of communication for the STEM school 4 This information is based on perceptions. No objective data on reasons for turnover was available to the evaluation team from secondary sources. xv initiative are unclear and not institutionalized. While the technical structure for the STEM school initiative, which includes the Central STEM Unit, the local STEM unit, and the STEM schools, has been formalized within the MOE structure via Decree 136 of April 2, 2017, and designates the STEM UNIT as part of the General Education sector reporting to the First Undersecretary/Head of the General Education Sector, this structure does not specify channels for decision-making or communications in specific key areas. This pending formalization is a potential point of vulnerability in terms of sustainability. Issue 5: Ambiguity among students and parents, and, in their eyes, among many preparatory school and university teachers about the vision/direction of the STEM school model raises questions about the projected nationwide expansion of the system and its sustainability. The interest of students, parents, and educators in the STEM school initiative is not necessarily a demand for a science and math curriculum. Only 11% of STEM students surveyed (n=1,223) cited an interest in STEM subjects as the reason for enrollment. Most students (89%) gave reasons other than STEM education, such as “better education” (40.3%) or to “try something new” (28.2%). Teachers say they applied to the STEM schools for professional development, a better education model, to try something new, or for financial incentives. In short, the survey data suggest that the salient underlying feature is a demand for an educational model based on problem-solving and applied learning, rather than memorization. According to the data, information about the STEM schools is spread mostly through word-of-mouth and social media, rather than through a systematic information campaign based on formal communication channels. This issue focuses on communication strategy. The absence of a clear cut and concise message about the vision/direction of the STEM school initiative contributes to a lack of consensus and ambiguity about the STEM school among stakeholders. This, in turn, may contribute to instability within the student community as well as with parents and higher education institutions because of differences and uncertain expectations about the content of the program and options for the future. Evaluation Question 5: To what extent has the ESSP succeeded to achieve gender balance? The ESSP, as a USAID-funded activity, did not include specific measures to achieve gender equality (e.g., activities to encourage girls to apply). The STEM school model, in accordance with Egyptian law, is gender neutral or blind, and designed to serve males and females on the same terms. The application process and school structure does not distinguish by gender. Gender differences in student enrollment and teaching staff therefore mirror differences in the population. Gender differences exist among students in school performance and graduation rates. Differences between boys and girls in academic performance are small, but girls are more likely than boys to drop out at the end of grade 11. Students, particularly girls, identify a gap in the STEM school model in their focus on academic issues, to the relative neglect of the non-academic aspects of the schools, including student life, psycho-social issues, and changes in gender dynamics (e.g., girls’ emerging sense of independence and empowerment). xvi RECOMMENDATIONS Recommendations are identified for each evaluation question, but because of the interconnections among the questions, some recommendations apply to more than one question, in whole or in part. 1. USAID should support the MOE to develop a comprehensive plan for institutionalizing the STEM school model, addressing fundamental governance structures and arrangements, both within and outside the ministry, at the central, governorate, and school levels. The plan should include: − the relationship and decision-making points, authority, and processes of the STEM Unit vis-à-vis other departments within MOE, as well as the NCEEE, CCIMD, and NCERD. For example, the periodic review and update of the STEM curriculum is not yet a fully integrated function within the mandate and standard operational practice of CCIMD; i.e , from a governance viewpoint, the CCIMD operates independently of the STEM Unit; − the relationship of the STEM schools to secondary education departments at the mudereya and district levels; − the resolution of issues related to the legal status of the boards of trustees (BOTs); representation of STEM schools in Student Unions and activities at the local and national levels; and the STEM School Code of Conduct. [Evaluation Questions 1, 2, 3] 2. In response to teacher uncertainty and dissatisfaction, the MOE, through PAT, should define the parameters of the career path for STEM teachers, within the provisions of the Special Cadre for Teachers Law, and institutionalize a corresponding professional development system. [Evaluation Question 1] 3. USAID should collaborate with the MOE to explore alternative approaches to partnering with the private sector and the community, based on a mutually beneficial relationship, and draw upon the experience in the US with partnerships between private firms and government entities. [Evaluation Question 2] 4. The MOE should ensure that an efficient and reliable management mechanism is in place, with adequate financial resources for infrastructure, equipment maintenance, and supplies. The STEM school model is resource-intense and highly dependent on technology. Maintenance and supplies must be timely. Consideration could be given to decentralization of functions and responsibilities. [Evaluation Question 3] 5. USAID and the Central STEM Unit should expand the mechanisms for strengthening and supporting teachers, within the schools and during the work week, building on the emerging experience with team teaching, coaching and mentoring, and peer support networks. [Evaluation Question 3] 6. The MOE should pursue a dialogue with the Ministry of Higher Education to explore mechanisms in the public universities to satisfy the aspirations of STEM graduates and to retain and build on the accomplishments of the STEM school secondary education for the benefit of Egypt. [Evaluation Questions 4, 3] 7. USAID should support the establishment of a guidance center in each STEM school to advise students on opportunities/options for higher education and careers, to provide assistance in completing university and scholarship applications, and to offer tools and practice sessions to prepare for entry exams. [Evaluation Questions 4, 5] 8. USAID should support a re-examination of the curriculum in university faculties of education to incorporate instruction in the teaching skills and pedagogical methods used in the STEM school education model. [Evaluation Questions 4, 1, and 3] 9. USAID and the MOE should develop technical and operational benchmarks to guide the pace of new STEM school construction (e.g., state of operation of existing schools in terms of student/teacher population, budget allocations, and professional development goals). [Evaluation xvii Question 4] 10. USAID should support the development of a formal media and outreach campaign to provide factual information about the STEM school experience and career options, as well as the vision or dream behind this type of education. [Evaluation Question 4] 11. USAID and the Central STEM Unit should consider incorporating student life and individual personal and career development into the STEM school model, through specialized staff, training, counseling services, and non-academic extracurricular activities. Gender awareness training should be included to alert staff to potential systematic differences in the experience of boys and girls in the school. [Evaluation Question 5, 4] Conclusion The technical support, leadership, and resources of the five-year Egypt STEM Schools Project have been critical to the advancement of the Ministry of Education STEM schools program. A tangible system is now in place on the ground, with demonstrated positive results. The less tangible achievements associated with the process of creating the STEM school model, are equally important. The commitment and ownership exhibited by stakeholders and participants are strong evidence of the value of building the program with the ministry staff and school communities from the beginning. At the same time, the STEM schools’ initiative is still in process. The evaluation identified factors within the operation of the schools that will improve the education experience for students and teachers, and issues that require action and policy reform by the MOE and USAID. 1 INTRODUCTION This final performance evaluation of USAID/Egypt STEM Schools Project (ESSP) was conducted by USAID/Egypt’s Services to Improve Performance Management, Enhance Learning and Evaluation (SIMPLE) activity, with a team of six evaluators, between March and May 2017. The ESSP supports the Ministry of Education (MOE) initiative in building a network of public secondary schools specialized in the STEM subjects (science, technology, engineering, and mathematics). This is designed to upgrade the quality of Egyptian education in science and mathematics. The ministry began this process with the establishment of two schools, the STEM School for Boys in 6th of October, which opened in 2011, and the STEM School for Girls in Maadi, which opened in 2012. Seven additional schools began operating in September 2015, in Dakahleya, Assiut, Alexandria, Ismaileya, Kafr El-Skeikh, Red Sea, and Luxor. In September 2016, the MOE enrolled students in new schools in Menufia and Gharbia, although these most recent schools are not included in the evaluation. The ESSP (AID-263-A-12-0005) began August 28, 2012, and ends August 27, 2017. Total USAID funding for the activity is US$29,965,648. The ESSP is implemented by World Learning, along with a consortium of US technical organizations focused on STEM education — 21st Century Partnership for STEM (21PSTEM), The Franklin Institute (TFI), and Teaching Institute for Excellence in STEM (TIES). The purpose of this end-of-project Final Performance Evaluation is two-fold: 1. To review, analyze, and evaluate the USAID-funded STEM activities implemented by the ESSP, and the degree to which they have achieved program objectives and completed deliverables; 2. To provide program recommendations and lessons learned to ensure that USAID-supported STEM activities have the highest potential to achieve their intended results in a sustainable manner. The audience for the evaluation is the USAID/Egypt Mission Education Team, the Egyptian Ministry of Education, and USAID/Washington. The evaluation and final report are structured around five evaluation questions: 1. To what extent has the ESSP achieved the terms and conditions (including the planned objectives) of its cooperative agreement since its start? 2. What are the most significant factors that support or constrain the implementing partners’ ability to achieve the project desired outcomes? 3. What factors have facilitated and/or impeded students’ improved capabilities in the ESSP education system? 4. What are the actions and policies that need to be taken by USAID and MOE to expand ESSP education to additional governorates? 5. To what extent has the ESSP succeeded to achieve gender balance? BACKGROUND According to the Evaluation Scope of Work (Annex I), the ESSP’s development hypothesis is, “If targeted technical assistance and capacity building are provided to the MOE to establish a rigorous 2 academic foundation for introducing and sustaining a successful and functional STEM prototype, the potential to stimulate innovation and quality improvement across the education system will be accelerated.” This hypothesis points to two important factors for the evaluation and recommendations. First, the ESSP is providing technical assistance and capacity building to the MOE in support of the MOE STEM school initiative. Second, the expected outcome is to stimulate a process of innovation and quality improvement across the education system. The five ESSP objectives below are directed at this same process of strengthening the foundation for a continuing process of innovation and change. 1. Increase student interest, participation, and achievement in science and mathematics with a special effort geared to underrepresented groups such as girls and economically marginalized students. 2. Strengthen the STEM school local initiative through developing an effective model of specialized high schools focusing on science, technology, and mathematics for gifted students. 3. Build the capacity of a highly-qualified cadre of STEM professionals and provide opportunities for training and sustained, intellectually rigorous professional training. 4. Strengthen MOE capacity at the systems and policy level to sustain and replicate these model schools. 5. Support the MOE in upgrading science and mathematics curriculum standards, student assessment and teacher preparation for the mainstream. The STEM school model, embodied in the first two prototype schools, is based on an entirely new approach to public secondary education in Egypt. The model is informed by international (US) experience, and emphasizes hands-on, project and inquiry-based, student-centered, integrated education. Students work in teams of three to five people to design and build a capstone project that responds to an assigned theme based on the Grand Challenges of Egypt. The model also includes a broad, rigorous academic schedule with a focus on science and mathematics. Classes are small (25 students each) and the language of instruction is English. The schools are boarding schools for students who are high achievers in preparatory schools (based on prep school final exam scores and an IQ test). Initially, the STEM School for Boys in 6th of October and STEM School for Girls in Maadi recruited students from across Egypt, and had a capacity for 150 boys and 120 girls. The seven schools that opened in 2015 are regional schools. They are also boarding schools (a feature unique to STEM schools in public secondary education), but are co-educational, with separate dormitories and classes for girls and boys. The entry requirements, curriculum, and use of English are the same in all schools. Since 2015, the STEM School for Boys in 6th of October and STEM School for Girls in Maadi have been designated as regional schools. According to the ESSP database, the nine STEM schools included in the evaluation have a 2016-17 student population of 2,152 students (1,193 males; 959 females), and employ 243 teachers (179 males; 64 females) and 15 administrators (12 males; 3 females). Only the 6th October and Maadi schools have grade 12 students this year. It is important to emphasize the extent to which the STEM school model represents a paradigm shift for public secondary education. Mainstream secondary schools are characterized by lectures and memorization geared entirely to the final exam at the end of grade 12. This final exam, the Thanaweya Amma, determines access to public and private universities in Egypt. In STEM schools, problem-solving and research replace memorization and, importantly, student assessment is based on multiple criteria, including a capstone project and a new University Readiness Test (URT), which is analogous to the 3 American College Testing (ACT) college readiness assessment. STEM school students do not take the Thanaweya Amma exam. Box 1: Egypt STEM School Model The three-year public secondary school offers an alternative educational model for high-achieving preparatory school students, both male and female. The student-centered learning model is project￾and inquiry-based, and hands-on. The curriculum is centered on science and mathematics, although students also study English, Arabic, German or French, social studies, religion, computer science and, to a much lesser extent, art, music, and physical education. The classes are limited to 25 students and, unlike the mainstream secondary program, schools are equipped with laboratories, and students are required to conduct experiments. While textbooks are available in the library, they are treated more as references than manuals. The curriculum is defined around learning outcomes, and learning occurs through student research and experimentation, with guidance from the teachers. Each student receives a laptop in grade 10, and all instruction is in English. All schools are boarding schools. The capstone project is a defining characteristic of the STEM school. Each semester, teams of three to five students develop unique projects around an assigned theme. Teachers serve as Capstone Leaders to facilitate the students’ research; give feedback through grading their journals, portfolios, and posters; and help with time management. Capstone themes are drawn from the Grand Challenges of Egypt (i.e., fundamental issues that threaten the future of the country that may be resolved through the application of science and technology.) The Grand Challenges for the STEM schools were defined at the beginning of the program in a high-level workshop with teachers, administrators, university professors, and others, as the basis for the curriculum and projects. Examples include water, pollution, health, and food production. They are reviewed periodically. Recently, climate change was added to the list. Each school is equipped with a FabLab for the student capstone projects. A FabLab is a small-scale workshop for digital fabrication through computer-controlled tools that cover several different scales and materials to create prototypes and models, such as a 3-D printer and laser cutter. Each FabLab has a full-time manager to ensure the safety of the students and machines. Perhaps the most important feature of the STEM school model is the system for student assessment. Instead of the score on the single standardized test at the end of grade 12, the Thanaweya Amma, as in the mainstream schools, the grade point average (GPA) of STEM school students combines their marks across various categories, including performance on the capstone project, attendance/class participation, midterm and final exams on concepts/learning outcomes, and a final, newly-developed grade 12 exam, the University Readiness Test (URT). Within the MOE, under the direction of the head of the public education sector, technical direction for the STEM school initiative is provided by the Central STEM Unit. This unit is composed of representatives of the math, science, and English Counselors’ offices, and of the MOE-affiliated agencies responsible for curriculum, student assessment, and teachers. The ESSP was instrumental in the formation of the Central STEM Unit and the definition of its role. The ESSP technical team worked with Central STEM Unit members in the development of the STEM curriculum and standards; the assessment system for STEM students (including the new URT); the student admission requirements and process; and the teacher requirements, recruitment, hiring, and training. At this point, in the final months of the ESSP, all these tasks are entirely managed by the Central STEM Unit. 4 In addition to the Central STEM Unit, a local STEM office in each mudereya/district that has a STEM school participates in the supervision, assessment, and coaching of STEM school teachers, assists in the identification of potential candidates for STEM school teaching positions, and provides general oversight and support to the STEM school. The MOE investment in the STEM school initiative has been large and consistent. Under the agreement with USAID, the MOE constructs and furnishes the schools, and covers operating costs (e.g., staff, dormitories, meals, and materials). The students pay no fees. The ESSP supports the development of the curriculum and standards, the assessment system, teacher training and professional development, specialized laboratory equipment and learning materials, and institutional development. The ESSP was initially designed to support the two original schools, and to “incubate” three new schools in Alexandria, Dakalea, and Assiut. When seven new schools were opened in 2015, instead of three, the project was amended to provide support to all nine schools. The project was further amended in 2016 to extend USAID funding of ESSP activities for a fifth year through August 2017. METHODS AND LIMITATIONS Two underlying characteristics of the ESSP are important in framing the analysis of the five evaluation questions. First, the ESSP is part of an ambitious, ongoing process of innovation. As a process of innovation, the plan and specific outputs are not completely charted out at the beginning, and the end point, described as a “dream” or “vision,” is not fully detailed. Success is not measured at one point in time. Rather, it is measured by the continuation and maturation of the process. The evaluation assesses the achievements to date, seeks out the points of vulnerability, and highlights the critical issues to be confronted for the process to continue. Second, implementation of the ESSP has been interwoven with the MOE STEM initiative. The method of implementation has been highly participatory, with a strong emphasis on building capacity for the MOE initiative to continue. It is important to note that the STEM school model is Egyptian, and is owned by the MOE and other Egyptian stakeholders. One of the main tasks of the ESSP has been to provide a platform for all these stakeholders to work together. The five evaluation questions framed the evaluation design and data collection tools. The mixed methods approach allows for the triangulation of data to verify findings, as well as the collection of opinions and feedback from multiple stakeholders, such as students, teachers, parents, and the MOE, all of whom have different relationships to the STEM school model and different expectations. One of the strengths of the evaluation is the extensive recording of feedback from within the school communities. The data collection tools for the qualitative interviews and group discussions drew on a prior review of USAID project documents, as well as Ministry Decrees related to the establishment of the MOE STEM school initiative. See Annex VI for the complete list of documents consulted, and Annex III for copies of the qualitative data collection tools. The primary sources of quantitative data were secondary data from World Learning (student and staff population, training statistics, student achievements), student academic scores from the schools, teachers’ scores on pedagogy from classroom observations, and three online surveys of teachers, current students, and graduates. 5 QUALITATIVE DATA The evaluation team spent three days in each of the nine schools included in the evaluation (Refer to Annex II, Table 6). Three groups of two evaluators each visited three schools over a three-week period. In each school, the team facilitated three discussion groups with teachers (STEM teachers, non-STEM teachers, and Capstone Leaders and facilitators) (an average of about 16 per school, with some overlap because capstone leaders are also teachers; n=146), and four discussion groups with students (separate groups for male and female students in grades 10 and 11) (34 groups with an average of 9 students each, n=3).1 Other discussion groups included the members of the Board of Trustees (BOTs), social workers/ psychologists, and members of the local STEM unit. Key informant interviews were completed with the school principal (n=9), deputy principal/academic coach (n=6), the FabLab manager (n=8), the librarian (n=9), and the IT specialist/manager (n=9). An MOE-trained classroom observer worked with the two evaluators, conducting six observations of 80-minute sessions (four STEM classes, one of which was a laboratory class, and two non-STEM classes, including English to the extent possible, with a mix of experienced and new teachers across grades). The classroom observation tool (i.e., the Classroom Observation Scale or COS), which was developed by the ESSP and is used by supervisors in coaching STEM school teachers, appears in Annex III. An illustrative data collection schedule for a school is included in Annex IV. Annex II has complete documentation of the key informant interviews and group discussions. The evaluators were well received in all of the schools. As requested by the Central STEM Unit, the administrators scheduled the meetings for the evaluators and notified the participants. The participants seemed eager to relate their experiences and points of view, and freely responded to the team’s questions. The second major source of qualitative information was a series of key informant interviews in Cairo with World Learning staff; representatives of the sub-grantees (TIES, TFI, and 21PSTEM) and Management Systems International (MSI); members of the Central STEM Unit and others from the collaborating agencies (Professional Academy for Teachers [PAT], National Center for Educational Evaluation and Examination [NCEEE], National Center for Education Research and Development [NCERD], and Center for Curriculum and Instructional Material Development [CCIMD]); MOE employees, including the Central STEM Unit manager, representatives of the Offices of the Counselors for Science, Mathematics, and English, and the Director of the Technology Development Center; and USAID. These interviews were conducted by one sub-team of two evaluators (including the team leader) during the period when school visits were not possible because of midterm exams. The qualitative data were coded and analyzed using Excel tally sheets. The contacts included approximately 626 consultations, through 89 group discussions and 70 key informant interviews. QUANTITATIVE DATA The details of the quantitative data collection process are included in Annex II. World Learning provided the team with all data from the project database concerning the student and teacher populations throughout the project’s term (i.e., August 2012-August 2017). This included detailed information on training for teachers and administrators, and on student achievements. The project does not have information on student status post-graduation. Other available data included training for other school staff and for the Central STEM Unit. The World Learning project database further functioned as the source of information on student and teacher turnover. 1 The exception is that, in the 6th October and Maadi schools, the team had only three student discussion groups, with girls in Maadi and boys in 6th October, in grades 10, 11, and 12. 6 The team requested and received individual academic assessment scores (GPA) by grade, year, and gender for students in the nine schools. These data were provided by the school administrators without names or other means of identification. Scores were not available for students in grade 12; these are secured in the central ministry. ESSP implementers recommended the use of online (rather than written) surveys to reach students, teachers, and graduates, since all are issued emails within the STEM system, and they are accustomed to being contacted online. The team utilized the Survey Monkey software for all surveys. All current students, teachers, and graduates were included in the surveys. The response rates were as follows: students, 66.5% (n=1432), teachers, 85.2% (n=207), and graduates, 22.4% (n=104) (Graph 1) (refer to Annex II, Tables 1 and 2). The three questionnaires were short and required approximately 10 minutes for completion. The questionnaires were distributed in each school by the IT manager, who manages the email addresses for students, teachers, and graduates. Two follow￾up messages went out for each survey, and requests were made through the school administrators to encourage students and teachers to respond. The analysis of the surveys, which was primarily descriptive, was done using SPSS and STATA. Primary comparisons were made by school, grade, and gender. LIMITATIONS There were minor limitations to the analysis due to the methodology. For the qualitative data collection, the principal constraint, especially in the schools, was the scheduled time. Discussion groups with students and teachers frequently pushed beyond the one-hour limit. In all the schools, some of the interviews had to be completed after the end of the official school day (approximately 7:45 am to 3:00 pm). In Cairo, the issue was scheduling. The team benefitted from outstanding support from World Learning in contacting and scheduling appointments, but the officials are busy and, inevitably, some of the interviews spilled over beyond the week allocated for these meetings and the standard work day. Limitations for the quantitative data are elaborated in Annex II. The decision to use online surveys rather than paper-and-pencil surveys saved considerable time in administration and coding, but at the cost of a robust response rate. The original intent was to include the entire populations of currently enrolled students and teachers. Although the response rate was good, it was not one hundred percent. The evaluation design always anticipated an online survey of graduates, who are widely scattered throughout Egypt and abroad. The 22.4% response rate is acceptable for an online instrument, but is biased toward students living in the Cairo area and more recent graduates, compared to those who finished their studies two or three years ago. The most serious limitation to the online surveys for teacher and students, however, was the incidence of missing responses, which increased exponentially toward the end of the questionnaires. As discussed in Annex II, analysis of the pattern of missing responses suggests that this problem was largely due to the poor quality of Internet services in the STEM schools. When the Internet connection was lost, the questionnaire was ended prematurely. 67% 22% 85% 0% 20% 40% 60% 80% 100% Enrolled Students (n = 1,432) Graduated Students (n = 104) Teachers (n = 207) Graph 1: Survey Response Rate of STEM Schools Population 7 All of the schools collaborated in the team’s request for (anonymous) student academic achievement scores, by gender and grade. However, the format for reporting these scores was not uniform or complete for all schools. These differences limited the number of possible comparisons, especially given the small range of variation in these marks. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS This section discusses the conclusions and supporting findings for each of the five evaluation questions listed in the Scope of Work, as well as the associated recommendations. As noted, some recommendations apply to more than one evaluation question. EVALUATION QUESTION 1: TO WHAT EXTENT HAS THE ESSP2 ACHIEVED THE TERMS AND CONDITIONS (INCLUDING THE PLANNED OBJECTIVES) OF ITS COOPERATIVE AGREEMENT SINCE ITS START? ESSP Objective 1: Increase student interest, participation and achievement in science and mathematics with a special effort geared to underrepresented groups such as girls and economically marginalized students. Conclusion: The expansion of the STEM school model, growth in the number of students, and students’ academic and international success all affirm the achievement of Objective 1. However, the ESSP dropped the planned tool for systematic outreach to preparatory schools because it was “determined that Advokit would not be the most suitable means to advocate and promote the STEM schools.”3 Neither the ESSP nor the MOE has taken any direct action to encourage underrepresented groups to apply to STEM schools. Findings and Supporting Evidence: The expansion of the STEM school network from the original two schools in 2012 to 11 by 2016-2017 reflects an increase in interest and participation. Student applications and enrollment have both increased. At present, 2,152 students are enrolled in the nine schools supported by the ESSP, including 1,193 males and 959 females4 (Graph 2) (see Annex V, Table A.1). As of 2016, 298 students have graduated from the STEM School for Boys in 6th of October and 166 from the STEM School for Girls in Maadi, for a total of 464 graduates (Graph 3) (see Annex V, Table 2 The Scope of Work referred to the Education Consortium for the Advancement of STEM in Egypt (ECASE) Program in the evaluation questions. We replaced this acronym with the acronym for the name of the activity, the Egypt STEM Schools Project (ESSP), as recommended in the oral debriefing. 3 See the following quote from the October 2012-September 2013 ESSP Annual Report: “1.2.3 Adapt Egyptian STEM School Advokit. After a close analysis of the tool and comparing the results to the actual need on the ground, it was determined that Advokit would not be the most suitable means to advocate and promote the STEM schools. This objective of Advokit will continue but under other the outreach activities being implemented by ECASE.” (ECASE Annual Report, October 2012- September 2013). 4 Data on the number of applicants were not available to the evaluation team. Although the number of enrollees has increased as the network has grown, not all schools are currently at capacity. 8 A.2). (The other schools will not have 12th grade students until the 2017-18 school year). According to currently enrolled students, all graduates have entered or intend to enter university. Complete data are not available on the number of students who have received full or partial university scholarships.5 Of graduates who responded to the online survey, 65.2% of those currently enrolled in universities (n=89) have full or partial scholarships (Graph 4). USAID also reports that 27 Maadi graduates from 2015 and 2016 are studying in the US with USAID-funded STEP scholarships. Study abroad is a further indicator of achievement that shows that STEM graduates are internationally competitive (refer to Annex V, Table B.12).6 Another indicator of student achievement is participation and success in international science, engineering, and math competitions and fairs. The International Science and Engineering Fair (ISEF), sponsored by INTEL for STEM students, begins with local and national competitions and builds to an annual international fair. In 2017 alone, 38 teams of one or two STEM students each participated in ISEF competitions in Egypt; seven teams were invited to compete in the international ISEF competition in Los Angeles in May (see Annex V, Table A.3). Maadi students were successful in the 2015 international ISEF competition, including Yasmeen Yehya Abdo Mostafa, who won first place. This was widely publicized in Egypt and piqued the interest of potential students (current student group discussions), as well as bolstered support for the STEM network among government and ministry decision makers. 5 World Learning does not collect data on students after graduation. Furthermore, in group discussions, graduates reported that some were offered partial scholarships they could not use, due to lack of the remaining funds. 6 In interpreting the survey results it is important to recognize the potential for strong response bias. 2011 - 12 2012 - 13 2013 - 14 2014 - 15 2015 - 16 2016 - 17 147 243 338 357 817 1,193 0 119 209 300 651 959 NUMBER OF STUDENTS ACADEMIC YEAR GRAPH 2: Enrolled STEM Students by Year and Gender Male Female Self￾financed 35% Scholarship 51% Partial Scholarship 14% Graph 4: STEM School Graduates (2016/17) Enrolled in Universities, by Source of Funding (n=89) 0 50 100 150 200 2013/2014 2014/2015 2015/2016 2016/2017 85 92 121 116 91 75 64 Grapgh 3: Number of Graduated Students by Year and School 6th of October School (Male) Maadi School (Female) 9 At the same time, ESSP annual reports note that two activities anticipated in the Cooperative Agreement were not implemented. These were a systematic outreach activity to preparatory schools about STEM high schools, and a “special effort” geared to underrepresented groups. While the proposed system-wide outreach was dropped, interviews in under-enrolled STEM schools revealed that there were school-based outreach efforts to preparatory schools in the region. In the survey, 18.4% of current students learned about the STEM school through their preparatory schools.7 In addition, as described under Objective 5 (page 12), the MOE Science Counselor has instituted a science curriculum for first year preparatory school students to introduce them to the inquiry-based approach of the STEM school. Also, while neither the ESSP nor MOE has taken direct action to increase the participation of economically marginalized students, STEM schools have no school fees and no charge for room and board, which eliminates a potentially significant barrier for families with limited resources. Student selection criteria are gender and socio-economically neutral. All applicants are rated based on the same criteria: (1) achievement at the 98% level or more on the preparatory school final exam (lowered in 2016 to 95%),8 with a perfect score on two of three sections for math, science, and English; and (2) score on an IQ-based entry exam (Ministerial Decree 290, 24 July 2012). The proportion of female students is substantially lower in the schools in Upper Egypt than elsewhere9 (see Annex V, Table A.1). The STEM schools are boarding schools with equal capacity for girls and boys. Total enrollment is limited by dormitory space.10 ESSP Objective 2: Strengthen the STEM school local initiative through developing an effective model of specialized high schools focusing on science, technology and mathematics for gifted students. Conclusion: The original design anticipated that each STEM school would be tailored to its local environment, particularly in terms of the capstone themes and extracurricular activities. To date, there is minimal incidence of local initiative. The STEM school model, curriculum, assessment, and capstone themes are uniform across schools. Findings and Supporting Evidence: The STEM school architectural design and equipment are intended to be uniform across governorates.11 Likewise, all schools have the same laboratory facilities, a FabLab, library and books, and other facilities (e.g., meeting hall, gymnasium space, health clinic). Furthermore, the curriculum, learning objectives, and capstone project themes are designed and managed by the Central STEM Unit of the MOE, and are standardized across schools. The eligibility requirements and hiring process for teachers, administrators, and staff are the same, and centralized, as are the assessment criteria and processes for students and teachers. Under the ESSP, all teachers have received the same training sequences according to their specializations. Interviews with the Central STEM Unit representatives for the MOE entities responsible for each component (CCIMD, PAT, NCERD, NCEEE, and the Counselors’ Offices) confirmed the singular design and implementation processes. 7 In Luxor, where there was outreach to the preparatory schools, 26.4% gave this response. 8 Ministerial Decree 219, 2 August 2016. 9 For 2016-2017, enrollments are as follows: Alexandria: 269 students, 46% female; Assiut: 150 students, 40% female; Luxor: 103 students, 37% female; Red Sea: 183 students, 36% female; Kafr El-Sheikh: 202 students, 51% female; Dakahliya: 257 students, 50% female; Ismailia: 285 students, 48% female. 10 The only exception is that the Maadi school for girls has 30 fewer spaces than the 6th October school for boys. All other schools are coeducational, with equal dormitory accommodations available for boys and girls. 11 The only variations in design are the schools in 6th October, Maadi, and Luxor. The Luxor school is housed in an unused elementary school building and does not yet have on-campus dorms. The 6th October originally was constructed for the military. 10 At the initiation of the ESSP, local variation among the schools was expected to emerge through the initiative of the school Board of Trustees (BOT) and its engagement with the local community, particularly, universities and research centers. Capstone project themes were to respond, as much as possible, to local manifestations of Egypt’s Grand Challenges (e.g., alternative energy, pollution, water). In addition, the model anticipated the establishment of links to local businesses and industries both in support of the school facilities, and in student interaction through extracurricular activities such as field visits, consultations, and lectures. Based on interviews with BOT members and school administrators,12 although some engagement has occurred through field trips and lectures, overall local initiatives have been constrained by MOE requirements for central approval of protocols with other organizations, required MOE permission for outsiders to enter the schools, and the limitation to in-kind donations only from local businesses. ESSP Objective 3: Build the capacity of a highly-qualified cadre of STEM professionals and provide opportunities for training and sustained, intellectually rigorous professional learning. Conclusion: The ESSP invested heavily in the creation of a cadre of STEM professionals. A sustained professional development system is in process, but not yet fully institutionalized. Findings and Supporting Evidence: Professional development is a central contribution of the ESSP to the MOE STEM school initiative. The STEM school model involves an entirely new approach to secondary education in Egypt, not only in its focus on science and mathematics but, more importantly, in its teaching methodology and assessment of learning and achievement. The ESSP invested heavily in an intensive and continuous training program to transform teachers, beginning with an orientation during the hiring process, before entering the classroom, and on a continuing basis each year during the summer and semester breaks. All teachers in the schools have received at least one training course. However, due to turnover and the increased demand for teachers when the new schools opened in 2015, the timing and depth of the initial training has varied. In discussion groups, some teachers reported receiving training at the end of their first semester in the schools (see Annex V, Table A.4). The ESSP also provided initial and continuous training to school principals and deputy principals, based on their administrative and academic roles. Other school staff, including FabLab managers, librarians, IT specialists, and activity teachers also received intermittent and specialized training. When asked to rate the benefit of the training for their teaching, teachers gave an average rating across training topics of 4.0 or above (on a scale of 1 to 5, where 5 is the most beneficial) on the teacher survey (see Annex V, Figure B.3). In group discussions and interviews, teachers and administrators also praised the quality of the training. They particularly noted the value of being trained by people with STEM school experience, namely, former teachers and principals. Some teachers cited the opportunity for professional development as a benefit of teaching in a STEM school. In group discussions, when questioned about additional training needed, the most frequent responses were English training, subject￾matter training and, for non-STEM and activity teachers, training on application of STEM pedagogy to their subjects. In addition to the formal training program, in 2013, the ESSP instituted a structure for mentoring and coaching within the network of STEM schools. Weekly online sessions, by specialization (e.g., school principals, capstone leaders, FabLab managers), utilize the GOTOMEETING mechanism for coaching with US technical specialists and colleagues in other STEM schools. According to the annual reports and 12 Group discussions with the BOT in five schools — Ismailia, Assiut, Luxor, Red Sea, and Dakahlia — indicate some activity on their part to link the school with universities and/or local industries. Kafr El Sheikh is the only school that has a formal protocol (endorsed by the Ministry) with the local university. 11 interviews in the schools, these meetings are well attended and highly regarded. Participants reported taking steps to continue networking after the ESSP.13 In 2015, the position of Deputy Principal was defined as Academic Coach for the teachers in the school, while the Principal is primarily focused on administration.14 Looking ahead, the ESSP standard training curricula have been certified by PAT and are available in the STEM Design Blueprint, a repository for all STEM-related documents on the MOE STEM Master domain (see ESSP Annual Report, September 2016). Likewise, interviews with World Learning and PAT have revealed that STEM Master Trainers have been selected and trained to deliver the STEM training program in the future, and certification is imminent. The Ministerial Decree of 2 April 2017 provides for a Training Coordinator in the Central STEM Unit; this person is responsible for conducting needs assessments, and planning, implementing, evaluating, and documenting training programs. In addition, performance standards for teachers in STEM schools are in place, and are used in annual assessments. Of the teachers responding to this question on the survey, 92.7% (n=164) said that they are aware of the standards, and 89.8% (n=158) said that the standards have been used in assessment. A standards-based classroom observation scale (COS) tool has been developed and tested, and deputy principals and/or local STEM Unit supervisors in five of the schools said they have been trained to use it (see Annex III, Data Collection Tools). Unlike previous assessment tools, the COS is designed to provide feedback and coaching to the teacher at the end of the observation session. Significant steps have been taken toward sustainability and growth of the professional development component, but some aspects are still in process (e.g., the trainers have not been certified, management and financing mechanisms are still under development). Teacher turnover (see Annex V, Table A.4) and quality of instruction are significant concerns (see Evaluation Questions 3 and 4), as are the growing demands for qualified personnel as the network of schools grows. In all teacher group discussions, teachers expressed dissatisfaction with the lack of a clear career path and incentive structure for STEM professionals (i.e., uncertainty about whether and how the Special Cadre for Teachers applies to STEM professionals). Although STEM school teachers receive an incentive in addition to their salary as Ministry employees, many feel it is not commensurate with the stringent qualifications, additional responsibilities, time demands, and training requirements. ESSP Objective 4: Strengthen MOE capacity at the systems and policy level to sustain and replicate these model schools. Conclusion 1: The ESSP has enhanced the capacity of NCEEE, CCIMD, and NCERD to replicate and revise the curriculum and assessment processes. Conclusion 2: The sustainability of this enhanced capacity at the institutional level is incomplete. Key individuals within each organization have been trained and participate in the Central STEM Unit, but integration of the STEM educational model into the broader mandate of the organizations is still in process. 13 For example, among the principals, one person takes the lead each week to set the agenda and initiate the call. Unreliable Internet connections and the cost of the GOTOMEETING tool could be issues for sustainability, so some groups are utilizing other formats, such as Facebook groups and facetime. 14 No evaluation or assessment of the coaching and mentoring experience was shared with the evaluation team, and there was no evidence that such an assessment was done. 12 Findings and Supporting Evidence: The MOE established the Central STEM Unit by Decree 172 in April 2014. It is composed of the principal STEM system stakeholders, as a flexible entity within the ministry, to support in-country capacity building and sustainability for the STEM schools (ECASE Quarterly Report, July-September 2014). From that point onward, the ESSP developed a plan to build the capacity of the STEM Unit entities, and to transfer ownership of all programs and processes to replicate the STEM school model. The process, as described by STEM Unit members in key informant interviews, was one of partnering with the US technical consultant team, first by shadowing as trainees, then gradually directly implementing the STEM curriculum and standards development and STEM assessment methods and tools. The ESSP also established a web portal and repository for documenting the processes related to the creation and operation of the STEM model schools, the Design Blueprint, which is housed in the MOE Technology Development Center (TDC). The structure and function of the Central STEM Unit was formalized in Ministerial Decree 136 on April 2, 2017, thereby completing the transfer process. The decree does not address the inter-relationships between the MOE and these entities as institutions, or the decision-making lines and processes among them. The Central STEM Unit is composed of representatives from the NCEEE, NCERD, CCIMD, PAT, and the Counselor’s Offices for Science, Mathematics, and English. Members participated in all training provided to STEM school teachers and administrators, visited the STEM schools, and worked closely with ESSP technical advisors. The members participated with the Central STEM Unit, in addition to their regular responsibilities within their agencies, with no additional compensation or privilege. In interviews, all the STEM Unit members described their work with the STEM schools in terms of commitment and passion, because they “believe” in the STEM model. They further cited the uniqueness of the STEM Unit, that is, the collaboration among the diverse entities supporting the STEM system. The ESSP has supported the development of the Egyptian STEM curriculum and assessment methods. Interviews with STEM Unit members indicated the internalization of the capacity to replicate, update, and sustain the curriculum and assessment tools and processes. In addition, documents and manuals detailing the processes for developing and updating STEM components have been uploaded to the Blueprint, as guides for future development. At the same time, a potential weak point in terms of sustainability is that these achievements have been dependent on select individual champions in each organization. Turnover within the Central STEM Unit has been minimal in the first five years of the program, but the loss and replacement of these key members over time could create a gap in knowledge, understanding, and commitment, without broader, direct involvement of additional staff. A second factor related to MOE capacity, discussed primarily by school principals and members of the local STEM units, is the absence of a formal Code of Conduct or Standard Operating Procedures tailored to the STEM school model. The standard procedures for public secondary schools are unevenly applied and, in some cases, not applicable at all, given the unique structure of the STEM educational model and the fact that STEM schools are boarding schools. ESSP Objective 5: Support the MOE in upgrading science and mathematics curriculum standards, student assessment and teacher preparation for the mainstream. Conclusion: To date, this objective has not been a significant part of the ESSP’s plans or discussions. Findings and Supportive Evidence: A review of quarterly and annual reports and recent Annual Implementation Plans, along with interviews with Central STEM Unit members and project implementers, support the conclusion that no substantive steps have yet been taken to integrate STEM 13 science and mathematics curriculum standards or student assessment and teacher preparation into mainstream secondary schools. CCIMD collaborated in the development of the curriculum standards for STEM subjects, and took the lead in developing the standards for the humanities subjects in STEM schools. During the most recent quarter, CCIMD has collaborated in workshops and discussions to compare the STEM school and mainstream curriculum standards in STEM subjects, and to identify potential points for integration. The Science Counselor, formerly of the STEM Unit, has initiated a pilot program to introduce a hands￾on problem-solving element into the science curriculum for the first year of preparatory school. The curriculum has been prepared with CCIMD, materials procured, and teachers trained, and the program is now being piloted in two preparatory schools in each governorate. Following the examination of the results from the test schools, the Science Counselor plans to implement the program in all 9,000 preparatory schools. An extension of this curriculum will be available to students in years two and three as an elective activity (interview with the Science Counselor). Recommendations: 1. This broad recommendation is also referenced in Evaluation Questions 2 and 3. USAID should support the MOE in developing a comprehensive plan for institutionalizing the STEM school model, and addressing fundamental governance structures and arrangements, both within and outside the ministry, and at central, local, and school levels. The plan should include: - the relationship and decision-making points, authority, and processes of the STEM Unit vis-à-vis other departments within MOE, as well as NCEEE, CCIMD, and NCERD; - the relationship of the STEM schools to secondary education departments at the mudereya and district levels; - resolution of issues related to the legal status of the BOTs; representation of STEM schools in Student Unions and activities at local and national levels; and the STEM school Code of Conduct. 2. In response to teacher uncertainty and dissatisfaction, the MOE, through PAT, should define the parameters of the career path for STEM teachers, within the provisions of the Special Cadre for Teachers Law, and institutionalize a corresponding professional development system. EVALUATION QUESTION 2: WHAT ARE THE MOST SIGNIFICANT FACTORS THAT SUPPORT OR CONSTRAIN THE IMPLEMENTING PARTNERS’ ABILITY TO ACHIEVE THE PROJECT’S DESIRED OUTCOMES? Supporting factors: Since the ESSP is linked to an ongoing MOE initiative, factors within the external environment and the MOE support the project outcomes, but also place constraints on the direction and pace of change. The task of the implementing partners has been to nourish a self-perpetuating process of innovation and adaptation to build on the support and manage the constraints. Conclusion: Growing attention to the importance of science and technology for development in Egypt helped open the door for an innovative model in secondary education based on science, technology, and mathematics. Findings and Supporting Evidence: A number of factors present in the early 2000s reflected a growing sense of the need to strengthen the quality of basic education in Egypt, especially in science and 14 mathematics. As noted in the evaluation’s Scope of Work, the 2007 international Trends in Mathematics and Science Study (TIMSS) showed low performance in Egypt, both in absolute terms and in relation to other Middle Eastern countries. The 2007/8 to 2011/12 National Strategic Plan for Pre-University Education Reform in Egypt (MOE, 2007) emphasized math, science, technology, and English as key elements of the Comprehensive Curriculum and Instructional Technology Reform. Implementing partners pointed to other pivotal factors. In 2011, central MOE officials, in a study tour to the US, visited STEM high schools and met with STEM technical specialists. President Obama’s June 2009 speech at Cairo University also highlighted the importance of science and technology, and increased the rationale for USAID support for STEM education. Finally, the success and media exposure of the first cohort of students in the STEM School for Boys in 6th of October and STEM School for Girls in Maadi bolstered interest among parents, teachers, and political leaders.15 Conclusion: The balance of skills and experience within the implementing partners’ team contributed to the achievement of outcomes, through an incremental process that involved adaptation to changing circumstances and initiative. Findings and Supporting Evidence: Key individuals in the World Learning management team had prior experience within the MOE. This contributed to their credibility and access to MOE constituent organizations, and to their understanding of the interactions, protocols, and parameters of ministry operations. This insider/outsider status allowed the implementing partners to bring the diverse organizations together in the Central STEM Unit (interviews with STEM Unit members and evaluator observation). At the same time, the technical sub-grantees (21PSTEM, TFI, and TIES) brought strong international expertise in STEM program development from on-the-ground experience. In interviews and group discussions, school staff and Central STEM Unit members cited the value of this practical, applied expertise. In interviews, the technical team stressed the importance of their previous experience working together for planning and collaborative division of labor. They were engaged as a team in support of the new, innovative task of building STEM education in Egypt. Conclusion: The participatory approach, incremental successes, and the sense of being a part of “something new” have contributed to strong ownership. This is reflected in the Central STEM Unit, the local STEM units, administrators, teachers, students, and school BOTs. Findings and Supporting Evidence: Another positive characteristic of the implementing partner’s approach was its clear participatory and supportive role in helping the MOE and STEM school staff take ownership and control of the STEM school model. Its intention from the beginning was to ensure that the STEM model, curriculum, and assessment be Egyptian, and be firmly grounded in the Egyptian public education context. Both Egyptian and American stakeholders interviewed by the evaluation team cited the value of drawing on international experience, rather than simply adopting the US model. A particularly poignant observation for the evaluation team is the across-the-board commitment of stakeholders to the STEM initiative and schools. Central STEM Unit members spoke of their involvement in terms of a passion or dream. Teachers and students referred to the importance of being a part of a new system (as compared to the mainstream system). In the schools, students and staff spoke of themselves in terms of “family” and “team,” and proudly described the characteristics that distinguish 15 For example, Yasmine Yehia Moustafa, a Maadi student, won the 2015 Intel international ISEF competition, and NASA named an asteroid after her. 15 their school from the others. Students see themselves as “Stemmers,” and maintain contact with their classmates through social media, and with students in other schools. In discussion groups, STEM school graduates stated that they maintain this identity and contact even after graduation.16 The commitment is not only verbal. Teachers and administrators work long hours, frequently meeting with students after school hours. Summer and semester breaks are spent in training. According to the interviews, STEM Unit members (central and local) do STEM work in addition to their normal responsibilities, without additional compensation. A frequently-voiced concern, particularly among teachers and students, is that things could revert to the old way. Their criticism and identification of problems were communicated in such a way as to improve and strengthen the STEM model, rather than abandon it. Survey data confirm this impression. As much as 74.9% of male students and 82.4% of female students said they recommend the STEM school to others; 94% of graduates gave the same response. Constraining factors: Evaluation Question 2 references factors that have constrained the implementing partners’ ability to achieve desired outcomes as well as those that supported the process. In addition to the positive factors in the external environment (i.e., broad recognition of science and technology as key elements of Egypt’s economic growth) and the implementation process (the focus on participation and collaboration) that supported the implementing partners, other factors constrained or modified implementation plans and outcomes. As a process, the STEM school initiative and the implementing partners had to adapt to changes in the society and government, and unexpected consequences in order to continue moving forward. Conclusion: The unanticipated rapid increase in the number of STEM schools contributed to the reallocation and scarcity of resources (especially human resources); to disruptions in schools’ operations; and to problems recruiting teachers and students, and retaining students in the first year. Findings and Supporting Evidence: The decision about when and where to establish new schools was external to the ESSP and to the Central STEM Unit. The ESSP was designed to support three to five STEM model schools. Opening seven new schools in 2015 required the ESSP to stretch resources intended to incubate the original five schools. Furthermore, the fact that some of the schools were located in more remote sites forced a change in methods of communication and interaction with the technical team, with an increased reliance on virtual methods and less in-person contact (ECASE Quarterly Report, July-September 2015). The requirement to recruit, hire, and train a large number of new staff also resulted in modifications to the training program. In group discussions, experienced teachers reported that, initially, because of the predominance of new teachers, the training content was repetitive for them, and not useful. In 2016, the technical team revised the training program to include separate sessions and topics for new and experienced teachers (ECASE Annual Report 2016). In group discussions and interviews in Maadi and 6th October, teachers and administrators noted a relaxation in qualifications for STEM teachers to meet the urgent staffing requirements. Teachers also stated that some experienced staff from these two schools were transferred to the new schools, thereby contributing to staff turnover and complaints about reduction in the quality of instruction and utilization of the STEM school teaching methods. Administrators confirmed that most, though not all teachers went through the standard tests and interviews for hiring, but some did not receive the 16 According to the survey of graduates, 59.6% (n=56/94) communicate regularly with STEM students, while 89.2% (n=83/93) communicate regularly with STEM graduates. 16 introductory training in curriculum, capstone, and pedagogy before they started. (Please see the discussion in Evaluation Question 3 about teacher qualifications, and the extended discussion of teacher qualifications and data on background and training in Evaluation Question 4). Key elements of the STEM model — Internet access points, FabLabs, science labs, adequate electricity, and dormitories — were not complete when the seven new schools opened in 2015 (ECASE Annual Report 2016; field observation; and school-based interviews and discussions). The incomplete facilities, especially the dormitories, were reflected in low enrollment rates (e.g., in Luxor and Assiut) and high dropout rates17 (see Annex V, Table A.1). In Luxor, the problems continue, as there are no on-campus dorms and the FabLab was completely equipped only in February 2017, a year and a half after the school opened. Conclusion: Political turmoil caused intermittent disruptions and delays in the proposed implementation process. Findings and Supporting Evidence: The delayed start date for the ESSP did not allow for the design phase anticipated in the original work plan before engagement with the STEM School for Boys in 6th of October and STEM School for Girls in Maadi (ECASE Annual Report, 2012-13). The report also notes that the schools and the students were distracted by general unrest in the country, which also impeded the travel and presence of the US technical experts for technical assistance and training. Instability and turnover in the MOE, with seven ministers in four years (2012-2016), also affected the pace of implementation. With each change, USAID and the ESSP had to explain the purpose and structure of the STEM model schools and network (ECASE Quarterly Progress Report, July-September 2015). Despite the change of ministers and their varying initial levels of support and understanding, the STEM school initiative continued uninterrupted, and the MOE contribution to their operations grew. The USAID recalibration in October 2013 required abandoning the approved annual implementation plan and terminating associated activities, as well as developing a wind-up plan for the orderly close-out of the program. Normal activities resumed after March 2014, when the original implementation plan was reconfigured. According to the Evaluation Scope of Work, the one-year extension of the ESSP from August 2016 to August 2017 was justified, in part, by the cost of the wind-up period. Conclusion: Centralization of the mechanisms to form public-private partnerships (PPP) has restricted schools’ access to financial resources from potential local partners for local schools. Findings and Supporting Evidence: In the original design, public-private partnerships were envisioned as a source of revenue for sustaining the STEM school network and its technological resources, such as electronic equipment. They were also meant to be partners in extra-curricular activities for students to expose themselves to broad career and research options and opportunities (Cooperative Agreement No.AID-263-A-12-00005, ECASE, 28 Aug 2012). This aspect of the program has not developed as planned and, for more than a year, the position on the ESSP team for developing partnerships remained vacant. Some partnerships were arranged in the initial years of the ESSP between World Learning and private firms and NGOs for the STEM program (ECASE Annual Report, September 17 Based on discussions in Alexandria, where dormitories opened this year, and in Luxor, which is still without on-campus facilities, the lack of on-campus dormitories is significant for two reasons. First, it limits student access to school facilities (especially the labs) outside of school hours, and constrains student-centered non-academic activities. Second, parents perceive that students, especially girls, are more secure and supervised in on-campus than off-campus facilities. 17 2014). However, implementing partners explained that partnerships between the ministry and private sector entities are difficult and require central high level negotiation and approval, which is not attractive to private sector firms. The recent economic crisis and currency devaluation have depressed the possibilities further. At the local level, partnerships with local universities and research centers, as well as local business and industry, were seen as a source of financial and in-kind support for the schools, to be arranged primarily through the school BOTs, and as a resource to enrich student learning. These protocols also require approval by the central ministry. Moreover, funds cannot be donated directly to the school through the BOT as a local institution, as in the past, but rather, recently, a Single Account was established to receive all donations/revenues from all schools (interviews with World Learning and school administrators). An additional complication, which further limits the extent to which the BOT can tap local entities for student extracurricular activities, is the restriction on entry to the schools for individuals who are not parents or BOT members. School entry also requires central ministry approval. Recommendations: 1. USAID should collaborate with the MOE to explore alternative approaches to partnering with the private sector and the community. This should be based on a mutually beneficial relationship, and draw on the US experience with partnerships between private firms and government entities. 2. The MOE should formalize the legal status of the STEM school Boards of Trustees (BOTs) so they can act on behalf of the school within the local community, hold and manage funds, and tap into community resources (financial, academic, and experiential) to benefit the students and school [See Evaluation Question 1, Recommendation 1]. EVALUATION QUESTION 3: WHAT FACTORS HAVE FACILITATED AND/OR IMPEDED STUDENTS’ IMPROVED CAPABILITIES IN THE ESSP EDUCATION SYSTEM? In the student and graduate surveys, students assessed their experience in the STEM schools as strongly positive overall. Within this context, the student assessment of the components of the STEM school model, drawing on the group discussions in each school and the student survey, emphasize the fact that the model is still evolving, and illustrate the value of continuous assessment by stakeholders and beneficiaries. Conclusion: The capstone project is at the center of the STEM education model in terms of both pedagogy and student assessment. The capstone project, and its various stages such as the journals, account for 60% of the student’s grade in years 10 and 11, and 20% in the final year. In discussion groups and in the student survey across all schools, students recognize and praise the capstone project as a multi-dimensional learning tool that engages them in project-based/inquiry-based self-learning. It is cited as a reason to apply to the STEM school. At the same time, students and some teachers suggest a need to refine some aspects of the capstone experience to address excessive time investments and distraction from other aspects of their education. Findings and Supporting Evidence (a): In the student discussion groups in all the schools, the capstone was cited as both a reason for entering the STEM school, and an important feature that distinguishes STEM education from mainstream education. Students value the opportunity to solve problems and take pride in their solutions. In some groups, primarily for grades 11 and 12, students 18 commented on the importance of grounding the capstone themes in Egypt’s Grand Challenges, since it strengthens their awareness and understanding of the potential for practical application of their projects. Many said they would like to have the opportunity to share their projects more broadly in a public arena. Teachers, capstone leaders, and students all agreed that applying the Engineering Design Process (EDP) to students’ work on the capstone project, particularly in grade 12 where the parameters for student solutions are broad, often advances students’ research skills and knowledge beyond that of the teachers. Students value the opportunity to access expertise in the education and research community outside the school to expand their knowledge. The ease of access to these resources is cited by students, teachers, and administrators as a differentiating factor among STEM schools. For example, access to university facilities is particularly difficult for students in Luxor and Hurghada, both of which do not have local universities. In both cases, the nearest campus is South Valley University in Qena. The campus is a one hour and 35 minute drive from Luxor (68.1 kilometers), and a two hour and 41 minute drive from Hurgada (221.2 kilometers). Other positive factors associated with the capstone experience raised in student and graduate discussion groups include the strong value they attach to team work. Results from the student survey suggest that female students, in particular, stress this component of the experience.18 Students also recognize the importance of the capstone experience in building their presentation skills and ability to express themselves in English. This assessment of the capstone experience gleaned from the student discussion groups is mirrored in the responses to the student survey. The questionnaire asked students to rate their satisfaction with various aspects of the capstone experience on a scale of 1 to 5 (least to most satisfied). Across all categories except two, a majority of students gave a rating of 4 or 5 (See Annex V, Table B.5). The two lower assessments (application of fundamental concepts, 46.6%; and assessment of the capstone, 46.5%) are discussed in the following section. Findings and Supporting Evidence (b): While students lauded the capstone project experience, they also identified negative aspects of the experience. The positive and negative factors emerged from the same student and teacher discussion groups. Both students and teachers cited instances of misalignment across semesters between the curriculum/learning outcomes and the capstone themes. This caused frustration and duplication. For example, students said a capstone project in the second semester of one year required applications of electricity that were not dealt with in the curriculum until the first semester of the following year (this type of frustration is likely reflected in the survey response noted above). Students invest significant non-classroom time in project preparations; most also expend personal funds (up to 800 LE) on materials for the projects. Given the time requirements for the capstone, students complain about having to attend class sessions as well. They contend that the time would be better spent on the capstone project, where more learning takes place. Students cite the inadequacy of the teachers/capstone leaders in facilitating capstone projects, due to their lack of academic knowledge, facilitation skills, and understanding of the capstone themes.19 Students also are dissatisfied with the time 18 On a scale of 1 to 5, from least to most satisfied, 72.8% of girls, compared to 59.1% of boys gave a rating of 4 or 5 to the item, “peer learning and group work.” 19 In interviews, the STEM technical specialists expressed frustration that the students devote an inordinate amount of time to the projects because they enjoy them, to the neglect of classes and learning outcomes. The specialists also explained that, to some extent, the criticism of the capstone teachers/leaders reflects a misunderstanding of their intended role, which is to guide, not to provide answers. 19 required for capstones and classes, and the lack of time and resources for activities such as sports, music, and art. “We’re not just scientists.” Students and teachers in all the schools discussed problems and dissatisfaction with the assessment of the capstone projects, as per the survey response. Students complain about the weight given to the capstone, relative to the required time and commitment (60% in grades 10 and 11, and 20% in grade 12). Students and teachers also observe inconsistencies in the assessment standards across schools. Assessment records provided by the schools generally confirm that students in the STEM School for Boys in 6th of October and STEM School for Girls in Maadi have higher scores than those in the newer schools (see Annex V, Table A.9).20 Finally, in all the schools, teachers and students are dissatisfied and highly critical of the entire journal process, including questions, assessment rubrics and process, time requirement, and fairness. The centralization of the journal process and timing across schools are intended to ensure “fairness,” but no one in the schools is defending this exercise.21 Conclusion: As above, student reaction to the laboratories has two sides: (a) laboratory facilities in the schools, especially the FabLabs, support student academic achievement; at the same time, (b) students face difficulties in fully utilizing the laboratories, especially the FabLabs. Findings and Supporting Evidence (a): Another key aspect of the STEM school model is the laboratories for students. In assessing this component, students recognize the value of their hands-on experience for applied learning and development of scientific operational skills. In group discussions, they contrasted their opportunity to perform experiments and observe the results to mainstream schools, where students memorize what the results would be, but do not observe them. Students use the labs for both classes and capstone projects. In schools where the option is available to them, they welcome the opportunity to work in the labs on their own research, outside of regular school hours. In group discussions in five schools, students specifically mentioned the FabLabs and the subject-specific labs in response to the general question, “What have been the most beneficial or useful features of the school for you in terms of your learning and experience?” FabLabs are relatively unknown in Egypt.22 With the support of the FabLab managers, students have created FabLab Student Support Teams to provide peer support and collaboration. Findings and Supporting Evidence (b): At the same time, students voice considerable frustration with accessing the labs, especially in preparation of their capstone projects. The satisfaction ratings for laboratories on the student survey reflect this frustration (see Annex V, Figure B.4). On a scale of 1 to 5, where 1 is the least satisfied and 5 the most, only 41.5% of the students give the laboratory facilities a rating of 4 or 5. Approximately the same rating was given to laboratory supplies. For the FabLabs, while 41% are mostly satisfied, a third of the students (33.2%) are dissatisfied (rating of 1 or 2). 20 Please note, however, that the data provide no basis to explain this difference (e.g., bias in favor of the flagship schools, experience, quality of teachers/students). 21 An additional problem is that all students in the entire STEM network must respond online to the journal questions at exactly the same time, and the Internet capacity is insufficient, causing outages and delays. 22 A FabLab is a technical prototyping platform for innovation and invention. The only FabLab outside the STEM schools is FabLab Egypt in Cairo. Established in 2012, it is the educational outreach component of MIT’s Center for Bits and Atoms (CBA), an extension of its research into digital fabrication and computation. The lab is a member of the Massachusetts Institute of Technology Fab Lab global network, and is available to the public. 20 In the group discussions, students (and teachers) cite various sources of dissatisfaction. The level of dissatisfaction and types of criticism vary across schools. Male students tend to be more satisfied than female students. In all schools, students and teachers report a lack of materials and supplies, especially for the capstone projects. Furthermore, students’ access to the FabLabs is often limited to school hours because the FabLab Manager must be present, due to safety concerns. At the same time, students and teachers report extensive after-hours use of the other laboratories, particularly during capstone projects. A key frustration for the other labs (especially the geology labs) is lack of supplies. The evaluation team interviewed the FabLab managers, most of whom are relatively new to the STEM schools. They voiced frustration with the lack of materials, delayed and restricted access to maintenance and repair services (in part because of warranty restrictions), and complications in maintaining software licenses for some of the equipment. Likewise, the absence of climate control (especially air conditioning and dust) damages and shortens the life of some of the equipment. The FabLab managers have formed a peer support group and confer weekly through GOTO MEETING. Many are interested in securing additional training on the labs and also techniques for interacting with students. They express a general concern about the long-term maintenance and updating of equipment in the absence of the ESSP. Conclusion: Access to international competition contributes to the development of globally competitive scholars. Findings and Supporting Evidence: Students believe that the opportunity to participate in national and international competitions is one of the benefits of attending a STEM school. In group discussions, they speak with enthusiasm about their plans to compete and, within the STEM school community (i.e., students, teachers, administrators), winning or placing in competition is routinely cited as an indicator of achievement – for the students and school. According to the ECASE Quarterly Report of July￾September 2014, the ESSP worked with the MOE Technology Development Center, which had an ongoing relationship with the Intel Science and Engineering Fair, to secure the participation of students from the STEM School for Boys in 6th of October and STEM School for Girls in Maadi in 2014. This was done through a pre-selection process in the 6th October school. Even though Egypt had participated in the fair for seven years before 2014, this was the first year that Egypt had competed successfully internationally (see Annex V, Table A.3). In discussion groups, the students said that, in the mainstream schools, they would not have known about the competition or had the support of teachers. According to the Capstone Leaders, projects for competition often derive from the capstone projects, although the team composition may vary. US technical experts also observed that the presentation skills derived from the capstone projects give the STEM students an advantage over others in the competitive arena. Conclusion: The boarding school model and dormitories contribute to students’ personal development, school commitment, and to the STEM school learning objectives. However, students note a lack of trained personnel in the dorms to deal with their personal issues and medical problems, and they are dissatisfied with the accommodations. Findings and Supporting Evidence: The first two STEM schools, the STEM School for Boys in 6th of October and STEM School for Girls in Maadi, were designed as boarding schools because they recruited students from all parts of Egypt. The schools that opened subsequently are regional schools that bring in students from more than one governorate. Even within the governorate, some students live too far away to commute daily. Unlike the STEM School for Boys in 6th of October and STEM School for Girls in Maadi, the 2015 schools are co-educational, with separate dorms and separate classes for boys and girls. The STEM schools are the first and only public secondary boarding schools in Egypt. While some students reported that parents were uncomfortable with the boarding school model (student group 21 discussions), the students themselves, especially the girls, generally praise the dormitory experience. It is important that the dorms be on campus, to allay parents’ concerns and maximize learning benefits. Some schools opened before completing construction of the dorms (e.g., Luxor, Assuit, Alexandria, and Kafr El-Sheikh). According to the school principals, this was reflected in enrollment and dropout rates (see Annex V, Tables A.1, A.5, and A.6). In discussion groups, students said that the dorms provide independence from family, and the possibility and need for independent decision making. They feel they have matured more quickly than their peers in the mainstream schools in terms of planning their own lives. The dorms also promote increased interactions and acceptance of personal and cultural differences. They become very attached to one another, like family, and support each other in academic and personal matters. Students utilize the school facilities after-hours for capstone projects and student-generated independent learning, including a variety of clubs (e.g., theater, debate, music, English proficiency, and computer technology). Teachers who board in the dorms often support their students in these activities. Many students, especially girls, commented that even though they can go home on weekends, they prefer to stay with their friends. Also, they accomplish more in terms of their studies. They seem to value the relative independence of the dorms. In discussion groups, some graduates commented that being isolated in the school around the clock allowed them to concentrate on their learning, without outside distractions. Nevertheless, they were so used to interacting only with one another, that communication with others in the university setting was difficult at first. Some current students (and school administrators) noted that dorm supervisors are not trained for this role23 and, overall, there is a lack of psycho-emotional support for the students and medical care/consultations. Regarding infrastructure and facilities, 72% of the students (male and female) on the student surveys were dissatisfied (i.e., rating of 1 or 2 on a five-point scale where 1 is the lowest and 5 is the highest). The ratings were especially low for the STEM School for Boys in 6th of October, Ismailia, and Kafr El-Sheikh, though none of the schools received high scores. All students complained about poor Internet and WiFi connections in the dorms. Because of the reliance on technology for their research, studies, and assignments, most students use their own flash drives. Conclusion: Students suggest that teacher qualifications be re-examined and their access to training be increased. Findings and Supporting Evidence: Current students and graduates point to teachers and the quality of instruction as major deficiencies in the schools. While students in all the schools are pleased with a few of the teachers, their overall assessment of the teaching staff is average. In all of the student discussion groups, when asked for recommendations to improve the schools, students included additional training for teachers. Students particularly noted the need for training in terms of teachers’ weak English skills and inability to teach in English, and the fact that some teachers continue to practice the mainstream teaching style of lectures and reliance on textbooks. They question teachers’ academic competence and maintain that teachers often are defensive and dismissive when students surpass them in knowledge or correct them. Students question the capacity of some teachers to judge their work on the capstone projects. In the survey of STEM school graduates, respondents were asked to select the three components out of a list of seven that are most in need of improvement. Seventy-seven of the 94 respondents (82%) chose 23 The dorm supervisors are employed by a private sector firm contracted to run the dorms and food service. In most schools, students also commented on the lack of social workers’ and school psychologists’ training and skills to handle psycho-emotional issues. 22 Teachers.24 In the survey of current students, respondents were asked to rate (on a scale of 1 to 5, lowest to highest) various aspects of teacher performance for STEM, English, and non-STEM teachers. The results are shown in Annex V, Tables B.7, B.8, and B.9. Generally, the students seem to be most satisfied with their English teachers, and least satisfied with the STEM teachers. The survey provides no explanation for this difference, which could be attributable to actual differences or to higher expectations for the STEM teachers. Unfortunately, the classroom observation data collected by the team also does not explain the differences. Students’ reflections on the issue of teacher quality echo the discussion of the supply and training of qualified teachers in Evaluation Questions 1 (Objective 3) and 4. Conclusion: Students identify multiple factors in the school management system that detract from the school experience and student performance. Findings and Supporting Evidence: The general student code of conduct and regulations are not tailored to the STEM schools, which are boarding schools that emphasize student initiative and independent study (see also Evaluation Question 1, Objective 4). Each school administrator has drawn up rules and regulations for his/her school, but they are not uniform across the system. In most student group discussions, students complained about their confinement, the restrictions on leaving campus, and the lack of entertainment and leisure time activities on and off the campus. A frequent complaint (from teachers as well as students) is that classroom sessions are too long (80 minutes) and, unlike the mainstream schools, students are required to attend. (They tell of being sought out in the dorms if they do not show up in class). In group discussions, students said they feel that policies concerning when they must be in the dorms and “lights off” limit study hours. (Administrators explain that these restrictions are for reasons of security and time management). Conclusion: Concerns about the effect of university admission policies on the future prospects for STEM graduates are causing students to drop out before graduation. Findings and Supporting Evidence: STEM school students are uncertain about their track to preferred and acceptable universities after graduation. This has plagued the model since the very beginning, and has increased with the growth in the number of schools, students, and potential graduates after next year. All student group discussions (grades 10 and 11) in all the schools raised this issue. Students note the strengths and weaknesses of their current STEM experience, and discuss the value they have derived from the educational model. Still, they contemplate plans to return to their mainstream schools at the end of grade 11 to take the Thanaweya Amma exam for university admission. The group discussions do not provide an estimation as to how many students will leave after grade 11. Many believe they have benefited personally from the two years of STEM education, and will carry this method of learning forward into their future. However, that future can only be guaranteed by leaving after grade 11 (see additional discussion of this serious issue in Evaluation Questions 1 and 4). Recommendations: 1. The MOE should ensure that an efficient and reliable management mechanism is in place, with adequate financial resources for infrastructure and equipment maintenance and supplies. The STEM school model is resource-intensive and highly dependent on technology. The FabLab, in particular, is a significant resource and substantial investment. Maintenance and supplies must be timely. Consideration could be given to decentralization of functions and responsibilities. 2. USAID should support the MOE and/or the Central STEM Unit in the development of a 24 The seven components were student-centered method; curriculum; teachers; infrastructure; extracurricular activities; administration; and capstone projects. The second highest category was Administrators (60/94). 23 school-based Code of Conduct, tailored to the STEM school model, and developed in consultation with the school community (administrators, teachers, students, and BOTs) (see also Evaluation Question 1). 3. The Central STEM Unit should consider student life in the STEM school model. See Recommendation 2, Evaluation Question 4; and Recommendation 1, Evaluation Question 5. 4. USAID and the Central STEM Unit should expand the mechanisms for strengthening and supporting teachers within the schools and during the work week, and build on the emerging experience with team teaching, coaching and mentoring, and peer support networks. EVALUATION QUESTION 4: WHAT ARE THE ACTIONS AND POLICIES THAT NEED TO BE TAKEN BY USAID AND MOE TO EXPAND ESSP EDUCATION TO ADDITIONAL GOVERNORATES? This question asks the evaluation team to articulate recommendations for high level (USAID and MOE) actions and policies. The response focuses, first, on the task of identifying which critical issues require actions and policies at this point in the STEM school development process. What are the salient threats and decision points requiring action and policy to move the STEM school process forward? Second, potential actions and policies are identified. Then, to the extent possible, these actions are presented in the recommendations, keeping in mind that recommendations must be actionable and within the manageable interest of USAID and the MOE. The issues draw on the analysis of findings and supporting evidence presented under the other evaluation questions (1-3, and 5) and are listed roughly in order of importance. Conclusion: Uncertainty about post-graduation status, opportunities, and implied risk poses an immediate threat to the continued viability of STEM schools as an alternative model of secondary education. Findings and Supporting Evidence: The most consistent problem for the STEM school model cited in all interviews and discussion groups is the problem of university admissions. The issue was emphasized most in the group discussions with the students and BOTs, which represent parents. In general, the issue is stated as a choice between a guaranteed and an unknown future. The uncertainty derives from lack of information about universities, admission requirements, career options, scholarship opportunities, and application procedures. Moreover, the effect of STEM school assessment processes on all these factors is contrasted with students’ and parents’ knowledge and experience with the process for admission through the mainstream schools and Thanaweya Amma. The conventional aspiration for talented students in public secondary schools in Egypt is to enter the faculties of medicine or engineering in a prestigious public university. Although many STEM students state their aspirations in terms of access to private technical universities in Egypt (e.g., Nile University and Zewail City) and study abroad, rather than in these conventional terms, they view those opportunities as limited, competitive, uncertain, and largely unknown. Moreover, STEM graduates see themselves at a disadvantage in admissions to the prestigious public universities.25 As a result, students 25 Ministerial Decree 202 of April 21, 2012 equates the STEM school “certificate” with Thanaweya Amma. However, according to students, teachers, and parents, when it is time to apply for universities, STEM graduates are delayed until the third wave of this process, when all places in prestigious universities have been granted. The September 5, 2013 Supreme Council of Universities Decree reserves a flexible percentage of admissions for STEM graduates in public universities. The flexible percentage promises STEM graduates a percentage of the total number of seats available in different faculties that year. That is, it is a percentage of available seats, not of total graduates. As a result, even if available seats increase, they will not be enough to 24 are dropping out of the STEM schools before grade 12 and graduation. The dropout rate for male students after grade 11 decreased from 20.6% in 2013 to 14.7% in 2016 (see Annex V Table A.5). However, the dropout rate for female students after grade 11 jumped from 15.7% in 2014 to 38.5% in 2016. Of the 115 girls who entered the Maadi school in grade 10 in 2014-15, only 64 have continued to grade 12 (see Annex V, Table A.6). Forty students dropped out at the end of grade 11. The numbers from 6th October also show a substantial increase in dropouts between 2016 and 2017 (Annex V, Table A.6). In 2013, 131 boys entered grade 10, though only 121 continued to graduation. In 2014, 138 boys entered grade 10, and only 116 continued to grade 12. In group discussions, students, especially girls, maintained that more girls than boys are leaving because females generally have fewer options for the future than males. Boys can choose to take a year off, to travel, or to work in a variety of jobs, while girls are more restricted by their families and their own perceptions of future career opportunities that are open to them compared to boys. The girls, and especially their parents, are less likely than boys to take the risk. Teachers and administrators also confirm this assessment. To date, the only data on students in grade 12 is from the STEM School for Boys in 6th of October and STEM School for Girls in Maadi. Clearly, these statistics could be affected both by circumstances unique to these two schools, as well as by concerns about university admissions. However, interviews conducted by the evaluation team in the other seven schools, which will have grade 12 students next year, confirm the likelihood that the pattern will be replicated next year across all the schools. This pattern includes the greater dropout rate among females. If this issue is not resolved, it could affect student achievement and enrollment, the viability of the STEM school model, and the long-term benefit of this investment for quality education in Egypt. Students and parents cite the following statistic. Nine schools will produce STEM graduates in 2018, as compared to two schools in 2017, with the potential for 939 graduates competing for university placement opportunities in 2018, compared to 182 in 2017. In their perception, the reserved spaces in public universities and the scholarship opportunities are insufficient to meet the needs and aspirations of all the graduates. Hence, if students return to their mainstream schools and prepare for the Thanaweya Amma, they will be assured a place in the preferred universities since they are high achievers. In the discussion groups, graduates said they would have benefitted from more mentoring in the STEM schools, and also from more guidance and social support about future plans. They needed more information about universities, scholarships, and test preparation. Financial constraints are a significant issue. Students offered partial scholarships or acceptance to universities abroad cannot take advantage of them because of the additional costs. When current students were asked how they get information about scholarships and other opportunities, they said they rely on word-of-mouth from other students, especially those ahead of them in the school. Stakeholder Proposed Actions and Policies for the MOE and USAID: Students, parents, and administrators proposed a variety of approaches to the problem, some of which are beyond the scope of the evaluation. ● Create a STEM track in Egyptian public universities; ● Establish a public STEM university; ● Increase the number of scholarships earmarked for STEM graduates; ● Guarantee all qualified STEM graduates a place in the public university of their choice; accommodate the growing number of STEM graduates. 25 ● Establish a dialogue between the Ministry of Education and the Ministry of Higher Education to set a defined course for STEM graduates to follow for university admissions; ● Establish a guidance center in each STEM school to advise students and parents on opportunities and options for higher education and career possibilities. Provide assistance in completing university and scholarship applications, and tools and practice sessions to prepare for entry exams. Conclusion: The number and capacity of available teachers is not sufficient to staff additional STEM schools with qualified teachers. Findings and Supporting Evidence: The most recent ESSP Annual Report (September 2016, 8) confirms the above conclusion. “Although the recruitment process is now completely owned and operated by PAT, the number of teachers passing through the system is less than what the schools require, as more applicants are required now that more schools are opened.” The qualifications for teaching in a STEM school, as defined in the MOE Decree 382 of 2 October 2012, are high. The intent is to attract the best teachers from the MOE system, consistent with the demands of gifted students and high-quality education. Particular attention is given to the skills of the STEM teachers (mathematics, physics, chemistry, biology, and geology) and English teachers. The qualifications are: previously traveled abroad on educational missions; MSc or PhD holders from among teachers working for the MOE or Egyptian universities; and expertise in the English language. Currently, all teachers are MOE employees, and university professors are not being considered. The teacher survey (n=265) shows the extent to which these qualifications are (and are not) being met. For the highest grade completed in the area of specialization, 13.2% have either an MS or PhD degree, 40.4% have a BA/BS, and 46.4% have a Diploma. Among STEM teachers (n=109), 14.6% have an MS or PhD degree, and 50.5% have a BA/BS. Two-thirds (66%) of the teachers, STEM and non-STEM, said they had participated in some STEM-related training before entering the STEM school program; 9.5% had traveled abroad for training. The teachers in the STEM schools, both STEM and non-STEM, are experienced teachers; 70.7% have taught for 10 years or more. The hiring process, managed by PAT, begins with an application, followed by a test of basic concepts and English proficiency. The formal application process, announced on the PAT website, occurs in April. Candidates who pass these two exams enter a two-week training program that introduces them to the fundamental components of the STEM education model. Finally, candidates are interviewed and hired by the appropriate Counselor’s Office in the MOE. According to data received from PAT, for 2016-2017, 1,589 MOE teachers applied for STEM positions, but only 311 sat for the tests, and 31 attended the training.26 Currently, there are 243 teachers in the nine STEM schools included in the evaluation — 103 STEM teachers (18 females, 85 males) and 140 non-STEM teachers (46 females, 94 males) (see Annex V, Table A.7). The number of teachers per school varies between 26 in Luxor and 46 in Maadi, with an average of 37/school. Interviews with school principals and STEM Unit members highlighted three factors that are important in assessing the availability of teachers. First, teachers prefer to work in their home governorates, and location is considered an important factor in hiring. Second, the pool of qualified teachers varies by 26 Statistical Report of Ministry of Education Teacher Applicants for STEM School Qualifying Examination and Training, Information, Communications and Technology Department, Professional Academy of Teachers, 2016/2017 Academic Year. 26 region, especially in terms of English proficiency, but also in the STEM subjects. English proficiency is highest in the large urban areas. Third, teacher turnover is a concern. A total of 141 teachers trained by the ESSP are no longer teaching in the STEM schools. This number is equivalent to 40.7% of the current number of teachers in the nine STEM schools (see Annex V, Table A.4). When asked in discussion groups why they joined the STEM school, teachers responded in much the same terms as the students — to try something new, to develop a different vision of education, because of the labs and teaching method, professional development and a learning community, the classes are small, the quality of the students, and the financial incentives. Some teachers also said they were nominated or advised to apply by their supervisors27 and decided to try it. Teacher turnover in a school may be due to departure or transfer, both of which are common. Many in the STEM system (STEM unit members, implementing partners, school staff) commented that teachers left because they were highly trained and acquired more lucrative positions in the gulf states. The data suggest that this process was more common in the early years of the program in the STEM School for Boys in 6th of October and STEM School for Girls in Maadi, than in recent years. The team did not encounter this situation in discussions about departures during the past year, for example. Recent departures are explained more in terms of the factors that teachers say they would like to change about the STEM school system — excessively long hours and heavy work load, lack of sufficient financial incentives (especially in view of the recent devaluation of the Egyptian pound), absence of paid vacation (any leave is deducted from salary, and school breaks are spent in training), and lack of job stability and a defined career path. Some also left because they could not adjust to the new pedagogy. Since the 2015 schools are only in their second year, turnover there has been low (i.e., one to four teachers). The clearest manifestation of the shortage of qualified teachers may be the assessment of teacher quality by current students and graduates (Evaluation Question 3). This is seen as a primary impediment to student learning in the STEM schools. Stakeholder Proposed Actions and Policies for USAID and the MOE: ● Re-examine the package of incentives for STEM teachers, including financial and non-financial benefits, and a clear definition of the STEM teacher career path; ● Broaden the recruitment pool by changing the requirement that applicants be experienced MOE employees; ● Consider options to discourage teachers from leaving the MOE after training (e.g., payment of a fine for leaving before completing a certain number of years in the system, or reimbursement for training received; ● Strengthen the program of continuous training in response to needs — based on requests, coaching, and classroom observation; ● Consider options to increase the supply of qualified teachers, to meet the growing need as new schools open, and the potential demand in mainstream schools through, for example, a re￾examination of the education model taught in the faculties of education so that new graduates have the skills to teach according to the STEM student-centered educational model. Conclusion: The current pace of expansion of STEM schools presents a risk for the quality and sustainability of the STEM school model and initiative. 27 Because of the shortage of applicants, supervisors in the mudereyas were asked to identify and nominate talented and qualified teachers for the STEM school in their regions. In some group discussions, teachers said they or their newest colleagues were simply told by supervisors to report to the STEM school (from group discussion at PAT, with local STEM unit members, and with teachers and administrators). 27 Findings and Supporting Evidence: Owing to the early success of the STEM model and STEM graduates, the MOE has pledged to establish a STEM school in each of the 27 governorates. In addition to the 11 schools already operating, four more are expected to open next year. In key informant interviews and discussion groups with STEM Unit members (central and local), implementing partners, school administrators, BOTs, teachers, and students, the response to the question about future expansion was that new construction should pause or slow down. In some cases, particularly in the two original schools, reasons focused on the “dilution” of the model, which was intended to create a corps of STEM leaders to strengthen future development and growth in Egypt, as well as Egypt’s position as a leader in science and technology in the Middle East. Spreading existing resources (especially human resources) across multiple schools will drain the flagship schools as well as the potential for leadership development. Others applaud the objective of broadening access to the model and the alternative approach to education, but point to the risks for quality and sustainability, because of human resource constraints and operating costs. To date, the ESSP has managed the processes for maintaining equipment and infrastructure and, to a large extent, for stocking and delivering operational supplies. Based primarily on interviews with school administrators and BOTs, the process for the MOE to assume these responsibilities is not clearly defined. To the extent that it is defined on paper, it is not operating effectively in the schools. When asked about their concerns for the future after the end of the ESSP, maintenance and supplies were high on the list. Some schools already operating are not complete, especially in so far as equipping laboratories and constructing dormitories. For example, the dorms in Kafr El-Sheik were completed in March of this year, and in Luxor, the students are housed an hour away from the campus (see discussion in Evaluation Question 3).28 As discussed above, schools that opened in 2015, before they were ready, experienced negative consequences in terms of the reputation of the school in the community and with parents. Administrators in Upper Egypt, in particular, discussed this issue in terms of the need to explain and “sell” the school to secure community acceptance. Some stakeholders also question the criteria for school locations. Not all governorates offer ready access to the universities, research centers, or industry for the extra-curricular aspect of the STEM model. Stakeholder Proposed Actions and Policies for USAID and MOE: ● Develop technical and operational benchmarks for school operations to guide the pace of new school construction (e.g., state of operation of existing schools in terms of student/teacher population, budget allocations, professional development goals); ● Establish clear lines of authority and operations for school maintenance, and monitor responsiveness and resolution. Conclusion: The decision-making structure and lines of communication for the STEM school initiative are unclear and not institutionalized. Findings and Supporting Evidence: As presented in Evaluation Question 1, ESSP support for the development of the STEM school model and the MOE STEM school initiative has focused on a participatory process of innovation and systematization. This process has produced impressive results, but is incomplete. While the technical structure for the STEM school initiative, which includes the Central STEM Unit, the local STEM unit, and the STEM schools, has been formalized within the MOE structure via Decree 136 28 Reportedly, the two schools opened this year in Gharbia and Menufia also are not complete. 28 of April 2, 2017, and designates the STEM UNIT as part of the General Education sector reporting to the First Undersecretary/Head of the General Education Sector, this structure does not specify channels for decision-making or communications in specific key areas. For example, the STEM Unit is not involved in decisions about school construction and infrastructure, budget, resource allocations, or public/private partnerships. This pending formalization is a potential point of vulnerability in terms of sustainability. The ESSP has played a key role in bringing together and training the main MOE entities of the Central STEM Unit, and formalizing the unit. An observed potential weakness in this structure is the reliance on individual STEM champions within these organizations. The integration of the STEM model within the organizations is not yet complete. Other indicators that the STEM school model is still not fully institutionalized within the MOE (which have been discussed above) include: ● STEM school professionals are uncertain about their career path; ● STEM school BOTs and Student Unions question whether the decrees for the legalization of these entities apply to the STEM schools, since they are not explicitly cited. BOT and Student Union representatives have reported cases in which their demands have been rejected on the grounds that they cannot legitimately make such demands; ● STEM schools are not included in local (mudereya, district) competitions and other activities for local public schools (e.g., sports events, festivals); ● STEM schools have no school-managed financial resources because the students do not pay fees. The STEM school BOTs also do not have the authority to open bank accounts for their schools. Stakeholder Proposed Actions or Policies for USAID and the MOE: ● Examine and clarify the position and decision-making authority and lines between the STEM Unit and other departments at the central and mudereya levels; ● Consider developing protocols with entities such as NCEEE, CCIMD, NCERD, and GAEB to internalize the reform process, and consider applications to mainstream education. Conclusion: Ambiguity among students and parents, and, in their eyes, among many preparatory school and university teachers about the vision/direction of the STEM school model raises questions about the projected nationwide expansion of the system and its sustainability.. Findings and Supporting Evidence: The interest of students, parents, and educators in the STEM school initiative is not necessarily a demand for a science and math curriculum. While the growth in applications and enrollment in STEM schools shows a demand for this type of school, the salient feature may not be science and mathematics, but a demand for an alternative educational model, based on problem-solving and applied learning, rather than memorization. As discussed in Evaluation Question 1, only 11% of STEM students surveyed (n=1,223) cited an interest in STEM subjects as the reason for enrollment. Most students (89%) gave reasons other than STEM education, such as “better education” (40.3%) or to “try something new” (28.2%). Teachers say they applied to the STEM schools for professional development, a better education model, to try something new, or for financial incentives. The lack of consensus and ambiguity among students, parents, and teachers about the vision/direction of the STEM school initiative means that the message being communicated is neither clear nor precise. This gap is evident in a lack of public awareness and information about STEM, especially among parents, preparatory schools, community leaders, and universities. Various impressions about the STEM schools are spread mostly through word-of-mouth and social media).29 They contribute to instability within the 29 In response to the student survey question asking, “How did you learn about the STEM school?” 52.1% heard about it through personal connections (friends, family, other STEM students), and 17.3% through social media. Another 17.9% were 29 student community, and with parents and higher education institutions because of differences in and uncertain expectations about the content of the program and options for the future. To date, with the exception of the pilot science education program in grade 7 of the preparatory schools, MOE outreach to preparatory schools is not systematic, and is largely dependent on STEM school principals’ individual initiatives (see Evaluation Question 1). Likewise, the MOE has not engaged in a systematic information campaign through the media or other formal communication channels, although the Central STEM Unit now designates a position for communications.30 Students (and some BOTs) talked about the potential value of using the students’ capstone projects to increase the exposure of STEM school accomplishments and their relevance to Egypt’s development challenges and needs, rather than simply putting them aside after the final scoring. Stakeholder Proposed Actions and Policies for USAID and MOE: ● A policy forum to discuss alternative scenarios for moving the initiative into the future, such as building key lessons or learning objectives into the mainstream secondary schools, re-examining the dominance of the Thanaweya Amma as the avenue to public higher education, and considering other specialty schools, in addition to STEM; ● Development of a formal media and outreach campaign to provide factual information about the STEM school experience and career options, as well as the vision or dream behind this type of education. Recommendations: 1. The Ministry of Education should pursue a dialogue with the Ministry of Higher Education to explore mechanisms in the public universities to satisfy the aspirations of STEM graduates and to retain and build on the accomplishments of the STEM school secondary education for the benefit of Egypt. 2. USAID should support the establishment of a guidance center in each STEM school to advise students on opportunities/options for higher education and careers, to provide assistance in completing university and scholarship applications, and to offer tools and practice sessions to prepare for entry exams (also see Evaluation Question 5). 3. USAID should support a re-examination of the curriculum in university faculties of education to incorporate instruction in the teaching skills and pedagogical methods used in the STEM school education model. (Also see recommendations about teachers in Evaluation Questions 1 and 3). 4. USAID and the MOE should develop technical and operational benchmarks to guide the pace of new STEM school construction (e.g., state of operation of existing schools in terms of student/teacher population, budget allocations, professional development goals). 5. USAID should support the development of a formal media and outreach campaign to provide factual information about the STEM school experience and career options, as well as the vision or dream behind this type of education. informed by prep school teachers, and only 13.7% heard about it through formal channels and the media. 30 Coordinator, Community Participation, International Cooperation and Communication. 30 EVALUATION QUESTION 5: TO WHAT EXTENT HAS THE ESSP SUCCEEDED TO ACHIEVE GENDER BALANCE? USAID policy states that its activities should seek to promote gender equality, and identifies key objectives for action. The ESSP did not address gender issues except in Objective 1, which specified, “a special effort for underrepresented groups such as girls…” In accordance with USAID practice, ESSP disaggregated all individual-level data and indicators to allow comparisons between males and females in terms of participation, results, and benefits. Gender inequality is not addressed in ESSP plans or reports, and there is no indication of an ESSP gender analysis or gender indicators. Conclusion: The ESSP has not applied specific measures to achieve gender equality. Findings and Supporting Evidence: The criteria to join the STEM schools as a student or teacher are gender blind. They do not take into account gender or factors related to inequality. As required by law, the application process for students is transparent and electronic, without any space for subjective judgment in selection. The final stage in the teacher selection process is a personal interview. The issue of gender bias was not raised in any of the discussions with teachers. Conclusion: The STEM school model is designed to serve males and females on an equal basis. Findings and Supporting Evidence: Initially, the MOE planned and built two schools, STEM School for Boys in 6th of October and, one year later, the STEM School for Girls in Maadi. The seven STEM schools established in 2015, and the additional two opened in 2016, are co-educational, with equal capacity for females and males in terms of dormitory space, activities, and class time. Males and females are separated in all academic activities and in the dorms. The evaluation team did not observe differences in school activities, rules, or standards based on gender. Male and female teachers have received the same amount and types of training, and have the same job responsibilities. The average number of training hours for teachers and administrators across subject matter has been about 30 hours/year between 2012-13 and 2016-17. For 2013-14, the average number of training hours was about 40, but there were no substantial differences in the average hours of training for STEM, English, and humanities teachers. All teachers in STEM schools have the same training requirements. Except for physical education, all teachers conduct classes for male and female students. Teacher discussion groups noted that, in STEM schools, the incentive compensation for STEM teachers is higher than for the humanities and activity teachers. Conclusion: Gender differences in student enrollment and in the teaching staff mirror differences in the population. For students in the nine schools included in the evaluation, 54.5% (n=1195) are males and 44.5% (n=959) are females. In Upper Egypt, approximately two-thirds of the students are male, reflecting regional differences in the secondary school population, and in the definition of the role of women. Differences in the number of male and female teachers, and in the subjects taught in STEM schools mirror differences in the teacher population. In Egypt, 63.4% of public secondary school teachers are male.31 In the STEM schools, 26.3% of teachers are female (n=64) and 73.7% are male (n=179) (see 31 D.B. Rao, Women as Educators, 2010. 31 Annex V, Table A.7). Most of the 64 women (73.7%) teach non-STEM subjects. Only 17.5% of the STEM subject teachers (n=103) are women. Conclusion: Gender differences exist among students in school performance and graduation rates. Findings and Supporting Evidence: While the differences in academic performance are small, in general, females outperform males in English and humanities in grades 10 and 11.32 Otherwise, on average, males and females perform the same. Significant differences by grade were observed between the two original schools and the newer schools, whereby the Maadi and 6th October had higher average marks, but the data do not provide any explanation for this difference (see Annex V, Table A.9). The single most apparent difference between male and female students, as discussed in Evaluation Questions 3 and 4, is the higher dropout rate for girls than boys. This is attributed to differences in parental pressure, and the perception of students and parents that girls have fewer career and university options than boys. (See the discussion of this issue in Evaluation Question 4, Conclusion 1). Conclusion: The ESSP and the STEM school model have focused on student academic issues to the neglect of the students’ non-academic life and psycho-social issues, including changes in gender dynamics. Findings and Supporting Evidence: Evaluation Question 3 noted that current and former students cite the need for more counseling and career advisory services. In discussion groups, girls, in particular, cite frustration with the changes they have experienced and the options open to them. Many express the importance of the ways in which the dormitory experience has liberated them from family and parental control. They talk about their independence and the importance of their friendships and alliances with fellow students. They also have a sense of pride and self-achievement in completing the capstone projects. Some female grade 11 students voiced a sense of resignation and personal dissatisfaction with the trend in female dropouts, attributing it to parental pressure and a lack of options (e.g., “it is not my decision to make”). Because of the lack of staff trained to counsel students in these issues, students rely on each other to solve problems without the benefit of outside information or professional counseling/guidance. Recommendation: USAID and the Central STEM Unit should consider incorporating student life and individual personal and career development into the STEM school model, through specialized staff, training, counseling services, and non-academic extracurricular activities. As a matter of best practice, gender awareness training should be included to alert staff to potential instances of bullying, intimidation, and gender￾based violence.33 32 Grade 12 academic performance data was not available to the evaluators. 33 It is not surprising that the evaluation team did not hear about any instances of bullying, intimidation, or gender-based violence since these topics are generally taboo, especially with outsiders. 32 OVERALL CONCLUSIONS AND RECOMMENDATIONS The technical support, leadership, and resources of the five-year Egypt STEM Schools Project have been critical to the advancement of the Ministry of Education STEM school program. The achievements since 2012 are concrete and substantial, including the commitment and momentum to continue. Measurable outputs — number of buildings equipped; students enrolled; students graduated; teachers, administrators, and MOE officials trained; BOTs formed; and textbooks and learning materials delivered — demonstrate the extent to which the ESSP achieved the original objectives. A tangible system of schools is now in place on the ground, with demonstrated positive results. The schools cannot be readily eliminated. The less tangible achievements, in terms of the process of creating a STEM educational model for Egypt, are equally important. The steps taken by the ESSP leadership and the technical team to ensure that the model is “Egyptian” is informed by the US STEM school experience. However, it is built around Egypt’s grand challenges, is adapted to local resources and expectations, and is molded by Egyptian culture, all of which are also significant. The commitment and ownership exhibited by stakeholders and participants are strong evidence of the value of building the program with the ministry staff and school communities from the beginning. The “handover” was not limited to the last year of the ESSP, but rather characterized the entire implementation process. The STEM school initiative is a process that is moving forward and will continue into the future. The evaluation identified factors within the operation of the schools that will improve the education experience for the students and teachers, and issues that require action and policy reform by the MOE and USAID in the future. To a large extent, these issues revolve around the fact that, to this point, the STEM school model has developed in a vacuum, separate from the broader public education system in Egypt. Students are prepared to think and work differently, but the outside system in higher education and the workplace is not yet prepared to receive them. Actionable recommendations directed to this broad issue of institutionalization and integration have been identified in each evaluation question, designating responsible parties. Recommendations for USAID Action 1. USAID should collaborate with the MOE to explore alternative approaches to partnering with the private sector and the community. This should be based on a mutually beneficial relationship, and draw on the experience in the US with partnerships between private firms and government entities. 2. USAID should support the establishment of a guidance center in each STEM school to advise students on opportunities and options for higher education and careers, to provide assistance in completing university and scholarship applications, and to offer tools and practice sessions to prepare for entry exams. 3. USAID should support a re-examination of the curriculum in university faculties of education to incorporate instruction in the teaching skills and pedagogical methods used in the STEM school education model. 4. USAID should support the development of a formal media and outreach campaign to provide factual information about the STEM school experience and career options, as well as the vision or dream behind this type of education. 33 Recommendations for MOE Action 1. In response to teacher uncertainty and dissatisfaction, the MOE, through PAT, should define the parameters of the career path for STEM teachers, within the provisions of the Special Cadre for Teachers Law, and institutionalize a corresponding professional development system. 2. The MOE should ensure that an efficient and reliable management mechanism is in place, with adequate financial resources for infrastructure and equipment maintenance and supplies. The STEM school model is resource-intensive and highly dependent on technology. Maintenance and supplies must be timely. Consideration could be given to decentralization of functions and responsibilities. 3. The MOE should pursue a dialogue with the Ministry of Higher Education to explore mechanisms in the public universities to satisfy the aspirations of STEM graduates and to retain and build on the accomplishments of the STEM school secondary education for the benefit of Egypt. Recommendations for Joint USAID/MOE Action 1. USAID should support the MOE in developing a comprehensive plan for institutionalizing the STEM school model, and addressing fundamental governance structures and arrangements, both within and outside the ministry, and at central, local, and school levels. The plan should include: a) the relationship and decision-making points, authority, and processes of the STEM Unit vis-à-vis other departments within MOE, as well as NCEEE, CCIMD, and NCERD; b) the relationship of the STEM schools to secondary education departments at the mudereya and district levels; c) resolution of issues related to the legal status of i) STEM school boards of directors, ii) representation of STEM schools in student unions and activities at local and national levels, and iii) a STEM school code of conduct. 2. USAID and the Central STEM Unit should expand the mechanisms for strengthening and supporting teachers within the schools and during the work week, and build on the emerging experience with team teaching, coaching and mentoring, and peer support networks. 3. USAID and the MOE should develop technical and operational benchmarks to guide the pace of new STEM school construction (e.g., state of operation of existing schools in terms of student/ teacher population, budget allocations, and professional development goals). 4. USAID and the Central STEM Unit should consider incorporating student life and individual personal and career development into the STEM school model, through specialized staff, training, counseling services, and non-academic extracurricular activities. As a matter of best practice, gender awareness training should be included. 34 ANNEX I: EVALUATION SCOPE OF WORK I. Activity Background Information A. Activity Identifying Information Award Title: Education Consortium for the Advancement of STEM in Egypt (ESSP). Cooperative Agreement: AID-263-A-12-0005. Total - USAID Amount: $29,965,648. Start Date: August 28, 2012. End Date: August 27, 2017. Activity Manager/AOR: Hala El Serafy. Evaluation Manager: Program Office, TBD. Partner: Ministry of Education. Implementing Partner: World Learning. Governorates of Implementation: Cairo, Giza, Dakahleya, Assiut, Alexandria, Ismaileya, Kafr El sheikh, Red Sea, Luxor, Menoufeya and Gharbeya. B. Background Over the last decade, Egypt has made significant progress in the basic education sector, particularly in terms of access. However, educational quality remains a serious concern, particularly the quality of science and mathematics education. The Trends in Science and Mathematics Study (TIMSS), an international periodic study of science and math learning conducted in approximately 60 countries at the 4th and 8th grade levels, has identified serious flaws in science and mathematics education in Egypt. These flaws were apparent in 2003 and again in 2007, both in absolute terms and in comparison, to other countries in the Middle East. As a result, only 30% of grade 10 students choose the science track of study, while 70% choose the liberal arts track, as indicated in the “Condition of Education in Egypt, 2010” report.1 These problems represent a major challenge for the education system in Egypt. USAID and the MOE have long collaborated to address key education activities, such as school construction, teachers’ professional development, use of technology in education, school governance and community participation. In recent years, this collaboration has evolved to address national education priorities and systemic improvements that will contribute to preparing students to be lifelong learners, equipping them with advanced skills for careers of the 21st century. Accordingly, USAID support for basic education is currently focusing on strengthening science and mathematics education, in addition to improving early grade learning. Specialized science, technology, engineering, and mathematics (STEM) schools that focus on providing strong academic foundation in science and math exist in many educational systems worldwide. The GOE recognized the value of these types of schools. As a result, the MOE established two STEM high schools: a STEM school in Giza for boys in 2011, and another school in Maadi for girls in 2012. These schools are informed by the STEM School experience in the U.S., and serve as centers of excellence, providing specialized education to the students who have the capacity to pursue advanced levels of science and mathematics, and progress to higher education according to their individual abilities and aptitudes. 1 The “Condition of Education in Egypt, 2010” report, prepared by the MOE with USAID’s technical assistance, describes the status of education, based the national education standards, during the period 2000‐2010. 35 Since 2011, a full slate of activities has been designed to support the establishment of STEM Schools. Due to the success of the STEM schools in strengthening students’ critical thinking, problem solving and project-based learning, the MOE decided to scale up the model by establishing a school in each governorate. To-date, nine additional schools were established. Seven schools were established in Dakahleya, Assiut, Alexandria, Ismaileya, Kafr El sheikh, Red Sea, and Luxor in 2015, and two more established in Menoufeya and Gharbeya in 2016. While the MOE builds the schools and provides basic equipment, USAID, mostly through the ESSP project, provides extensive technical assistance, advanced lab equipment and capacity building necessary to develop them as model STEM Schools. The teachers and students of these schools have been carefully selected through a competitive process. C. Project goal The overarching goal of the Egyptian STEM Model Schools is to create a network of self- sufficient schools with established curriculum, trained teachers, industry partnerships and operation under the MOE's public schooling system to educate high performing students from across Egypt, regardless of socioeconomic status. The STEM School model was designed to create “centers of excellence” that embody transdisciplinary collaboration, real world applications, critical thinking, and a pioneering spirit. Ultimately, STEM schools support a new vision of Egypt as a country equipped with both the human capital and the institutional resources to compete and excel in the international marketplace. D. Results to date To-date, eleven STEM schools have been established by the Ministry. Nine have been supported by the ESSP project. While the first two schools in Cairo and Giza have students enrolled in grades 10, 11 and 12, the seven schools established in 2015 have students enrolled in grades 10 and 11 only. Through partnership with the MOE, the project has designed integrated STEM curricula and assessment system (including a different Exit Exam and item bank for STEM schools), trained STEM teachers and principals, and building the MOE capacities through training of trainers, professional development for educators and institutionalizing STEM education within Egypt’s educational system. These schools have become a collaborative network that will serve as a catalyst for change, not only for future STEM schools but also for system-wide math and science education reform. They prepare the STEM students with specialized skills focused on project and inquiry-based learning, critical thinking, problem solving and innovative use of technology. The ESSP project reinforces collaboration with U.S. STEM schools and universities and raises awareness of specialized science and mathematics education among the public. Project activities focus on supporting the MOE through building knowledge, skills and systems in order for these schools to serve as models of best practices in STEM education and play an integral role in training and mentoring future STEM professionals. The ESSP project trains STEM teachers and trainers, develops curricula and assessment systems, and provides schools with books, educational materials, IT and lab equipment. To sustain and expand STEM education in Egypt, the MOE has established a STEM Unit, formed from representatives from all MOE technical centers and departments to coordinate and support STEM education in Egypt. Specifically, the STEM Unit consists of members from the National Center for Examination and Educational Evaluation (NCEEE), the Center for Curriculum and Instructional Material Development (CCIMD), the Professional Academy for Teachers (PAT), the National Center for Educational Research and MOE Science and Mathematics Advisors. The ESSP project works closely with the STEM Unit to build its capacity, knowledge and skills through involving them in the development of STEM curricula, extracurricular activities, and professional development and assessment systems. It also provides the MOE’s STEM Unit and schools with high level technical assistance grounded in the local 36 context and focused on sustainability. In 2015, the MOE established STEM units in the governorates where STEM schools exist. The project started to provide them with capacity building to enable them to provide day-to-day support for the schools. The prime implementing partner of the ESSP project is World Learning, which has assembled a consortium of three technical partners with extensive STEM experience namely: The Franklin Institute (TFI), The Teaching Institute for Excellence in STEM (TIES) and 21st Century Partnership for STEM Education (21PSTEM) to lead the development and implementation of technical program components, training and capacity building. E. Project Modifications One year after the beginning of the ESSP Project (in October 2013), implementation was interrupted due to a recalibration of U.S. assistance, and the project started planning for an orderly close out of its activities, and the project Cooperative Agreement was modified to reflect the wind-up plan. In March 2014, USAID/Egypt obtained the required authority to resume a broader range of programs. Accordingly, another modification was implemented to remove the wind-up plan and allow project to continue its activities as originally designed and awarded. F. STEM Extension In response to USAID’s request, World Learning and consortium partners (21PSTEM, TIES, and TFI) propose activities during the one-year extension (until August 27, 2017) that deepen capacity development support to the MOE, extend services to the four STEM schools in Ismailiya, Kafr el sheikh, Red Sea, and Luxor, and develop the systems to sustain the expansion of the nine schools. The activities continue many of the activities supported in the original program description, including a continued focus on the provision of: technical assistance to learning via training and professional development of teachers and STEM educators; provision of essential STEM IT equipment and classroom infrastructure, fabrication labs and textbooks; and the development of STEM curriculum standards, assessment system, and STEM teacher' standards, necessary for the establishment of the STEM Education System in Egypt. During the remainder of Year 5 and throughout the one-year extension period, the ESSP project will continue to strengthen the STEM education model in the first two schools in 6th of October City and Maadi, and support its establishment in the seven new schools to become fully functional and recognized as model STEM Schools at both national and regional levels. G. Link of ESSP Objectives to Strategic Plan and Results Framework This program supports USAID/Egypt's Education Objective (Assistance Objective 2.2) - Improved Access to Quality Education – through the provision of science and mathematics education and equal opportunities to interested and qualified students. Despite the impetus of change and commitment of the MOE to improving the educational quality, an institution this large is not easily reformed unless change comes from within. The ESSP Development Hypothesis suggests that, if targeted technical assistance and capacity building is provided to the Ministry of Education to support the establishment of a successful and functional STEM prototype, the potential to stimulate innovation and quality improvement across the education system will likely be accelerated. Hence, the ESSP Program Objectives are supporting the achievements of USAID’s Intermediate Results as follows: Intermediate Result 1. Expanded equitable access to education opportunities: supported through providing girls and economically disadvantaged students with targeted outreach and enrollment 37 opportunities. ESSP Objective 1. Increase student interest, participation, and achievement in science and mathematics with a special effort geared to underrepresented groups such as girls and economically marginalized students. ESSP Objective 2. Strengthen the STEM School local initiative through developing an effective model of specialized high schools focusing on science, technology and mathematics for gifted students. Intermediate Result 2. Improved instructional methods supported by providing professional development to teachers, administrators, supervisors, and other education staff at various levels. ESSP Objective 3. Build the capacity of a highly-qualified cadre of STEM professionals and provide opportunities for training and sustained, intellectually rigorous professional learning. Intermediate Result 3. Enabled public participation in education supported by strengthening governance structures, such as Boards of Trustees at the school level and Advisory Boards at the governorate and MOE levels, and developing their knowledge, commitment and advocacy skills regarding science and mathematics education. ESSP Objective 4. Strengthen MOE capacity at the systems and policy level to sustain and replicate these model schools. ESSP Objective 5. Support the MOE in upgrading science and mathematics curriculum standards, student assessment and teacher preparation for the mainstream. H. Technical Considerations: The ESSP project supports the creation of a locally appropriate educational environment which raises the quality of teaching and learning in science and mathematics, leading to whole school and community improvement. It emulates the development of high schools worldwide that provide specialized science and mathematics education supported by innovative uses of technology, while considering local needs, opportunities and limitations in terms of resources, capacities and bureaucracy. As part of public education, the STEM schools serve as models to effectively guide systemic reform efforts and long-term thinking regarding the two disciplines of sciences and mathematics. The schools are using technology as a primary resource to inspire discovery, and to foster a culture of innovation based on the needs of the society. Modeling and linkages between Egyptian and American STEM schools will be formulated to support students and teacher learning. The program will assist the MOE building public support for making improvement in science and mathematics performance among students as a national priority. I. Gender Considerations: As USAID requires gender to be considered in all its programs, the female students’ school aims to increase the interest and participation of its students, especially those economically marginalized, in science and math. Thus, they can contribute to workforce development by focusing on specific science and mathematics fields, allowing a diverse group of gifted female students to develop into higher education and preparing them to be efficient workers in their society and help in the economic growth of Egypt. 38 In accordance with the ESSP’s first objective, the project supports the establishment of STEM schools to serve as centers of excellence and allow enrollment to a range of gifted students regardless of their gender, social, or economic background. Jointly with the MOE, the ESSP project ensures that the student admission system is transparent and fair, and allows for the inclusion of boys and girls without any type of exclusion. Similarly, the teacher selection is transparent based on competencies without any gender considerations. Currently, the project supports one school for girls and another for boys, and ensures they have equal learning opportunities through their schools as well as participation in activities and national and international competitions. J. ESSP (STEM) Development Hypothesis: The ESSP main purpose is: (1) to introduce an Egyptian model of STEM secondary schools as an integral part of the education system; (2) to serve as incubators for developing the skills and knowledge of gifted students in science, mathematics and innovative use of technology; and (3) help students progress into higher education according to their individual abilities and aptitude. The ESSP development hypothesis suggests that if targeted technical assistance and capacity building is provided to the Ministry of Education to establish a rigorous academic foundation for introducing and sustaining a successful and functional STEM prototype, the potential to stimulate innovation and quality improvement across the education system will likely be accelerated. K. Critical assumptions: Ministry of Education (MOE), Professional Academy of Teachers (PAT) and National Center for Examinations and Educational Evaluations (NCEEE) will devote time and resources to institutionalize STEM model. II. Evaluation Rationale USAID/Egypt Mission intends to conduct a final performance evaluation of the ESSP Project in order to identify its strengths and weaknesses. In addition, evaluation recommendations and lessons learned will be used to inform the design of future Basic Education programs. A. Purpose This is a final performance evaluation and its main purposes are: ● Review, analyze and evaluate the USAID-funded STEM activities, implemented by the ESSP project, and the degree to which they have achieved program objectives and completing deliverables; and ● Provide program recommendations and lessons learned to ensure that USAID-supported STEM activities have the highest potential to achieve their intended results in a sustainable manner. B. Audience and Intended Uses The audience for the evaluation will be the Ministry of Education, USAID/Egypt Mission, specifically the Education Team, and USAID/Washington. The evaluation results will be shared with the Ministry of Education and other stakeholders, such as donors, universities and relevant civil society organizations, in a workshop setting. The report should be made accessible to the public via USAID’s Development 39 Experience Clearinghouse (DEC), within three months of report completion. III. Evaluation Questions 1. To what extent has the ECASE Program achieved the terms and conditions (including the planned objectives) of its cooperative agreement since its start? 2. What are the most significant factors that support or constrain the implementing partners’ ability to achieve the project desired outcomes? 3. What factors have facilitated and/or impeded students’ improved capabilities in the ECASE education system? 4. What actions and policies are needed to be taken by USAID and MOE to expand ECASE education to additional governorates? 5. To what extent has the ECASE succeeded to achieve gender balance? IV. Evaluation Design and Methodology A. Evaluation Design This performance evaluation is planned to evaluate the effectiveness of the ESSP project activities. It will evaluate: how the award has been implemented; what has been achieved; whether expected results have occurred according to the award’s design planned activities and intended results; and how activities are perceived, valued, and sustained by various stakeholders. The evaluation should provide specific recommendations, based on the findings and lessons learned, and suggest actions for USAID and MOE to be taken into consideration for future basic education programs. B. Data Collection Methods The Evaluation Team should consider a range of possible quantitative and qualitative methods and approaches for collecting and analyzing the information which is required to assess the evaluation objectives. The evaluation team shall share evaluation plan, methodology and data collection tools with USAID for review, feedback and/or discussion with sufficient time for USAID’s review before they are applied in the field. The data collection methodology will include a mix of methods appropriate to answering the evaluation questions. Illustrative methods include a combination of document review, in-depth interviews with key informants, and focus group discussions. As part of the evaluation methodology, the evaluation team is expected to conduct field visits to the nine STEM schools. The evaluation team should propose an evaluation approach that incorporates in-school analysis, as it relates to student test scores, pre-tests, and other knowledge measures that would help tracking individual student achievement over time as well as disaggregated achievement levels (e.g. relative achievement level of boys vs. girls over the same period of time within particular grade levels). The evaluation team should also incorporate analysis of gender differential outcomes and examine the effect of STEM schools on girls’ skills and capabilities in STEM education. The team can also conduct focus 40 group discussions and use observation tools (e.g. the STEM Observation Tool2) with teachers who are working in the STEM schools to assess their perceptions and the degree to which they master STEM teaching methodologies. The evaluation team will be expected to submit an evaluation design matrix that clearly links evaluation questions and methods as part of the evaluation work plan. Interviews and Site Visits: The Evaluation Team will conduct in-depth interviews and focus group discussions, at a minimum, with the following organizations/staff: ● MOE officials and staff. ● STEM Teachers ● STEM students (both male students and female students) in all grades. ● STEM graduates. ● STEM School principals ● STEM School Boards 

 ● Members of the MOE STEM Unit ● Project staff (World Learning and all three partner organizations: TFI, TIES and 21PSTEM) ● USAID Project Manager (AOR) and other relevant staff in the Mission. The USAID team member will be approving the work plan submitted by the evaluators. The USAID team member should provide the list of stakeholders to the evaluation team but will not be the individual setting up the meetings in order to preserve the independence of the evaluation findings. The evaluation team will be responsible for making all logistical arrangements throughout the evaluation period. The methodology should take into consideration the old and the new schools. C. Relevant Documentation The evaluation team should consult a broad range of background sources including project documents and other relevant materials. USAID and the ESSP project will provide the evaluation team with a package of background materials prior to the team’s arrival in Egypt: ● Program design documents. ● Annual reports. ● Quarterly reports. ● Performance Management Plan updated report. ● Cooperative Agreement and modifications. ● List of beneficiaries and contact information for key informants. The international evaluation team will start working on the paper review of all, but not limited to, documents cited above prior to arrival to Egypt. The local members of the evaluation team should complete the paper review prior the international team member’s arrival. 2 The STEM Observation Tool is a teacher’s observation tool being developed jointly by the ESSP project and the MOE STEM Unit, and will be shared with the Evaluation Team upon signing the award. 41 D. Data Limitation, Quality and Analysis: The evaluators may face some data limitation which will need to be discussed with the USAID education office. For example, there is no baseline data for the project. However, the evaluators may use the student test scores for the tests conducted at the beginning of the first year after admission. The evaluation team shall ensure that the data they collect clearly and adequately represents answers to the evaluation questions, and is sufficiently precise to present a fair picture of performance at an appropriate level of detail. Prior to the start of data collection, the evaluation team will develop and present, for USAID/Egypt review and approval, a data analysis plan. The data analysis plan should be submitted as part of the evaluation work plan. The methodology will be discussed with and approved by USAID/Egypt Contracting Officer’s Representative (AOR) and the Evaluation Manager prior to implementation. The evaluation team will explain: how focus group interviews will be transcribed and analyzed and how qualitative data from the focus group discussions and in-depth interviews with the key informants and other stakeholders will be integrated with quantitative data from the different relevant documents to reach conclusions about the effectiveness of the STEM program. Evaluation findings and conclusions should be clearly supported by specific and clear evidence from the data. In documenting the evaluation findings, the evaluation team should be clear about the unit of analysis (e.g. MOE staff, teachers, students) as well as the sample number on which specific conclusions are based. V. Team Composition and Roles: The team composition will include the following as a minimum and the contractor can propose an alternative team composition. The contractor must provide a statement of qualifications, resume and three references for each proposed candidate. Team Leader. A senior international consultant with 10 years of experience in conducting evaluations, particularly for pre-university education activities. S/he should also have a minimum of 5 years in leading evaluation teams, interpersonal relations and preparing high quality documents. S/he should have extensive experience in conducting qualitative and quantitative evaluations. Excellent oral and written English Language skills are required. STEM experience will be an asset. Team Members (2). Two mid-level local consultants with a minimum of 5 years’ experience in monitoring and evaluating or designing education projects, with strong English writing skills, excellent understanding of the Egyptian public education system, as well as USAID programs. One of these consultants should have an experience in conducting evaluations to programs with gender focus. Each of the team members should also have a proven experience in conducting evaluations and drafting high quality reports. Each member will participate in different evaluation activities and may be assigned specific tasks by the Team Leader as appropriate. Logistics Coordinator. A full-time logistics coordinator with three (3) to five (5) years’ experience and the required qualifications include: ● The ability to be resourceful and to successfully execute complex logistical coordination; ability to multi-task, work well in stressful environment and perform tasks independently with minimal supervision. ● The capacity for effective time management and flexibility. ● The ability to interact effectively with a broad range of internal and external partners, including international organizations and host country government officials. 42 ● Fluency in both English and Arabic. ● The ability to communicate clearly in Arabic and English, concisely and effectively both orally and in writing. Team member responsibilities include: Team Leader ● Finalize and negotiate with USAID/Egypt the evaluation work plan; ● Establish evaluation team roles, responsibilities, and tasks; ● Facilitate the team planning meeting (TPM) ● Ensure that the field meetings are complete. ● Manage team coordination meetings in-country and ensure that team members are working to schedule; ● Coordinate the process of assembling individual input/findings for the evaluation report and finalizing the evaluation report; ● Lead the preparation and presentation of key evaluation findings and recommendations to USAID/Egypt team prior to departing Egypt. Team Members ● Designing the evaluation plan. ● Developing a data collection plan. ● Conducting field visits, surveys, and interviews. ● Collecting the data. ● Recording and summarizing the data. ● Analyzing the data collected. ● Preparing reports and presentations for discussing the findings. Logistics Coordinator ● Handle travel related logistics and provide administrative support to the evaluation team members. ● Be responsible for setting up meetings with USAID and stakeholders. ● Arranging for the translation from English to Arabic and vice versa during interviews, meeting venues and focus group discussions with MOE and other stakeholders. VI. Evaluation Products A. Deliverables Team Planning Meeting: A team planning meeting will be held in Egypt to ● Review and finalize final evaluation questions ● Review and finalize the assignment timeline and share with USAID/Egypt ● Present and discuss data collection methods, instruments, tools and guidelines ● Review and clarify any logistical and administrative procedures for the assignment. Work Plan: In the planning meeting, the team will discuss a detailed work plan which should be prepared by the 43 team prior to the meeting. The work plan should include the methodologies to be used in the evaluation, timeline and detailed Gantt chart. The work plan will be submitted to the Evaluation Manager at USAID/Egypt for review. A detailed methodology and data analysis plan (evaluation design, data analysis steps and details, operational work plan) will be discussed with USAID during the planning meeting. Upon arrival, USAID will provide the Evaluation Team with stakeholders’ overview that includes an initial list of interviewees, from which the Evaluation Team can work to create a more comprehensive list. Prior to data collection, the Evaluation Team will provide USAID with a list of interviewees and a schedule for conducting the interviews. The evaluation team should submit the Work Plan and evaluation methodology to the Evaluation Manager for approval. Prior to starting field work, the evaluation team will share the data collection tools with the USAID Evaluation Manager for review and approval. Debriefings: During the evaluation period, the Evaluation Team will conduct a mid-term debrief to USAID to provide a comprehensive progress report, including challenges and any unpredicted changes in scheduling and/or implementation plan. At the end, the team will need to do another debrief to present the major findings and conclusions of the evaluation to USAID/Egypt. The debriefing will include a discussion of achievements and issues as well as recommendations for the future activity design and implementation. The team will address USAID/Egypt comments and revise the draft report according to the draft report peer review. The team will then present their major findings/conclusions to the Ministry of Education and address the MOE’s comments and present a revised draft. Draft evaluation report: A draft report of the findings and recommendations should be submitted to the USAID Evaluation Program Manager prior to the Team’s departure from Egypt. USAID will provide written comments on the draft report within 10 working days of receiving the document. Final Report: The Evaluation Team will submit a final report that incorporates responses to Mission comments and suggestions. This report should not exceed 30 pages in length (not including appendices, lists of contacts, etc.). The format will include an executive summary, table of contents, glossary, methodology, findings, recommendations and conclusions. The report will be submitted in English, electronically, and will be disseminated within USAID and to project stakeholders according to the dissemination plan developed by USAID. The evaluation report will be deemed final only with USAID/Egypt approval. The Executive Summary of the Report will be translated into Arabic and submitted within 30 days after the final report is submitted. The evaluation report must be submitted to the Development Experience Clearinghouse (DEC) at http://dec.usaid.gov within three months of completion, with an infographic if applicable. Data Sets: At the time of submission of the final report, the survey instruments, interviews and data sets should be submitted on a flash drive to the evaluation program manager. All data on the flash drive will be in an unlocked, editable format. Expanded Executive Summary: The team will submit an expanded executive summary to accompany the final report that will include a background summary on the evaluation purpose and methodology, and an overview of the main data 44 points, findings, and conclusions. The expanded executive summary should be easy to read for wide distribution to local audiences. The expanded executive summary will be submitted in English and Arabic, in hard copy (3 English and 3 Arabic) and electronically. The report will be disseminated within USAID and to stakeholders according to the dissemination plan. Reporting Guidelines: ● The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not, and why. ● The evaluation report shall address all evaluation questions included in the scope of work. ● The report should include the evaluation Scope of Work as an annex. All modifications, whether in technical requirements, evaluation questions, evaluation team composition, methodology, budget, or timeline need to be agreed upon in writing by the AOR. ● The Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report. ● The Evaluation findings will assess outcomes of the STEM on the students disaggregated by gender. ● Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.) and what is being done to mitigate the threats to validity. ● Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. ● Sources of information need to be properly identified and listed in an annex. ● Recommendations need to be supported by a specific set of findings. ● Recommendations should be action-oriented – organized according to whether recommendations are short-term or long-term, practical, and specific, with defined responsibility for the action. The final report will be reviewed using the Checklist for Assessing USAID Evaluation Reports (http://www.usaid.gov/policy/evalweb/evaluation_resources.html). B. Evaluation Report Requirements The format for the evaluation report is as follows: Executive Summary. Concisely state the evaluation methodology, key findings and recommendation in no more than 3-5 pages; Table of Contents. Organized presentation of all materials; Introduction. State purpose, audience, and summary of task; Background. Provide brief overview of the STEM program in Egypt, USAID program strategy and activities implemented in response to the problem, brief description of ESP projects/components, purpose of the evaluation; Methodology. Describe evaluation methods, including threats to validity, constraints and gaps; Findings/Conclusions. Based on the evaluation questions include data quality and reporting system that should present verification of spot checks, issues, and outcomes; Challenges. Provide a list of key technical and/or administrative, if any; References. Include bibliographical documentation, meetings, interviews and focus group discussions; 45 Annexes. Provide annexes that document the evaluation methods, schedules, evaluation scope of work, interview lists and tables— should be succinct, pertinent and readable. C. Timeline Work is to be carried out over a period of 12 weeks, beginning on March 15th, 2017 and final report and close out concluding by June 15th, 2017. It will be 6-day work week. The estimated deliverables timeline is as follows: ● Desk Review and training of data collection team, preparation for data collection tools (First week) ● Team planning meeting with USAID, including submission of the draft schedule of data collection interviews, draft methodology and data analysis plan (Second week) ● Submission of final methodology and data analysis plan (Five days after the team planning meeting) ● Submission of a final schedule of data collection interviews (Five days after the team planning meeting) ● Conduct data gathering, compilation and analysis (Three weeks from the team planning meeting) ● Draft Report in English (including an Executive Summary) (One week after the data gathering) ● Debriefing session with USAID (One week after the draft report) ● Debriefing session with the Ministry of Education (One week after the debriefing) ● USAID will provide written comments to the team as the per the draft report peer review outcome (one week after the stakeholders debriefing) (USAID reviews within 10 workings days from draft report submission) ● Final Report in English (One week after USAID comments) ● Executive summary translated into Arabic (One week after the final report) D. Estimated LOE Task/Deliverable Team Leader Data Collection (2 local evaluation specialists) Logistical support/ Administration Travel to Egypt 1 Review background documents, draft work plan, methodology and data collection tools 5 days 5 day x 2 1 day Team Planning meeting, meeting with USAID/Egypt, team training and finalizing meeting schedule. 5 days 5 day x 2 1 day Data collection. Includes interviews with key stakeholders (stakeholders and USAID staff) and site visits as well as coordinating and monitoring efforts 20 days 20 days x 2 20 days Discussion, analysis, findings, trends and recommendations 10 days 10 days x 2 10 days 46 Debrief preparation and meeting with USAID 2 days 12 days x 2 1 day Debrief preparation and meeting with stakeholders 2 days 2 days x 2 1 day Preparation of draft report 15 days 15 day x 2 1 day Depart Egypt/travel 1 Completion of final report after receiving feedback from USAID on draft report 7 7 days x 2 Total estimated LOE 68 days 67 days x 2 35 days 47 ANNEX II: EVALUATION METHODS AND LIMITATIONS The evaluation used a mixed methods approach, tapping three types of data — project documentation, quantitative, and qualitative. The strength of the methodology is that is draws upon multiple sources of information for each question, to build a multi-layered case for the validity of the findings. The implementing partners have used ongoing assessments of various types (e.g., pre- and post-training tests, classroom observation, monitoring of the capstone processes) to track the strengths and weaknesses of the innovative process. This ESSP final performance evaluation basically paused this process at one point in time, and took a snapshot of achievements to date and remaining challenges. Importantly, the snapshot involved a comprehensive examination of the subjective assessment of the STEM experience by stakeholders, participants, and beneficiaries. Project Documentation The USAID Activity Manager for the ESSP provided copies of the cooperative agreement and a wide range of quarterly, annual, and Performance Monitoring Plan reports for the five years of the program. These documents were used by the EGYPT SIMPLE team to get an overview of the entire process as well as factual information about actions and decisions, so as to design the evaluation. Quantitative Data Sources The evaluation team used three main sources for the quantitative data: 1. Three online surveys using Survey Monkey. These were designed and administered by the evaluation team to the following target populations: ● Currently enrolled students at the nine STEM schools included in the evaluation; ● Teachers currently working at the nine STEM schools; ● Graduates of the STEM School for Girls in Maadi and STEM School for Boys in 6th of October.3 2. Secondary data from two main sources, the World Learning project database and the administrators of the nine schools included in the evaluation. 3. Classroom observation data using a Likert scale to measure the extent to which teachers were utilizing various aspects of the STEM inquiry-based education model in their classrooms. The forms and rubric were provided by the ESSP, and are used in ongoing teacher assessment and coaching. The evaluation team worked with trained observers to conduct six observations in each of the nine schools, for a total of 54. The paper-and-pencil forms were coded and entered into an Excel database for analysis. I. Online Surveys Sample Selection The three surveys targeted the entire population of currently enrolled students, teachers, and graduated students. No sampling was used. They were conducted via Survey Monkey, which is an online survey development cloud-based software application. The evaluation team developed and tested the surveys before posting them online on March 29, 2017. 3 Only the STEM School for Girls in Maadi and STEM School for Boys in 6th of October have had graduates and previous grade 12 students. 48 After discussions with the school principals, the links for the three surveys were e-mailed to the school IT managers consecutively, according to the school visits. The IT manager in each school e-mailed the links using the school’s e-mail lists.4 A week after the survey’s launch, a reminder was sent to the principals asking them to encourage students and faculty to respond. Each questionnaire required approximately 10 minutes to complete. Response rates by school are shown in Table 1. Table 1: Response Rates by School, Enrolled Students’ Survey School Number of Enrolled Students Number of Responses Response Rate % Alexandria 269 154 57.2 Assuit 150 83 55.3 Dakahlia 257 175 68.1 Ismailia 285 223 78.2 Kafr Elsheikh 202 129 63.9 Luxor 103 87 84.5 Red Sea 183 149 81.4 Maadi 299 265 88.6 October 404 167 41.1 Grand Total 2,152 1,432 66.5 The STEM School for Girls in Maadi and STEM School for Boys in 6th of October e-mailed the schools’ graduates using their e-mail list. In addition, during the graduates’ group discussions, facilitators encouraged the attendees to fill out the questionnaires and to e-mail the link to other graduates they knew. This led to a doubling of the responses. The currently enrolled student survey and the teacher survey were both taken offline in the early morning of April 20, 2017. Due to the low response, the graduate survey remained open online until April 23, 2017. The three surveys were conducted in English. Table 2 shows the number of responses received for each survey, the number after data cleaning, and the response rate of each survey. Table 2: Survey Response Rates and Numbers, before and after Cleaning Survey Number Received Number after Cleaning Size of Target Population Response Rate % Enrolled Students 1,439 1,432 2,152 66.5 Graduated Students 106 104 464 22.4 Teachers 267 207 243 85.2 Limitations Encountered An important limitation was the weak Internet connection in the schools.5 The weakness and instability of the connection are believed to have been the main cause of replicated responses from the same individual (i.e., when a person was cut off, s/he started over again). As a result, in the case of the 4 All students, teachers, and graduates are issued STEM e-mail addresses when they enter the school. E-mail is the primary means of communication among students and staff. 5 Complaints about the weakness and instability of the Internet connection were addressed in the group discussions. Some 47.9% of students who responded to the survey reported they were least satisfied with the Internet connection. 49 teachers, the number of responses was higher than the population size, as shown in Table 3. Table 3: Response Rates, by School, Teachers' Online Survey School Number of Teachers Number of Response Response Rate % Alexandria 26 23 88.5 Assuit 24 15 62.5 Dakahlia 27 22 81.5 Ismailia 27 30 111.1 Kafr Elsheikh 24 21 87.5 Luxor 19 21 110.5 Red Sea 26 26 100.0 Maadi 35 39 111.4 October 35 10 28.6 Grand Total 243 207 85.2 It is also believed that the Internet connection was a cause of the second limitation. This was the increased number of missing values in the students’ survey as the survey progressed, where any apparent disconnection prevented the completion of the survey. The percentage of the missing values began around 22.1% in Question 10 and increased until it reached 43.4% of missing values in the last question. The overall response rate varied among schools. The lowest was the STEM School for Boys in 6th of October (a 41.1% response rate), whereas the highest was the STEM School for Girls in Maadi, with an 88.6% response rate. A third limitation was that the teacher survey was sent to all employees in some schools. As a result, in those schools, responses were received from other staff in addition to teachers. Those responses were excluded from the data. The percentage of missing values in the teacher survey was, at most, 30.4%. The final limitation was the outdated e-mail list for graduates. That list was based on the school e-mails, not on a new list, which resulted in a response rate of 22.4%. This rate increased by only 10% after in￾person encouragement to some graduates in the group discussions. Table 4 shows that the response rate was even lower among graduates from earlier years. The low response rate increases the potential for non-response bias. The percentage of the missing values in the graduated students’ survey did not exceed 11.5%. Table 4: Response Rates by Graduation Year, Graduates’ Survey School Graduation Year Number of Students Graduated Number of Responses Response rate (%) 2014 85 12 14.1 2015 183 32 17.5 2016 196 59 30.1 Data Cleaning Data sets were imported from Survey Monkey and cleaned. Records filled before the launch date were excluded, as they appeared to be evaluation team trials that, mistakenly, were not removed. 50 The teachers’ data were corrected by eliminating the non-teacher responses. A comparison was also made between the number of teachers by school, subject, and gender in the online survey data and the secondary data received from World Learning. Several duplicated records with missing values were deleted, following an exercise in which all the responses to all the questions were compared one by one. This decreased the total number of responses further. The final number of observations was 207. Data Coding Due to the nature of Survey Monkey, only a few questions required coding. Only one question was coded in the student survey, which was the option “other” in Question 8, “How did you learn about the STEM schools?” There was a decrease in the number of “other” responses from 41 to 11, after going through them one by one and coding them to one of the existing options. Two open questions in the teachers’ survey required coding. Those were the field of study (Question 5), and the subjects taught (Question 6). The coding was done in consultation with the education specialist from the evaluation team. The two questions were key to identifying and excluding the non￾STEM teachers who answered the survey. II. Secondary Data World Learning ESSP Database World Learning provided the evaluation team with all the requested data in Excel sheet format. The files consisted of: ● Number of students enrolled in STEM schools from 2011 to the present, disaggregated by school, gender, and grade. ● Number of teachers and staff currently working at STEM schools, disaggregated by school, gender, and subject taught or job position. ● Average hours of training teachers received since joining the STEM project, disaggregated by teaching groups (STEM, NON-STEM, English, activities, and principals). The same file provided the number of teachers who received training and stayed at the school, as well as the numbers who were trained, but left. ● Finally, World Learning furnished data on the number of students who participated in local and international competitions between the academic years 2013/2014 and 2016/2017, as well as the awards they received. Also included was the number of scholarships the students received, from either Misr El Kheir or STEP. Schools’ Administrations Upon request, school principals provided the evaluation team with their students’ GPAs (scores) for some of the academic years. Some schools also supplied data about transferred students. Table 5 shows the GPA data, by academic year and grade. The data were disaggregated by gender and subject. Gaps and Limitations in the Secondary Data The only limitation in the secondary data was the inconsistency of the information and format of the scores received from schools. The team converted the scores into a consistent format. Due to the unbalanced nature of the scores from the various academic years, only the scores of the first term of 2016/2017 were used, since they were the only ones available for all the schools. 51 Table 5: Students’ Scores from Schools, by Academic Year and Grade School Grade 10 Grade 11 Year 14/15 Year 15/16 Year 16/17 Year 14/15 Year 15/16 Year 16/17 Term 1 Term 2 Term 1 Term 2 Term 1 Term 1 Term 2 Term 1 Term 2 Term 1 Alexandria 1 1 Assuit 1 1 Dakahlia 1 1 1 1 Ismailia 1 1 1 Kafr El Sheikh 1 1 1 Luxor 1 1 Maadi 1 1 1 1 1 1 1 1 1 1 6th of October 1 1 1 1 1 1 1 1 Red Sea 1 1 * Shaded areas indicate that schools (grade) were not open yet. Software Used (across Data Sources) Data files from Survey Monkey were imported in Excel format, and transferred into SPSS 19 and STATA 13 statistical software packages. All secondary data files were in Excel sheets, and only the files containing the students’ scores were transferred to STATA 13, after combining data from the different schools. Data Analysis (across Data Sources) The team used descriptive analysis throughout the report to develop frequency tables and cross tabulations in order to compare distributions across gender, schools, and grades. In addition, a t-test was used to test for significant differences in the mean scores by gender and school. Qualitative Data Sources The bulk of the qualitative data was collected through interviews and group discussions in the nine schools included in the evaluation. Table 6 presents key descriptive information about each school. Table 6: Key Characteristics of STEM Schools included in the ESSP Evaluation Name/Location Year Opened Number of Students (2016-2017) Number of Teachers (2016- 2017) Grade 10 Grade 11 Grade 12 STEM Non￾STEM M F M F M F Maadi 2011 118 117 64 16 19 6th of October 2012 141 147 118 17 18 Ismailia 2015 72 66 74 73 ----- ------ 11 16 Assiut 2015 65 46 25 14 ----- ------ 10 14 Luxor 2015 45 23 20 15 ----- ------ 7 12 52 Red Sea/Hurgada 2015 69 43 49 22 ---- ---- 10 16 Alexandria 2015 72 63 72 62 ---- ---- 11 15 Kafr El Sheikh 2015 37 40 61 64 ---- ---- 10 14 Dakahleya 2015 67 66 61 63 ---- ---- 11 16 The evaluation team spent three days in each of the nine schools. Three sub-team of two evaluators and a classroom observer visited three schools each over a three-week period. In each school, the team held three discussion groups with teachers (STEM teachers, non-STEM teachers, and Capstone Leaders and facilitators), for a total of 27 groups, and an average of about 16 teachers per school (n=146), (with overlap because capstone leaders also are teachers), as well as four discussion groups with students (separate groups for male and female students in grades 10 and 11). Only three student discussion groups were convened in the Maadi and 6th of October schools (one each for grades 10, 11, and 12). In total, 34 group discussions were held with an average of about nine students each (n=312), including 152 males and 160 females. Other discussion groups in each school included the members of the Board of Trustees (BOTs) (n=21 members across schools), social workers/psychologists (n=30 across schools), and members of the local STEM units (n=28 across schools). Key informant interviews were completed with the school principal (n=9), deputy principal/academic coach (n=6, 3 schools do not have deputies), the FabLab manager (n=9), the librarian (n=9), and the IT specialist/manager (n=9). An MOE-trained classroom observer worked with the two evaluators, conducting six observations of 80-minute sessions (four STEM classes, one of which was a laboratory class, and two non-STEM classes, including English to the extent possible, with a mix of experienced and new teachers across grades). The classroom observation tool (i.e., the Classroom Observation Scale or COS), which was developed by the ESSP and is used by supervisors in coaching STEM school teachers, appears in Annex III. An illustrative data collection schedule for a school is included in Annex IV. The evaluators were well received in all of the schools. As requested by the Central STEM Unit, the administrators scheduled the meetings for the evaluators and notified the participants. The participants seemed eager to relate their experiences and points of view, and freely responded to the team’s questions. The second major source of qualitative information was a series of key informant interviews in Cairo with World Learning staff; representatives of the sub-grantees (TIES, TFI, and 21PSTEM) and Management Systems International (MSI); members of the Central STEM Unit and others from the collaborating agencies (PAT, NCEEE, NCERD, and CCIMD); MOE employees, including the Central STEM Unit manager, representatives of the Offices of the Counselors for Science, Mathematics, and English, and the Director of the Technology Development Center; and USAID. These interviews were conducted by one sub-team of two evaluators (including the team leader) during the week when school visits were not possible because of midterm exams. A total of 30 individuals were included in 20 key informant interview sessions. The protocols for these interviews are in Annex III. The qualitative data were coded and analyzed using Excel tally sheets. The contacts included approximately 663 consultations, through 85 group discussions, 62 key informant interviews, and 54 classroom observation sessions. 53 ANNEX III: DATA COLLECTION INSTRUMENTS Instruments Page Key Informant Interview Protocols Key Informant Interview Protocol for USAID Egypt STEM Schools Project Related Staff 54 Interview Protocol for ESSP Sub-Grantees 56 Key Informant Interview Protocol for the MOE Counselors in Math and Science 58 Key Informant Interview and/or Group Discussion Protocol for the Ministry of Education STEM Unit 60 Key Informant Interview Protocol for Center for Curriculum and Instructional Materials Development (CCIMD) 62 Key Informant Interview Protocol for the National Center for Examinations and Educational Evaluation (NCEEE) 65 Key Informant Interview Protocol for National Center for Educational Research and Development (NCERD) 67 Interview Protocol for the District STEM Units (Key Informant Interview or Group Discussion) 69 Key Informant Interview Protocol for STEM School Administrators (Principals and Assistant Principals) 70 Key Informant Interview Protocol for STEM Implementing Partners (World Learning) 73 Key Informant Interview Protocol for the Professional Academy for Teachers (PAT) 76 Key Informant Interview Protocol for the Director of MOE Technology Development Center (TDC) 78 Key Informant Interview Protocol for MSI 80 Key Informant Interview Protocol for the MOE English Counselor 82 Group Discussion Protocols Group Discussion Protocol for the STEM School Board of Trustees 84 Group Discussion Protocol for Currently Enrolled STEM School Students 86 Group Discussion Protocol for STEM School Graduated Students 88 Group Discussion/Key Informant Interview Protocol for STEM School Support Staff (Capstone Director, Social Worker, Librarian, Lab Technician) 90 Group Discussion Protocol for STEM School Teachers 92 Surveys Online Survey for STEM School Graduated Students 94 Survey for Currently Enrolled STEM Students 97 Teacher Survey 101 Observation Tool Egypt STEM Schools STEM Teacher Data Collection Form 106 54 KEY INFORMANT INTERVIEW PROTOCOL FOR USAID EGYPT STEM SCHOOLS PROJECT RELATED STAFF EVALUATION OF THE EGYPT STEM SCHOOLS PROJECT 1. Conditions in Egypt have changed between the initial discussion of STEM education as a component of basic education programming, the funding of the cooperative agreement, and the present. - What, if any, changes have there been in your interpretation of the goal of the program, for example, in emphasis, priorities? If there have been changes, have these affected the services and support provided? 2. USAID has expressed a special interest in networking STEM model schools, promoting advanced instruction in science and mathematics among high-achieving students, and mainstreaming advanced instruction in science and mathematics in traditional schools? - What, if any, affect has the STEM project had on the broader USAID basic education programming? 3. The goal and objectives of the ESSP project cite the importance of equal opportunity in terms of socio-economic status and gender. To what extent does USAID consider that this objective has been achieved? 4. Particularly in terms of economic status, has there been any monitoring of the balance in the student population, or of ways in which economic status (or region or rural/urban residence) may have affected the application, acceptance, and achievement processes? - On all levels (USAID, STEM schools, Modareyas, MOE), were actions taken to achieve this objective? Why or why not? 5. One of the main components of USAID support to the STEM school system is professional capacity building for teachers, principals, and the STEM Unit. - What was the basis and/or process for determining the type and content of training provided? For example, were needs assessments conducted or was it an existing training package? - What tools have been used to measure the effectiveness and impact of this training? 6. A number of tools or activities were mentioned in the Cooperative Agreement that seem not to have been carried through in the project; e.g., Youth Truth Survey, PARLO, AdvoKit, Assessment Action Team, et.al. - To what extent were the following items carried forward? - What types of circumstances or barriers affected their use? 7. Sustainability of the USAID project inputs and of the existing future STEM schools is an 55 important question for USAID and for the MOE, especially because of the expense involved in setting up and maintaining a STEM school. Public/private partnerships were seen as one avenue to get support for the schools and to open up job opportunities in STEM fields in Egypt rather than abroad. To date, these do not appear to have materialized as a significant source of support or labor market development. - In this context, how do you assess the prospects for long-term sustainability of the STEM model? 56 INTERVIEW PROTOCOL FOR ESSP SUB-GRANTEES (The Franklin Institute, Teaching Institute for Excellence in STEM, and 21st Century Partnership for STEM Education) EVALUATION OF THE EGYPT STEM SCHOOLS PROJECT Good morning/afternoon. My name is ______________. I am a member of an evaluation team endorsed by the Ministry of Education to conduct an evaluation of the Egypt STEM Schools Project (ESSP). The evaluation team will review, analyze and evaluate the USAID-funded STEM Schools Project activities carried out by World Learning and the degree to which they have achieved their objectives. The evaluation team has been endorsed by the Ministry of Education to review, analyze and evaluate the USAID￾funded STEM school activities. The USAID Office of Education and Health is funding three three-person field teams specialized in evaluation to conduct interviews, group discussions, classroom observations, and surveys in the STEM schools. The evaluation team also will interview officials in the MOE STEM Unit and members of the World Learning project team. The findings, conclusions and recommendations of this evaluation are intended to provide program recommendations and lessons learned to ensure that STEM activities in Egypt have the highest potential to achieve their intended results in a sustainable manner. Findings of this evaluation will be managed in a confidential and anonymous manner. Your participation is entirely voluntary but your participation is considered vital to this important initiative. I thank you in advance for your valued contribution and I would be pleased to answer any questions that you may have in the course of our discussion. Note: Each organization will be interviewed separately through a telephone call to the home office in the US. The topics to be discussed are the same in each case. A list of the ESSP activities of each organization, based on the ESSP project documents, is attached to tailor the questions to each specific role. 1. What were the primary activities of [insert name of organization interviewed] in the Egypt Stem Schools Project? - Was this your first experience working with STEM in Egypt? (other comparable experience?) - To what extent did you implement these activities in Egypt and/or virtually? - Did you have face-to-face contact with the MOE, school teachers and administrators, or students? - How many [insert name of organization interviewed] staff were involved in this project, in the US and in Egypt? 2. In your opinion, what has been your primary achievement or contribution in this project? - To what extent did you coordinate your work with World Learning and the other partner organizations? - Was your primary achievement the result of a joint effort or a result of your independent work? - How do you assess the effect of the coordination on your capacity to carry out your role in the project? 3. In your opinion, what were the primary shortcomings and unfinished tasks in terms of your 57 [specify activities] in the project? - What obstacles did you encounter and how did you respond to them? - Did you face any unexpected constraints that affected your ability to complete your work? 4. How do you assess the capacity and resources of the MOE to maintain and build on your contributions [specify activities, achievements] after the conclusion of the ESSP, especially in terms of the planned expansion of the STEM school model across Egypt? 5. What recommendations do you make for next steps for the MOE and/or USAID in terms of [specify activities, achievements]? 6. What are the next steps for Egypt in terms of international experience and standards in STEM education? 58 KEY INFORMANT INTERVIEW PROTOCOL FOR THE MOE COUNSELORS IN MATH AND SCIENCE EVALUATION OF THE EDUCATION CONSORTIUM FOR THE ADVANCEMENT OF STEM IN EGYPT Good morning/afternoon. My name is ______________. I am a member of an evaluation team endorsed by the Ministry of Education to conduct an evaluation of the Egypt STEM Schools Project (ESSP). The evaluation team will review, analyze and evaluate the USAID-funded STEM Schools Project activities carried out by World Learning and the degree to which they have achieved their objectives. The evaluation team has been endorsed by the Ministry of Education to review, analyze and evaluate the USAID￾funded STEM school activities. The USAID Office of Education and Health is funding three three-person field teams specialized in evaluation to conduct interviews, group discussions, classroom observations, and surveys in the STEM schools. The evaluation team also will interview officials in the MOE STEM Unit and members of the World Learning project team. The findings, conclusions and recommendations of this evaluation are intended to provide program recommendations and lessons learned to ensure that STEM activities in Egypt have the highest potential to achieve their intended results in a sustainable manner. Findings of this evaluation will be managed in a confidential and anonymous manner. Your participation is entirely voluntary but your participation is considered vital to this important initiative. I thank you in advance for your valued contribution and I would be pleased to answer any questions that you may have in the course of our discussion. 1. When did you join the STEM Unit? 2. What types of training have you received from the USAID STEM project? What is your assessment of this training? - What about the quality of the training? - How well did it match your needs? Were you involved in the design of the training program? - Have you identified gaps in the training you received? - How do you assess the leadership training in particular? - Any other comments? 3. How were the math and science standards developed for the STEM schools? - Do these standards reflect the integration of math and science learning in the STEM model? - To what extent are these standards contextualized to fit Egypt? - Were the teachers involved in the development of the standards and the contextualization? - Are the students aware of these standards? - Do you think teachers are appropriately trained to incorporate these standards in their teaching methodology? 4. Do you see the STEM standards contributing to mainstream math and science education? How? 5. What do you think about the assessment criteria for the STEM students? 59 - Formative assessment - Inventory exam (who design it & how) - Distribution of weights (between exam, capstone, etc.) 6. Has there been any effort to modify assessment standards of mainstream science and math education? Are you considering this step in the future? 7. Is there a standard guide for the Capstone projects? If there is no guide, when and how will it be developed? If there is a guide: - What is your assessment of the guide? - Who manages the utilization of the guide and the application of the regulations? (grading of the projects) - Are students aware of the standards for grading their projects? - To what extent does the guide require or encourage links between the Capstone projects and needs of the local community? 60 KEY INFORMANT INTERVIEW AND/OR GROUP DISCUSSION PROTOCOL FOR THE MINISTRY OF EDUCATION STEM UNIT EVALUATION OF THE EGYPT STEM SCHOOLS PROJECT Good morning/afternoon. My name is ______________. I am a member of an evaluation team endorsed by the Ministry of Education to conduct an evaluation of the Egypt STEM Schools Project (ESSP). The evaluation team will review, analyze and evaluate the USAID-funded STEM Schools Project activities carried out by World Learning and the degree to which they have achieved their objectives. The evaluation team has been endorsed by the Ministry of Education to review, analyze and evaluate the USAID￾funded STEM school activities. The USAID Office of Education and Health is funding three three-person field teams specialized in evaluation to conduct interviews, group discussions, classroom observations, and surveys in the STEM schools. The evaluation team also will interview officials in the MOE STEM Unit and members of the World Learning project team. The findings, conclusions and recommendations of this evaluation are intended to provide program recommendations and lessons learned to ensure that STEM activities in Egypt have the highest potential to achieve their intended results in a sustainable manner. Findings of this evaluation will be managed in a confidential and anonymous manner. Your participation is entirely voluntary but your participation is considered vital to this important initiative. I thank you in advance for your valued contribution and I would be pleased to answer any questions that you may have in the course of our discussion. 1. Could we begin by talking about the history of the STEM Unit? (This may be answered in the 19 March 2017 introductory meeting) - When was it formed? - Who are the members? - How has it changed over time? - Have the offices represented in the Stem Unit changed? - Have the individual people in the STEM Unit changed? - How often does the Unit meet as a group? - What other activities and responsibilities do members of the STEM Unit have? 2. The STEM model schools are a new approach to building capacity and leadership in science, mathematics and technology in Egypt. - Where or how is the model positioned within the Ministry’s organizational structure? 3. What is your assessment of the training that the members of the STEM Unit have received through the ESSP project? - What about the quality of the training? - How well did it match your needs? - Were members of the STEM Unit involved in the design of the training program? - Have you identified gaps in the training you received? - How do you assess the leadership training in particular? - Any other comments with regard to training? 61 4. In terms of how the STEM Unit operates, what is the role of the STEM Unit in each of the following components of the STEM Schools Project? - Application, selection, and admission of students - Application, selection, and training of the teachers - How are the principals selected? - With regard to governance and autonomy, do the principals report to the STEM Unit? - Describe the working relationship of the STEM school principals with the STEM Unit - How do you assess the leadership training provided to the principals and the weekly calls to each school for assistance and coaching? 5. What, if any, gaps still exist in the application, selection, and admission processes for students? What problems are you encountering and how are you correcting them? 6. What, if any, gaps still exist in the application, selection, admission and training processes for teachers? What problems are you encountering and how are you correcting them? 7. Describe the process for curriculum development. How has the STEM Unit been involved? What are the plans for curriculum development going forward after the USAID project ends? 8. What has been the process for the development, design, and grading of the Capstone Projects? 9. How, if at all, is the STEM Unit involved in the Capstone Projects? What is the plan, if any, for STEM Unit involvement in the Capstone Projects after the USAID project ends? 10. Thinking about the future, after the USAID project, what will the role of the STEM Unit be in: - Maintenance and construction/acquisition of infrastructure? - Advancement of public/private partnerships (PPPs)? - Community engagement? - Strengthening the Boards of Trustees (BOTs)? - Upgrading the science and math curriculum in the mainstream schools? 11. How do you assess the various products that have been delivered to the STEM Unit from the USAID STEM project; e.g., Design Blueprint, New School Startup Manual, Master Checklist, Master Trainers, other? 12. In your view, what are the advantages and disadvantages of having converted the STEM Unit into a formal unit within the MOE? 13. What recommendations, if any, can you provide for improving the Central STEM Unit’s performance? 14. What recommendations, if any, can you provide for improving the Governorate STEM Unit’s performance? 62 KEY INFORMANT INTERVIEW PROTOCOL FOR CENTER FOR CURRICULUM AND INSTRUCTIONAL MATERIALS DEVELOPMENT (CCIMD) EVALUATION OF THE EGYPT STEM SCHOOLS PROJECT Good morning/afternoon. My name is ______________. I am a member of an evaluation team endorsed by the Ministry of Education to conduct an evaluation of the Egypt STEM Schools Project (ESSP). The evaluation team will review, analyze and evaluate the USAID-funded STEM Schools Project activities carried out by World Learning and the degree to which they have achieved their objectives. The evaluation team has been endorsed by the Ministry of Education to review, analyze and evaluate the USAID￾funded STEM school activities. The USAID Office of Education and Health is funding three three-person field teams specialized in evaluation to conduct interviews, group discussions, classroom observations, and surveys in the STEM schools. The evaluation team also will interview officials in the MOE STEM Unit and members of the World Learning project team. The findings, conclusions and recommendations of this evaluation are intended to provide program recommendations and lessons learned to ensure that STEM activities in Egypt have the highest potential to achieve their intended results in a sustainable manner. Findings of this evaluation will be managed in a confidential and anonymous manner. Your participation is entirely voluntary but your participation is considered vital to this important initiative. I thank you in advance for your valued contribution and I would be pleased to answer any questions that you may have in the course of our discussion. 1. What is your position within CCIMD? 2. How long have you been working with STEM education in Egypt? 3. How long have you been collaborating with the MOE STEM Unit? 4. How do you describe CCIMD’s role in the development of the STEM model schools in Egypt? 5. How would you describe CCIMD’s contributions to STEM model school development in Egypt? 6. We understand CCIMD has played, or continues to play, a role in the preparation of outreach materials to introduce the STEM education model to preparatory school students. - Please outline CCIMD’s role in the preparation of outreach materials and the role of other institutions that may be involved in the process. - Are outreach materials in their final form or do they continue to be development? - Do outreach materials require approval by the MOE STEM Unit? - If they are not in their final form, when do you expect them to be finalized? - When finalized, do you expect them to be used in preparatory schools? - What do you see as the achievements and the problems in the preparation of outreach materials? 7. We understand CCIMD has played, or continues to play, a role in the development of extracurricular activities for STEM school students. - Please outline CCIMD’s role in the development of extracurricular activities and the 63 role of other institutions that may be involved in the process. - Are extracurricular activities materials in their final form or do they continue to be development? - Do extracurricular activities require approval by the MOE STEM Unit? - If they are not in their final form, when do you expect them to be finalized? - When finalized, do you expect them to be used in preparatory schools? - What do you see as the achievements and the problems in the development of extracurricular activities? 8. We understand CCIMD has played, or continues to play, a role in the development of a University Readiness Test (URT) for STEM school students. - Could you please outline CCIMD’s role in the development of a University Readiness Test and the role of other institutions that may be involved in the process? - Why did the MOE decide to develop a new test for STEM students, compared to the existing exam used in the mainstream schools and the ACT? - Is this URT in its final form or does it continue to be development? - Does the URT require approval by the MOE STEM Unit? - If it is not in its final form, when do you expect it to be finalized? - When finalized, do you expect it to be used in preparatory schools? - What do you see as the achievements and the problems in the development of the URT? - What does the URT aim to assess/measure? 9. We understand CCIMD has been working on the adaptation of science standards to reflect best practices in STEM education to inform wider reform efforts of science education in mainstream secondary schools focused on the curriculum and learning outcomes. - Please explain the rationale behind the adaptation of science standards. - Are science standards in their final form or do they continue to be in development? - Are similar steps being taken for establishing math standards? - Are math standards in their final form or do they continue to be in development? - Do the science and/or math standards require approval by the MOE STEM Unit? - If these standards are not in their final form, when do you expect them to be finalized? - Following the establishment of science and/or math standards, is it CCIMD’s intent to modify said standards to and apply them to mainstream secondary education? - In the affirmative, when does CCIMD expect the modified science and/or math curriculum to be applied to mainstream secondary education? - What policies, measures and resources does the MOE need to have in place to successfully adapt mainstream secondary science in Egypt to reflect best practices in STEM education? 10. In your opinion, should STEM model schools be scaled-up; i.e., offered to more students nation￾wide? 11. In the affirmative, what policies, measures and resources does the MOE need to have in place to scale-up successfully? 64 12. Has CCIMD received any support, or professional development training from the ESSP project? - In the affirmative, what type of support or professional development training (identify topics and duration) have been provided? - In the affirmative, to what extent has this support been beneficial a) to CCIMD staff/personnel professional development, b) for strengthening STEM schools and c) for improving science/math standards? 65 KEY INFORMANT INTERVIEW PROTOCOL FOR THE NATIONAL CENTER FOR EXAMINATIONS AND EDUCATIONAL EVALUATION (NCEEE) EVALUATION OF THE EGYPT STEM SCHOOLS PROJECT Good morning/afternoon. My name is ______________. I am a member of an evaluation team endorsed by the Ministry of Education to conduct an evaluation of the Egypt STEM Schools Project (ESSP). The evaluation team will review, analyze and evaluate the USAID-funded STEM Schools Project activities carried out by World Learning and the degree to which they have achieved their objectives. The evaluation team has been endorsed by the Ministry of Education to review, analyze and evaluate the USAID￾funded STEM school activities. The USAID Office of Education and Health is funding three three-person field teams specialized in evaluation to conduct interviews, group discussions, classroom observations, and surveys in the STEM schools. The evaluation team also will interview officials in the MOE STEM Unit and members of the World Learning project team. The findings, conclusions and recommendations of this evaluation are intended to provide program recommendations and lessons learned to ensure that STEM activities in Egypt have the highest potential to achieve their intended results in a sustainable manner. Findings of this evaluation will be managed in a confidential and anonymous manner. Your participation is entirely voluntary but your participation is considered vital to this important initiative. I thank you in advance for your valued contribution and I would be pleased to answer any questions that you may have in the course of our discussion. 1. What is your position within NCEEE? 2. How long have you been working with STEM education in Egypt? 3. How long have you been collaborating with the MOE STEM Unit? 4. How do you describe NCEEE’s role in the development of the STEM model schools in Egypt? 5. How would you describe NCEEE’s contributions to STEM model school development in Egypt? 6. We understand NCEEE develops tests to assess applicants’ (prep school graduates) creative thinking skills and IQ levels as requirements for application to and admission into STEM schools, and that these tests are administered in the field with the support (or under the supervision) of NCEEE. The tests are then graded by NCEEE and the results communicated to the STEM Unit at the Ministry of Education. - Is this an accurate description of the process? - To the contrary, please explain what is different? - What is the rationale behind the student assessment process? - Please describe what “creative thinking skills” tests aim to measure/assess? - How is that particularly relevant to STEM education? - How often are creative thinking skills and/or IQ level tests updated? - Describe the process by which a decision is made to update creative thinking skills and/or IQ level tests? 66 7. Other than for the end-of-year-three exit exam, has NCEEE cooperated with the MOE STEM Unit or the ESSP project for developing additional tools/measures to assess students’ performance? - In the affirmative, describe the tools/measures. - How frequently these additional tools being administered in the STEM schools? - How frequently are they being (or should be) updated? 8. Has NCEEE cooperated with the MOE STEM Unit or the ESSP project for developing the university readiness test (URT)? - In the affirmative, describe what has been NCEEE’s role in this process? - Please describe what the “university readiness test” aims to measure? - What organization/entity is responsible for grading the university readiness test (URT)? - If the URT is not in its final form, when do you expect it to be finalized? - Does the URT require approval by the MOE STEM Unit? - Has it already been administered in schools? - If not yet, when is it expected to be administered? - How frequently is the URT (or should be) updated? 9. Should NCEEE play a different role in the assessment processes of STEM schools? Please explain. 10. Has NCEEE received any support, or professional development training from the ESSP project? - In the affirmative, what type of support or professional development training (identify topics and duration) have been provided? - In the affirmative, to what extent has this support been beneficial a) to NCEEE staff/personnel professional development, b) for strengthening STEM schools and c) for improving science/math standards? 67 KEY INFORMANT INTERVIEW PROTOCOL FOR NATIONAL CENTER FOR EDUCATIONAL RESEARCH AND DEVELOPMENT (NCERD) EVALUATION OF THE EGYPT STEM SCHOOLS PROJECT Good morning/afternoon. My name is ______________. I am a member of an evaluation team endorsed by the Ministry of Education to conduct an evaluation of the Egypt STEM Schools Project (ESSP). The evaluation team will review, analyze and evaluate the USAID-funded STEM Schools Project activities carried out by World Learning and the degree to which they have achieved their objectives. The evaluation team has been endorsed by the Ministry of Education to review, analyze and evaluate the USAID￾funded STEM school activities. The USAID Office of Education and Health is funding three three-person field teams specialized in evaluation to conduct interviews, group discussions, classroom observations, and surveys in the STEM schools. The evaluation team also will interview officials in the MOE STEM Unit and members of the World Learning project team. The findings, conclusions and recommendations of this evaluation are intended to provide program recommendations and lessons learned to ensure that STEM activities in Egypt have the highest potential to achieve their intended results in a sustainable manner. Findings of this evaluation will be managed in a confidential and anonymous manner. Your participation is entirely voluntary but your participation is considered vital to this important initiative. I thank you in advance for your valued contribution and I would be pleased to answer any questions that you may have in the course of our discussion. 1. What is your position within NCERD? 2. How long have you been working with STEM education in Egypt? 3. How long have you been collaborating with the MOE STEM Unit? 4. What is the mandate of NCERD, and what is its relationship to the MOE? 5. How do you describe NCERD’s role in the development of the STEM model schools in Egypt? [Probes: curriculum development, research, standards, assessment] 6. Has this role changed since 2012? If yes, in what way? 7. How would you describe NCERD’s contributions to STEM model school development in Egypt? 8. What relationship has NCERD had with the World Learning STEM project? [Probe: work with consultants, workshop participation, providing guidance or input to ESSP activities?] 9. Has NCERD received any support, or professional development training from the ESSP project? - What type of support or professional development training (identify topics and duration) have been provided? - To what extent has this support been beneficial a) to CCIMD staff/personnel professional development, b) for strengthening STEM schools and c) for improving science/math standards? 68 10. Go to STEM Unit protocol for additional questions on the Technical STEM Unit, if appropriate. Give particular attention to the Capstone projects if this has not already been covered. 11. In your opinion, should STEM model schools be scaled-up; i.e., offered to more students nation￾wide? If in the affirmative, what policies, measures and resources does the MOE need to have in place to scale-up successfully? 69 INTERVIEW PROTOCOL FOR THE DISTRICT STEM UNITS (KEY INFORMANT INTERVIEW OR GROUP DISCUSSION) EVALUATION OF THE EDUCATION CONSORTIUM FOR THE ADVANCEMENT OF STEM IN EGYPT Good morning/afternoon. My name is ______________. I am a member of an evaluation team endorsed by the Ministry of Education to conduct an evaluation of the Egypt STEM Schools Project (ESSP). The evaluation team will review, analyze and evaluate the USAID-funded STEM Schools Project activities carried out by World Learning and the degree to which they have achieved their objectives. The evaluation team has been endorsed by the Ministry of Education to review, analyze and evaluate the USAID￾funded STEM school activities. The USAID Office of Education and Health is funding three three-person field teams specialized in evaluation to conduct interviews, group discussions, classroom observations, and surveys in the STEM schools. The evaluation team also will interview officials in the MOE STEM Unit and members of the World Learning project team. The findings, conclusions and recommendations of this evaluation are intended to provide program recommendations and lessons learned to ensure that STEM activities in Egypt have the highest potential to achieve their intended results in a sustainable manner. Findings of this evaluation will be managed in a confidential and anonymous manner. Your participation is entirely voluntary but your participation is considered vital to this important initiative. I thank you in advance for your valued contribution and I would be pleased to answer any questions that you may have in the course of our discussion. 1. When was the STEM Unit formed in this district? What is the structure of the Unit? Who are the members? 2. What are the tasks and responsibilities of this Unit? What is your relationship to the central STEM Unit? How do you interact with the STEM model school in your district? 3. Is this Unit active in recruitment and selection of students and teachers in this governorate? 4. Looking forward, what are your plans for strengthening the STEM program in this district? - Strengthening the ties to the community? - Developing PPPs? - Incorporating the STEM school practices into the mainstream schools? - Other plans? 5. Have you and other members of the STEM Unit received training from the USAID STEM project? What training have you received? Has it been useful to you? What additional training is needed? 6. In your experience, what have been the main problems with the STEM model school in your district? 70 KEY INFORMANT INTERVIEW PROTOCOL FOR STEM SCHOOL ADMINISTRATORS (PRINCIPALS AND ASSISTANT PRINCIPALS) EVALUATION OF THE EDUCATION CONSORTIUM FOR THE ADVANCEMENT OF STEM IN EGYPT Good morning/afternoon. My name is ______________. I am a member of an evaluation team endorsed by the Ministry of Education to conduct an evaluation of the Egypt STEM Schools Project (ESSP). The evaluation team will review, analyze and evaluate the USAID-funded STEM Schools Project activities carried out by World Learning and the degree to which they have achieved their objectives. The evaluation team has been endorsed by the Ministry of Education to review, analyze and evaluate the USAID￾funded STEM school activities. The USAID Office of Education and Health is funding three three-person field teams specialized in evaluation to conduct interviews, group discussions, classroom observations, and surveys in the STEM schools. The evaluation team also will interview officials in the MOE STEM Unit and members of the World Learning project team. The findings, conclusions and recommendations of this evaluation are intended to provide program recommendations and lessons learned to ensure that STEM activities in Egypt have the highest potential to achieve their intended results in a sustainable manner. Findings of this evaluation will be managed in a confidential and anonymous manner. Your participation is entirely voluntary but your participation is considered vital to this important initiative. I thank you in advance for your valued contribution and I would be pleased to answer any questions that you may have in the course of our discussion. STEM School: (Note: The Deputy Principle is referred to as the academic coach, his or her primary responsibility is technical supervision of the teachers) Gender: 1. Personal background: previous career, work experience, education (subject, degrees); time/roles in the STEM school (and in this school) (additional factors: gender, international experience) 2. Role: What do you consider the most important part of your job? What responsibilities consume most of your time? Probes: operational responsibilities; interaction with the students- teachers-parents; providing guidance for staff; Capstone Project; school/community relations; generating support for STEM (PPPs and other). - What has your role been in the recruitment, selection, and contracting of teachers? - How do you assess this role? - What have you observed in terms of turnover and availability of qualified staff – in reference to this school? 71 - Do you have a role in teacher assessment and coaching? - Does this school have a BOT? Why or why not; when formed? - How do you collaborate with the BOT? - How do you assess your role in relation to the central STEM Unit and the MOE bureaucracy? - Is there a STEM Unit in the moderaya? - Probes: Do they set the requirements; they provide guidance; do they seek your input on decisions; is innovation on your part encouraged, rewarded? Annual School Action Plan? 3. Training: The World Learning STEM Project has taken the lead in training for STEM school principals. Have you received training? (What, how much, how often, online, in-person.) How do you assess this training? Has it been relevant and useful in meeting your responsibilities? Quality of presentation? Additional training needed? Sessions that could be omitted? 4. Community Outreach and Relations: Outreach and relevance to the community is an important aspect of the STEM school model in Egypt. What steps are you taking to integrate the school and the community? Probes: Annual School Action Plan; community mapping; Capstone Projects; community events; BOT participation; FabLab) Outreach to local companies, businesses, NGOs and other community groups? Tailoring of the curriculum or the Capstone Projects to community interests and needs? Outreach to surrounding Preparatory Schools and students? Are you pursuing any local PPPs? 5. Future Plans and Recommendations: - What do you see as the most important next steps to strengthen this school as a model 72 STEM school? - What are you doing to move in that direction? - What are the principal obstacles – or - what additional resources do you need to make it happen? - Do you have any recommendations for actions and policies that would strengthen the STEM school model and this school? 73 KEY INFORMANT INTERVIEW PROTOCOL FOR STEM IMPLEMENTING PARTNERS (World Learning) EVALUATION OF THE EDUCATION CONSORTIUM FOR THE ADVANCEMENT OF STEM IN EGYPT Good morning/afternoon. My name is ______________. I am a member of an evaluation team endorsed by the Ministry of Education to conduct an evaluation of the Egypt STEM Schools Project (ESSP). The evaluation team will review, analyze and evaluate the USAID-funded STEM Schools Project activities carried out by World Learning and the degree to which they have achieved their objectives. The evaluation team has been endorsed by the Ministry of Education to review, analyze and evaluate the USAID￾funded STEM school activities. The USAID Office of Education and Health is funding three three-person field teams specialized in evaluation to conduct interviews, group discussions, classroom observations, and surveys in the STEM schools. The evaluation team also will interview officials in the MOE STEM Unit and members of the World Learning project team. The findings, conclusions and recommendations of this evaluation are intended to provide program recommendations and lessons learned to ensure that STEM activities in Egypt have the highest potential to achieve their intended results in a sustainable manner. Findings of this evaluation will be managed in a confidential and anonymous manner. Your participation is entirely voluntary but your participation is considered vital to this important initiative. I thank you in advance for your valued contribution and I would be pleased to answer any questions that you may have. Overview and Background – organizational structure and mode of operation [Some or all of these questions will be covered in the introductory meetings with World Learning and the MOE on Sunday, 19 March 2017.] 1. ESSP organizational structure: long-term in-country staff: positions and tasks. What is the “field team,” as referenced in the annual reports? 2. Describe how the ESSP works with the Ministry of Education. - Who is the principal point of contact or partner? (especially in terms of decision-making) - What is the relationship of the ESSP to the MOE STEM Unit? What about the local/moderaya STEM Units? - What is the relationship between the ESSP/World Learning and the Counselors for Science and Mathematics? (Probe: The STEM model introduces a new approach to these subjects not only in the STEM schools, but also in the mainstream high schools.) - In what ways, if at all, has the relationship between the ESSP and the MOE changed during the course of the project? - 3. How does the ESSP interact/work with the other MOE organizations, i.e., PAT, NCEEE, NCERD, CCIMD? (process over the term of the project, points of contact, and types of interaction) 4. Describe the contact between World Learning staff and the STEM schools. - Who are the principal points of contact for World Learning and in the schools? - How often does World Learning have direct interaction with the school? - Do World Learning staff go to the schools? 74 5. World Learning worked with three sub-grantees in implementing the cooperative agreement. - What has their role been in design and implementation? - What were the logistics of working with them? - We will need information on the key people to contact to get their feedback on project implementation and results. Achievement of Objectives and Next Steps 1. Thinking about the STEM project over the past five years, what do you see as ESSP’s primary achievement? [Ask this question of COP and DCOP, and of people in other key technical positions It also could be a question for a group discussion.] - As follow-up, reference the question to each of the target groups. What do you see as the primary achievement with . . . students, STEM Unit, school principals, teachers, and communities? - - and/or reference the question to the objectives. What was the primary achievement in terms of . . . (1) increased student interest and achievement in STEM; (2) Developing model schools as centers of excellence in diverse communities; (3) building capacity of a cadre of STEM professionals in Egypt; (4) strengthening MOE systems and policies to sustain and replicate the model; (5) upgrading science and mathematics curriculum, student assessment, and teacher preparation in mainstream schools? 2. In your work with the MOE, in implementing the ESSP, what do you consider to be the strengths of your relationship (the factors that contributed to achieving the project objectives)? 3. In your work with the MOE, in implementing the project, what obstacles did you encounter (especially unexpected factors) that slowed project implementation or resulted in some change in direction? How did you deal with [these obstacles]? - Repeat questions 2 and 3 with reference to other target groups – students, teachers, administrators, and communities. (Probe about the positive, negative, and unexpected aspects of gender as a factor in meeting objectives.) 4. Now, at the end of the project, in your opinion, what gaps remain (what are the next steps) in: - Engaging students in STEM subjects and careers (Probe about gender differences) - Strengthening the model STEM schools in in the communities - Developing the central STEM Unit and the local STEM units - Implementing the STEM school model in terms of staffing (teachers, administrators, and other support staff - Incorporating best practices in science and math from the STEM schools into the mainstream schools 75 - Handing-off the management, maintenance, and further development of the tools, systems, and policies developed through ESSP. In your eyes, what are the greatest risks in terms of sustainability and growth? (Probe: available resources in terms of human capital and financial.) 5. As the implementing partner for a USAID project, what steps have you taken to meet USAID requirements for Monitoring and Evaluation and for gender mainstreaming? - Suggest a separate meeting to discuss the indicators, data quality, and any additional studies and reports developed. (For example, information on student achievement in terms of international standards.) - Examine the documentation on gender mainstreaming in the project, and discuss actual and potential steps taken to build gender balance. 76 KEY INFORMANT INTERVIEW PROTOCOL FOR THE PROFESSIONAL ACADEMY FOR TEACHERS (PAT) EVALUATION OF THE EGYPT STEM SCHOOLS PROJECT Good morning/afternoon. My name is ______________. I am a member of an evaluation team endorsed by the Ministry of Education to conduct an evaluation of the Egypt STEM Schools Project (ESSP). The evaluation team will review, analyze and evaluate the USAID-funded STEM Schools Project activities carried out by World Learning and the degree to which they have achieved their objectives. The evaluation team has been endorsed by the Ministry of Education to review, analyze and evaluate the USAID￾funded STEM school activities. The USAID Office of Education and Health is funding three three-person field teams specialized in evaluation to conduct interviews, group discussions, classroom observations, and surveys in the STEM schools. The evaluation team also will interview officials in the MOE STEM Unit and members of the World Learning project team. The findings, conclusions and recommendations of this evaluation are intended to provide program recommendations and lessons learned to ensure that STEM activities in Egypt have the highest potential to achieve their intended results in a sustainable manner. Findings of this evaluation will be managed in a confidential and anonymous manner. Your participation is entirely voluntary but your participation is considered vital to this important initiative. I thank you in advance for your valued contribution and I would be pleased to answer any questions that you may have in the course of our discussion. 1. What is your position within PAT? 2. How long have you been working with STEM education in Egypt? 3. How long have you been collaborating with the MOE STEM Unit? 4. How do you describe PAT’s role in the development of the STEM model schools in Egypt? 5. How would you describe PAT’s contributions to STEM model school development in Egypt? 6. We understand that PAT manages the recruitment process for STEM school personnel; including, for example, teachers, principals, deputy principals, et.al. 7. Please describe the recruitment process. 8. Please describe PAT’s role, and other institutions with which it collaborates, in the recruitment process? 9. It is our understanding that one of the principle challenges for the development and expansion of STEM schools is a shortage of qualified teachers. 10. How would you describe the seriousness of this shortage? 11. Is this shortage more prevalent in particular subjects? Please explain. 77 12. Is this shortage more prevalent in certain governorates than others? 13. In your opinion, why is the pool of teacher applicants limited? Why don’t more teachers apply to work in STEM schools? 14. To the best of your knowledge, is there a shortage in applicants for select positions in STEM schools; e.g., principals, deputy principals, other? Please explain. 15. Has PAT monitored the characteristics of the individuals who have applied to teach in the STEM schools? 16. Does PAT have any documentation as to the sources of the shortage of teachers or of the shortcomings of the applicants in terms of qualifications? 17. It is our understanding that PAT has been cooperating with the ESSP project in order to have a number of trained teachers certified by PAT as Trainers of Trainers. - What is the current status of the training of trainers? - To date, how many Master Trainers have been certified by PAT? - Which STEM schools now have Master Trainers? - What are the reasons as to why some have not been certified by PAT? - How long does it take to train a Master Trainer? - What is required for certification as a Master Trainer? - Describe how PAT’s utilizes Master Trainers? 18. In your opinion, what policies, measures and resources does the MOE need to have in place to ensure adequate numbers of qualified personnel are recruited and hired, on time, for STEM schools? 19. In your opinion, what policies, measures and resources does the MOE need to have in place to ensure qualified personnel are retained in STEM schools? 20. Has PAT received any support, or professional development training from the ESSP project? - In the affirmative, what type of support or professional development training (identify topics and duration) have been provided? - In the affirmative, to what extent has this support been beneficial a) to PAT staff/personnel professional development, b) for strengthening STEM schools and c) for improving science/math standards? 78 KEY INFORMANT INTERVIEW PROTOCOL FOR THE DIRECTOR OF MOE TECHNOLOGY DEVELOPMENT CENTER (TDC) EVALUATION OF THE EDUCATION CONSORTIUM FOR THE ADVANCEMENT OF STEM IN EGYPT Good morning/afternoon. My name is ______________. I am a member of an evaluation team endorsed by the Ministry of Education to conduct an evaluation of the Egypt STEM Schools Project (ESSP). The evaluation team will review, analyze and evaluate the USAID-funded STEM Schools Project activities carried out by World Learning and the degree to which they have achieved their objectives. The evaluation team has been endorsed by the Ministry of Education to review, analyze and evaluate the USAID￾funded STEM school activities. The USAID Office of Education and Health is funding three three-person field teams specialized in evaluation to conduct interviews, group discussions, classroom observations, and surveys in the STEM schools. The evaluation team also will interview officials in the MOE STEM Unit and members of the World Learning project team. The findings, conclusions and recommendations of this evaluation are intended to provide program recommendations and lessons learned to ensure that STEM activities in Egypt have the highest potential to achieve their intended results in a sustainable manner. Findings of this evaluation will be managed in a confidential and anonymous manner. Your participation is entirely voluntary but your participation is considered vital to this important initiative. I thank you in advance for your valued contribution and I would be pleased to answer any questions that you may have in the course of our discussion. 1. General information about the TDC (e.g., function, staffing, location in the bureaucracy, when started and how role has changed.) 2. What is the function of the TDC units in the governorates? 3. What is the relationship of the TDC to the STEM Unit? 4. Representation and participation? 5. Reporting to the STEM Unit? 6. How has the relationship changed over time? 7. What interaction has the TDC had with the World Learning STEM project? 8. What are the functions of the TDC in the STEM school system? 9. Student recruitment and admission? 10. Student examinations and marks/achievement scores? 11. STEM Blueprint? 12. PARLO? 13. Start-up of new schools? 14. Provision of equipment and/or infrastructure? 15. Now that the World Learning project is ending, what has the process been for the transfer of functions and systems from World Learning to the TDC? How do you assess this process? 16. Authorization 17. Collaboration and designation of personnel 18. Training 19. Adjustments within the TDC in terms of infrastructure and personnel 20. What is the process for the TDC to receive and manage the STEM system information? 21. How does the TDC receive data from the STEM schools? 22. How does the TDC receive data from the STEM Unit? 79 23. What is the process for managing the student applications and admissions? 24. What is the process for managing the examinations and achievement scores? 25. Are there other sources of information for the STEM system? 26. What procedures are in place for managing, verifying, and protecting the STEM system data? What are the procedures for updating and day-to-day maintenance of the files? 27. What is your assessment of the adequacy of these procedures? 28. What gaps have you identified and what is needed to correct these problems? 29. What is the staffing configuration for the STEM data systems? 30. Is this staffing adequate to manage the system now and as it expands? 31. What training has the staff received? What additional training, if any, is needed? 32. What is your assessment of the adequacy of the infrastructure for the STEM system data in terms of hardware and software? 33. Is the infrastructure sufficient to accommodate the coming expansion of the STEM system schools? 34. What procedures are in place for maintenance, updating, and replacement of the infrastructure? 35. What concerns do you have about the role of the TDC in the STEM system going forward? What steps are being taken to correct these potential problems? 36. Do you have any additional comments or recommendations for the evaluation? 80 KEY INFORMANT INTERVIEW PROTOCOL FOR MSI EVALUATION OF THE EGYPT STEM SCHOOLS PROJECT Good morning/afternoon. My name is ______________. I am a member of an evaluation team endorsed by the Ministry of Education to conduct an evaluation of the Egypt STEM Schools Project (ESSP). The evaluation team will review, analyze and evaluate the USAID-funded STEM Schools Project activities carried out by World Learning and the degree to which they have achieved their objectives. The evaluation team has been endorsed by the Ministry of Education to review, analyze and evaluate the USAID￾funded STEM school activities. The USAID Office of Education and Health is funding three three-person field teams specialized in evaluation to conduct interviews, group discussions, classroom observations, and surveys in the STEM schools. The evaluation team also will interview officials in the MOE STEM Unit and members of the World Learning project team. The findings, conclusions and recommendations of this evaluation are intended to provide program recommendations and lessons learned to ensure that STEM activities in Egypt have the highest potential to achieve their intended results in a sustainable manner. Findings of this evaluation will be managed in a confidential and anonymous manner. Your participation is entirely voluntary but your participation is considered vital to this important initiative. I thank you in advance for your valued contribution and I would be pleased to answer any questions that you may have in the course of our discussion. Note: Each organization will be interviewed separately through a telephone call to the home office in the US. The topics to be discussed are the same in each case. A list of the ESSP activities of each organization, based on the ESSP project documents, is attached to tailor the questions to each specific role. 1. What were the primary activities of [insert name of organization interviewed] in the Egypt Stem Schools Project? - Was this your first experience working with STEM in Egypt? (other comparable experience?) - To what extent did you implement these activities in Egypt and/or virtually? - Did you have face-to-face contact with the MOE, school teachers and administrators, or students? - How many [insert name of organization interviewed] staff were involved in this project, in the US and in Egypt? 2. In your opinion, what has been your primary achievement or contribution in this project? - To what extent did you coordinate your work with World Learning and the other partner organizations? - Was your primary achievement the result of a joint effort or a result of your independent work? - How do you assess the effect of the coordination on your capacity to carry out your role in the project? 3. In your opinion, what were the primary shortcomings and unfinished tasks in terms of your [specify activities] in the project? 81 - What obstacles did you encounter and how did you respond to them? - Did you face any unexpected constraints that affected your ability to complete your work? 4. How do you assess the capacity and resources of the MOE to maintain and build on your contributions [specify activities, achievements] after the conclusion of the ESSP, especially in terms of the planned expansion of the STEM school model across Egypt? 5. What recommendations do you make for next steps for the MOE and/or USAID in terms of [specify activities, achievements]? 6. What are the next steps for Egypt in terms of international experience and standards in STEM education? 7. Please describe the background and process for development of the URT [University Readiness Test]. Where are you now in the process? 8. Who or what office in the MOE is responsible for managing the URT? What is the transfer process from MSI to the Ministry? Where there be regular revisions/updates to the exam? 9. Please comment on the following sources of concern that we have heard about in out interviews and discussions with students: - It is a combination of parts of the Thenaweya Amma and the ACT. - There is no way to study for the URT and the students don’t know what to expect. - Universities do not accept the URT – students applying for scholarships abroad still have to take the ACT and/or SAT. - Why is it given so much weight in the student’s final grade? 82 KEY INFORMANT INTERVIEW PROTOCOL FOR THE MOE ENGLISH COUNSELOR EVALUATION OF THE EDUCATION CONSORTIUM FOR THE ADVANCEMENT OF STEM IN EGYPT Good morning/afternoon. My name is ______________. I am a member of an evaluation team endorsed by the Ministry of Education to conduct an evaluation of the Egypt STEM Schools Project (ESSP). The evaluation team will review, analyze and evaluate the USAID-funded STEM Schools Project activities carried out by World Learning and the degree to which they have achieved their objectives. The evaluation team has been endorsed by the Ministry of Education to review, analyze and evaluate the USAID￾funded STEM school activities. The USAID Office of Education and Health is funding three three-person field teams specialized in evaluation to conduct interviews, group discussions, classroom observations, and surveys in the STEM schools. The evaluation team also will interview officials in the MOE STEM Unit and members of the World Learning project team. The findings, conclusions and recommendations of this evaluation are intended to provide program recommendations and lessons learned to ensure that STEM activities in Egypt have the highest potential to achieve their intended results in a sustainable manner. Findings of this evaluation will be managed in a confidential and anonymous manner. Your participation is entirely voluntary but your participation is considered vital to this important initiative. I thank you in advance for your valued contribution and I would be pleased to answer any questions that you may have in the course of our discussion. 1. When did you join the STEM Unit? 2. What types of training have you received from the USAID STEM project? What is your assessment of this training? - What about the quality of the training? - How well did it match your needs? Were you involved in the design of the training program? - Have you identified gaps in the training you received? - How do you assess the leadership training in particular? - Any other comments? 3. Please explain how you have worked with the STEM Unit. What has your role/contribution been since joining the Unit: - with the English Counselor’s Office - the STEM Unit itself and World Learning - with the schools, teachers, and students? 4. What is the English curriculum for STEM students? How, if at all, does it differ from the mainstream English curriculum? Where are the gaps in the English program in the STEM schools? 5. E-STEM – What role did you have in the development of the program? What do you see as its benefits and problems? To what extent do the teachers and students use E-STEM? 6. Next steps, recommendations – as per the Counselor’s protocol. 83 GROUP DISCUSSION PROTOCOL FOR THE STEM SCHOOL BOARD OF TRUSTEES EVALUATION OF THE EDUCATION CONSORTIUM FOR THE ADVANCEMENT OF STEM IN EGYPT Good morning/afternoon. My name is ______________. I am a member of an evaluation team endorsed by the Ministry of Education to conduct an evaluation of the Egypt STEM Schools Project (ESSP). The evaluation team will review, analyze and evaluate the USAID-funded STEM Schools Project activities carried out by World Learning and the degree to which they have achieved their objectives. The evaluation team has been endorsed by the Ministry of Education to review, analyze and evaluate the USAID￾funded STEM school activities. The USAID Office of Education and Health is funding three three-person field teams specialized in evaluation to conduct interviews, group discussions, classroom observations, and surveys in the STEM schools. The evaluation team also will interview officials in the MOE STEM Unit and members of the World Learning project team. The findings, conclusions and recommendations of this evaluation are intended to provide program recommendations and lessons learned to ensure that STEM activities in Egypt have the highest potential to achieve their intended results in a sustainable manner. Findings of this evaluation will be managed in a confidential and anonymous manner. Your participation is entirely voluntary but your participation is considered vital to this important initiative. I thank you in advance for your valued contribution and I would be pleased to answer any questions that you may have in the course of our discussion. 1. Introductions: a. Who (parent, community, school) are the members of the BOT? b. How were they selected? c. When was the BOT formed? How long has each member served? d. Total number of members/active members e. Frequency of elections and next election f. Do any students participate in or contact the BOT? 2. Does the BOT have a formal structure, officers, and records of their meetings and activities? 3. What training have the members received? Who provided the training? 4. How frequently does the board meet? 5. What are the BOT’s main activities this year? 6. Have the BOT’s contributed to school improvement plans this year or in previous years? 7. Can we access any of their “Asset Map”/ community mapping to inform action plans? 8. What is the plan for future activities? 9. In what ways does the BOT collaborate with the school administration? 10. Does the BOT have any recommendations for actions or policies to strengthen the STEM school model? 84 GROUP DISCUSSION PROTOCOL FOR CURRENTLY ENROLLED STEM SCHOOL STUDENTS EVALUATION OF THE EDUCATION CONSORTIUM FOR THE ADVANCEMENT OF STEM IN EGYPT Good morning/afternoon. My name is ______________. I am a member of an evaluation team endorsed by the Ministry of Education to conduct an evaluation of the Egypt STEM Schools Project (ESSP). The evaluation team will review, analyze and evaluate the USAID-funded STEM Schools Project activities carried out by World Learning and the degree to which they have achieved their objectives. The evaluation team has been endorsed by the Ministry of Education to review, analyze and evaluate the USAID￾funded STEM school activities. The USAID Office of Education and Health is funding three three-person field teams specialized in evaluation to conduct interviews, group discussions, classroom observations, and surveys in the STEM schools. The evaluation team also will interview officials in the MOE STEM Unit and members of the World Learning project team. The findings, conclusions and recommendations of this evaluation are intended to provide program recommendations and lessons learned to ensure that STEM activities in Egypt have the highest potential to achieve their intended results in a sustainable manner. Findings of this evaluation will be managed in a confidential and anonymous manner. Your participation is entirely voluntary but your participation is considered vital to this important initiative. I thank you in advance for your valued contribution and I would be pleased to answer any questions that you may have in the course of our discussion. Introduction: - STEM School - Grade - Gender - 12th grade specialization (as applicable) 1. Why did you want to join the STEM school? 2. What opportunities do you see for yourself in this school? 3. What is different in this model compared to other high schools? 4. So far, what have been the most beneficial or useful features of the school for you in terms of your learning and experience? 5. What do you think of the Capstone Project? Probes: relevance to the community; support from the school; adequacy of resources; guidance from the school; role of the teacher. Did you use your Capstone project to enter any competitions? 6. Have you participated in any extracurricular activities? Probes: student union, symposium, field visit, internship, leadership summer camp. 85 What do you think of them? How have they affected your high school experience and learning? 7. What are the challenges/barriers in the STEM school on the personal and academic level? Probes: Parents feelings about the STEM school; housing and transportation; infrastructure; Internet; English proficiency; teachers; administration; channels of communications; student input to school operation; university seats (access); examination/assessment; student distribution within grades. Who do you turn to for advice for personal or school-related problems? Do you find enough support in the school? Some students leave the STEM schools before graduating. In your opinion, what are the main reasons that STEM students drop out? 8. How do you see your next step after graduation from secondary school? If you plan to attend university, how do you think graduation from a STEM school will affect your options? Probes: university quota; possible fields of study; expected career path; funding; travel abroad; study abroad. 9. What suggestions do you have to improve the STEM schools? NOTE: These are the questions for the online survey. The instructions in parentheses are directed to the person who will format the questionnaire for Survey Monkey after approval of the content. 86 GROUP DISCUSSION PROTOCOL FOR STEM SCHOOL GRADUATED STUDENTS EVALUATION OF THE EDUCATION CONSORTIUM FOR THE ADVANCEMENT OF STEM IN EGYPT Good morning/afternoon. My name is ______________. I am a member of an evaluation team endorsed by the Ministry of Education to conduct an evaluation of the Egypt STEM Schools Project (ESSP). The evaluation team will review, analyze and evaluate the USAID-funded STEM Schools Project activities carried out by World Learning and the degree to which they have achieved their objectives. The evaluation team has been endorsed by the Ministry of Education to review, analyze and evaluate the USAID￾funded STEM school activities. The USAID Office of Education and Health is funding three three-person field teams specialized in evaluation to conduct interviews, group discussions, classroom observations, and surveys in the STEM schools. The evaluation team also will interview officials in the MOE STEM Unit and members of the World Learning project team. The findings, conclusions and recommendations of this evaluation are intended to provide program recommendations and lessons learned to ensure that STEM activities in Egypt have the highest potential to achieve their intended results in a sustainable manner. Findings of this evaluation will be managed in a confidential and anonymous manner. Your participation is entirely voluntary but your participation is considered vital to this important initiative. I thank you in advance for your valued contribution and I would be pleased to answer any questions that you may have in the course of our discussion. Introductions: ● Academic Focus in STEM school – Math, Science ● University (currently enrolled, where, discipline, funding) or Other (what?) Achievement: Looking back, how would you reflect on your experience/education at the STEM school? ● From an academic perspective: Have you benefited? How did it prepare you for your academic career (knowledge, skills: critical thinking, collaborative work)? Did the experience affect your choice of discipline in the university? ● From a personal perspective: How did the experience affect you? Influential Factors: ● Looking back, what were the features/components (curriculum, inquiry based, student-centered, infrastructure, etc.) of the STEM school you valued most (that were most beneficial for you)? Why? ● On the other hand, what were the features/components that you have found less useful or 87 beneficial for you? Why? Constraints: ● In your experience, during your study at the STEM school, what were the main challenges that you faced in the school? How did you overcome them? ● Did you know STEM students in your class who dropped out of school before graduating? If yes, do you know the reasons? Policies/Ministry of Education: Do you know of other prep school students who are interested in attending STEM schools? If yes, ● Would you recommend that certain/some features/components be modified, or added? What features, and why? ● Should the STEM methodology (including the teaching methods, curriculum, etc.) be mainstreamed to public education? ● What are your career plans post university? ● To what extent was your education in the STEM school related to the needs of the community? Did it make you more prepared to respond to community needs? 88 GROUP DISCUSSION/KEY INFORMANT INTERVIEW PROTOCOL FOR STEM SCHOOL SUPPORT STAFF (Capstone Director, Social Worker, Librarian, Lab Technician) EVALUATION OF THE EDUCATION CONSORTIUM FOR THE ADVANCEMENT OF STEM IN EGYPT Good morning/afternoon. My name is ______________. I am a member of an evaluation team endorsed by the Ministry of Education to conduct an evaluation of the Egypt STEM Schools Project (ESSP). The evaluation team will review, analyze and evaluate the USAID-funded STEM Schools Project activities carried out by World Learning and the degree to which they have achieved their objectives. The evaluation team has been endorsed by the Ministry of Education to review, analyze and evaluate the USAID￾funded STEM school activities. The USAID Office of Education and Health is funding three three-person field teams specialized in evaluation to conduct interviews, group discussions, classroom observations, and surveys in the STEM schools. The evaluation team also will interview officials in the MOE STEM Unit and members of the World Learning project team. The findings, conclusions and recommendations of this evaluation are intended to provide program recommendations and lessons learned to ensure that STEM activities in Egypt have the highest potential to achieve their intended results in a sustainable manner. Findings of this evaluation will be managed in a confidential and anonymous manner. Your participation is entirely voluntary but your participation is considered vital to this important initiative. I thank you in advance for your valued contribution and I would be pleased to answer any questions that you may have in the course of our discussion. STEM School: General: 1. What was the hiring process for your position? Probes: What were the qualifications; how did you apply; who made the hiring decision; how long have you been in this position? 2. What are your main responsibilities? (What other school staff do you work with most? Who supervises your work?) 3. Have you received training from the STEM project? Probes: What training; has it been useful? Have you received other training since you started this job at this school? What additional training do you need for this position? Are there gaps in your skills? 4. What are the challenges you have faced in this job? Probes: English; infrastructure; materials; dealing with students, teachers, or administration. 5. How do you interact with the STEM students? What do you think of the STEM students? What are the challenges and rewards? 89 6. Do you have any recommendations to improve the model STEM school? Capstone Leaders Discussion: 7. How many Capstone themes are reflected in the projects in this school? How/who decides on the Capstone theme for the project(s)? (Probe: differences among grades?) 8. What is your role in carrying out the Capstone projects? Probes: when do you enter the process? Describe how you interact with the students, teachers, and administrators. 9. Are you involved in helping the students prepare for competitions? 10. Do you have a role in communicating the results of the Capstone projects to the community? Social Worker: 11. Describe how you work with the BOT. Have you done any training for the BOT members? What plans do you have for strengthening the role of the BOT? 12. Describe how you interact with students and parents? What are students’ social backgrounds? What kinds of non-academic problems do the students face in the STEM school? In your experience in this school, how many students have dropped out and why? Librarian/Laboratory Technician/FavLab Manager: 13. How do you assess the school infrastructure and facilities? Probes: adequacy; quality, materials. 14. Have you encountered any issues in day-to-day management/use of this facility? 15. Have you encountered any issues regarding how the students and/or teachers use the facility? 90 GROUP DISCUSSION PROTOCOL FOR STEM SCHOOL TEACHERS EVALUATION OF THE EDUCATION CONSORTIUM FOR THE ADVANCEMENT OF STEM IN EGYPT Good morning/afternoon. My name is ______________. I am a member of an evaluation team endorsed by the Ministry of Education to conduct an evaluation of the Egypt STEM Schools Project (ESSP). The evaluation team will review, analyze and evaluate the USAID-funded STEM Schools Project activities carried out by World Learning and the degree to which they have achieved their objectives. The evaluation team has been endorsed by the Ministry of Education to review, analyze and evaluate the USAID￾funded STEM school activities. The USAID Office of Education and Health is funding three three-person field teams specialized in evaluation to conduct interviews, group discussions, classroom observations, and surveys in the STEM schools. The evaluation team also will interview officials in the MOE STEM Unit and members of the World Learning project team. The findings, conclusions and recommendations of this evaluation are intended to provide program recommendations and lessons learned to ensure that STEM activities in Egypt have the highest potential to achieve their intended results in a sustainable manner. Findings of this evaluation will be managed in a confidential and anonymous manner. Your participation is entirely voluntary but your participation is considered vital to this important initiative. I thank you in advance for your valued contribution and I would be pleased to answer any questions that you may have in the course of our discussion. Introduction: - STEM School - Grade Level - Gender 1- Why did you apply to teach in a STEM school? Probes: educational resources; commitment to STEM; teaching/learning environment; monetary or career incentives; to build cadre of STEM professionals in Egypt. - How do you feel about the recruitment process? Probes: access to information, requirements, objectivity, and transparency. - How do you feel about the 1 year contract? Probes: Does it affect your career plans or commitment to teaching? - What is different in this model compared to other high schools? - So far, what have been the most beneficial or useful features of the school for you in terms of your learning and professional experience? 2- Have you observed any changes in the characteristics of the teachers in the STEM schools over time (or since you joined the school)? Probes: skills; career paths; commitment/turnover; background; diversity. - Why are these changes occurring? 91 Probes: Workload; career path; incentives; competing employment opportunities. - What observations do you have in comparing teachers of STEM and non-STEM subjects? - What about the interactions between them in the school? 3-What are the gaps in skills or additional training programs that you need to improve your work in the STEM school? Probes: dealing with adolescents' needs; moods; egos; teaching gifted students; becoming a facilitator rather than a lecturer; knowledge of subject. - Have you looked for/used additional training/resources (online/certificates-sessions- readings) on your own since joining STEM? 4- Thinking about the various components of the STEM school program, what factors contribute to your work in the school? What factors hinder your work? Explain. (Probes: labs; FabLab; library; training; textbooks; school community (students, colleagues, administration); student-centered versus teacher-centered teaching methodology. - What about the tools and techniques for interacting with students and providing feedback? Probes: technological interactive techniques with students e.g. PARLO, Qitidi, on-line databases. 5- Thinking about your involvement with the students, how do you evaluate the changes and achievements in their learning, knowledge, and application? - How do you evaluate changes you observe in their personalities and character? 6- What are your recommendations to strengthen: - Teachers’ performance in the STEM school? - Students’ learning experience in the STEM school? 7- How do you assess the school infrastructure and facilities? Have you encountered any issues in day￾to-day management/use of this facility? (Probes: adequacy; quality, materials). 92 ONLINE SURVEY FOR STEM SCHOOL GRADUATED STUDENTS EVALUATION OF THE EDUCATION CONSORTIUM FOR THE ADVANCEMENT OF STEM IN EGYPT This survey is being conducted for the evaluation of the Egypt STEM Schools Project (ESSP), implemented by World Learning. You have been asked to participate in this survey because you are a part of the STEM school community in Egypt. Welcome to the survey and thank you for you valued contributions to this evaluation. The evaluation team has been endorsed by the Ministry of Education to review, analyze and evaluate the USAID￾funded STEM school activities. The USAID Office of Education and Health is funding three three-person field teams specialized in evaluation to conduct interviews, group discussions, classroom observations, and surveys in the STEM schools. The evaluation team also will interview officials in the MOE STEM Unit and members of the World Learning project team. Your participation in this survey is very important as your responses, together with those of other students and teachers, will inform the findings, conclusions and recommendations of the evaluation for USAID and the Ministry of Education, to strengthen the STEM school programs in Egypt. The questions should take about 15 minutes to complete. All responses are completely anonymous. No individual responses will be reported and the results of this study will be used for research purposes only. Your participation is entirely voluntary, but your support is important to ensure that all points of view are reflected in the evaluation. Thank you for your valued contribution to this important initiative. 1. Year of birth (drop down menu only years) (from 1995-2005) 2. Gender (male, female) (drop down menu) 3. Home governorate (drop down menu) 4. Place of residence (City / Village, drop down both options in order to choose only one) 5. Year of STEM school graduation: (drop down menu) (2014-2016) 6. STEM academic discipline: math – science (drop down menu) 7. How did you learn about the STEM schools? (Choose only the MAIN one) (checkbox, allow for one only) ▪ Ministry website ▪ Newspaper/TV ▪ Social media ▪ My preparatory school (announcement, teacher, social worker, other personnel) ▪ Word-of-mouth (family, friend, peers…) ▪ Older STEM student/graduate ▪ Other (specify): ___________________________________ 8. Why did you want to enroll in the STEM school? (Choose only the MAIN one) (checkbox, allow for one only) ▪ Parents suggestions/recommendations ▪ I wanted to try something new ▪ Interest in studying STEM subjects ▪ Better education 93 ▪ Better career qualifications ▪ Other (specify): _____________________________________ 9. Are you currently enrolled in university? Yes/No If Yes (skip for those answered No to questions from 10 through 13) 10. Location of the university (drop down menu with the following options: home Governorate in Egypt, other Governorate in Egypt, North America, Europe, Asia, Other) 11. Type of university (drop down menu with the options: public, private). 12. Field of study (Open ended Question) 13. What is funding source (drop down menu with the following options) ▪ Self-financed ▪ Scholarship ▪ Combined (partial scholarship) 14. What are your plans after graduation? (Choose only the MAIN one) (checkbox, allow for one only) ▪ Seek employment in Egypt ▪ Seek employment abroad ▪ Self-employment in Egypt ▪ Self-employment abroad ▪ Seek post-graduate degree in Egypt ▪ Seek post-graduate degree abroad ▪ Not decided yet ▪ Other (specify): ____________________________________ 15. If not enrolled in university, do you plan to attend university? Yes/No 16. On a scale of 1 to 5, where 1 is the least and 5 is the most, to what extent has the STEM school prepared you for university? (scale from 1 to 5 and add N/A column) 17. On a scale of 1 to 5, where 1 is the least and 5 is the most, how much does each contribute to your preparation of your university / career (a matrix with scale for each item of the following and add N/A column) ▪ Student-centered method ▪ Curriculum ▪ Teachers ▪ Infrastructure (laboratory, computer facilities, library) ▪ Extracurricular activities ▪ National/International Competitions ▪ Capstone project ▪ English ▪ Problem solving, team work and creative thinking 18. Of the following components, choose the 3 that are most in need of improvement? (tick box, 94 don't allow for more than three) ▪ Student-centered method ▪ Curriculum ▪ Teachers ▪ Infrastructure (laboratory, computer facilities, library) ▪ Extracurricular activities ▪ Administration ▪ Capstone projects 19. Would you recommend the STEM schools to others? Y/N 20. Are you connected to current STEM students? Y/N 21. Are you connected to other STEM graduates? Y/N 95 SURVEY FOR CURRENTLY ENROLLED STEM STUDENTS EVALUATION OF THE EDUCATION CONSORTIUM FOR THE ADVANCEMENT OF STEM IN EGYPT This survey is being conducted for the evaluation of the Egypt STEM Schools Project (ESSP), implemented by World Learning. You have been asked to participate in this survey because you are a part of the STEM school community in Egypt. Welcome to the survey and thank you for you valued contributions to this evaluation. The evaluation team has been endorsed by the Ministry of Education to review, analyze and evaluate the USAID￾funded STEM school activities. The USAID Office of Education and Health is funding three three-person field teams specialized in evaluation to conduct interviews, group discussions, classroom observations, and surveys in the STEM schools. The evaluation team also will interview officials in the MOE STEM Unit and members of the World Learning project team. Your participation in this survey is very important as your responses, together with those of other students and teachers, will inform the findings, conclusions and recommendations of the evaluation for USAID and the Ministry of Education, to strengthen the STEM school programs in Egypt. The questions should take about 15 minutes to complete. All responses are completely anonymous. No individual responses will be reported and the results of this study will be used for research purposes only. Your participation is entirely voluntary, but your support is important to ensure that all points of view are reflected in the evaluation. Thank you for your valued contribution to this important initiative. A. Basic Demographic Information (Please check the appropriate boxes) 1. School Name: ……………………………………………. 2. Gender: 1. Male. 2. Female. 3. Home Governorate: __________________________________________ 4. Place of Residence 1. City 2. Village 5. Current Grade Level: 1. Grade 10 2. Grade 11 3. Grade 12 6. If Grade 12, what is your Academic Discipline? 1. Math 2. Science 7. Check the type of preparatory school you attended: 1. Public School 2. Experimental School 3. Private Arabic School 4. Private English School 96 B. Outreach and Admission 1. How did you learn about the STEM schools? (Check ONLY the MAIN One) 1. Ministry website 2. Newspaper/TV 3. Social media 4. My preparatory school (announcement, teacher, social worker, other personnel) 5. Word of mouth (e.g., family, friend, peers) 6. Older STEM student/graduate 7. Other (specify):______________________ 2. Why did you want to enroll in the STEM school? (Check ONLY the MAIN ONE) 1. Parents suggestions/recommendations 2. I wanted to try something new 3. Interest in studying STEM subjects 4. Better education 5. Better career qualifications 10. Other C. School Infrastructure and Facilities 1. On a scale of 1 to 5, where 1 is the least satisfied, and 5 is the most satisfied, how satisfied are you with the following: 1 Least 2 3 4 5 Most Not Applicable 1) Laboratory Facilities 2) Laboratory Supplies 3) FabLab 4) Internet Connection 5) Laptops 6) Dorms 7) Classrooms 8) Library D. Teachers 1. Thinking of your teachers for THIS year, on a scale of 1 to 5, where 1 is the least satisfied and 5 is the most satisfied, how satisfied are you with the following: 1. STEM teachers: Please write your satisfaction score in each of the following cells. Math, Science and Technology Teachers Level of Satisfaction from 1 to 5 English Teachers Level of Satisfaction from 1 to 5 1) Teachers’ knowledge of the subject 97 2) Teachers’ English proficiency 3) Use of inquiry-based teaching methods 4) Engaging students in classroom discussions 5) Encourage hands-on activities 6) Feedback from the teacher 7) Availability outside the classroom 2. Non-STEM teachers (Arabic, religion, civics) On a scale of 1 to 5, where 1 is the least satisfied and 5 is the most satisfied, how satisfied are you with the following: Arabic, religion, civics teachers Level of Satisfaction from 1 to 5 1) Teachers’ knowledge of the subject 2) Use of inquiry-based teaching methods 3) Engaging students in classroom discussions 4) Encourage hands-on activities 5) Feedback from the teacher 6) Availability outside the classroom E. STEM Curriculum 1. On a scale of 1 to 5, where 1 is the least satisfied and 5 is the most satisfied, thinking about the STEM curriculum in your school, how satisfied are you with the following: (Please write your satisfaction score in each of the following cells) I Don’t Know 1) Problem solving over memorizing 2) Linking theory to applications 3) Promoting my thinking as a scientist 4) Relevance to the community 5) Textbooks availability 6) Textbooks enhanced my learning F. E-STEM Course 1. On a scale of 1 to 5, where 1 is the least satisfied, and 5 is the most satisfied, how satisfied are you with in following: 98 (Please write your satisfaction score in each of the following cells). 1) E-STEM course 2) Studying in English 3) Writing my assignment in English 4) Interacting in classroom in English G. Capstone Project 1. On a scale of 1 to 5, where 5 is the highest and 1 is the lowest, how would you rate your capstone experience regarding the following: (Please write your satisfaction score in each of the following cells) 1) Developing my understanding of fundamental concepts 2) Applications of these concepts 3) Peer learning and group work 4) Relevance to the community 5) Percentage assigned to the capstone project out of the total score. H. Assessment Methods: (Please check the appropriate box) 1) Were assessment methods announced at the beginning of the school year? 1. Yes 2. No 2) Were assessment methods clear? 1. Yes 2. No 3) Were assessment methods fair? 1. Yes 2. No 4) Have you used the PARLO assessment software? 1. Yes 2. No I. Would you recommend the STEM school to others? 1. Yes 2. No 99 TEACHER SURVEY EVALUATION OF THE EDUCATION CONSORTIUM FOR THE ADVANCEMENT OF STEM IN EGYPT This survey is being conducted for the evaluation of the Egypt STEM Schools Project (ESSP), implemented by World Learning. You have been asked to participate in this survey because you are a part of the STEM school community in Egypt. Welcome to the survey and thank you for you valued contributions to this evaluation. The evaluation team has been endorsed by the Ministry of Education to review, analyze and evaluate the USAID￾funded STEM school activities. The USAID Office of Education and Health is funding three three-person field teams specialized in evaluation to conduct interviews, group discussions, classroom observations, and surveys in the STEM schools. The evaluation team also will interview officials in the MOE STEM Unit and members of the World Learning project team. Your participation in this survey is very important as your responses, together with those of other students and teachers, will inform the findings, conclusions and recommendations of the evaluation for USAID and the Ministry of Education, to strengthen the STEM school programs in Egypt. The questions should take about 15 minutes to complete. All responses are completely anonymous. No individual responses will be reported and the results of this study will be used for research purposes only. Your participation is entirely voluntary, but your support is important to ensure that all points of view are reflected in the evaluation. Thank you for your valued contribution to this important initiative. 1. Name of School: ________________________________________________ 2. Gender: 1) Male 2) Female 3. Governorate of home residence: __________________________________ 4. What is the highest degree you completed? (Please mark the appropriate box) 1) B.Sc. 2) BA 3) Diploma 4) M.Sc. 5) Ph.D. 5. What was your field of study for your highest degree? …………………………………………………………………………………………….. 6. What is (are) the subject(s) you teach in the STEM school: 1)……………………………………….. 2)……………………………………….. 7. Did you participate in any STEM-related training activity, workshop, or diploma program before joining the STEM school? 1) Yes 2) No 100 7a If Yes, where? 1) In Egypt 2) Abroad 3) Both 8. Number of years of teaching before entering STEM school? (Please mark the appropriate box) 1. None, STEM school is my first teaching experience 2. 1 to 5 years 3. 6 to 10 years 4. More than 10 years IF THIS STEM SCHOOL IS YOUR FIRST TEACHING EXPERIENCE, SKIP to Question 11. 9. Which level did you teach? (Please mark the ALL that apply) 1. Preparatory 2. Secondary 3. University 10. If you taught in a school, what type of school was it? (Please mark ALL that apply) 1. Public School 2. Experimental School 3. Private Arabic School 4. Private English School . SKIP to Question 12 11. How many years have you been teaching in a STEM school? (Please mark the appropriate box) 1. Less than one year 2. One to three years 3- Four to six years 12. What training have you received since entering the STEM school? Theme of Training Method of delivery Number of days of training From scale 1 to 5, where 1 is the least and 5 is the most, how much did you benefit from it in your teaching? On-line In￾Person Both 101 English Capstone Project Lab/technical Training Student Assessment Teaching Strategies/Pedagog y Mentoring and Coaching Other (Specify) 13. Are you familiar with the STEM Teacher Performance Standards? 1) Yes 2) No 14a If yes, are these standards used in your performance assessment? 1) Yes 2) No 15. Which of the following tools have been used in assessment and feedback for you as a teacher in the STEM school. (Please mark all the appropriate boxes) Name of the tool How frequent? Do you get a feedback? 1) Classroom observation 0) Not used 1) Daily 2) Weekly 3) Monthly 4) By term 5) Annually 1) Yes 2) No 2) Annual Teacher Survey 0) Not used 2) Annually 1) Yes 2) No 3) Assessment by Supervisor 0) Not used 1) Daily 2) Weekly 3) Monthly 4) By term 1) Yes 2) No 102 5) Annually 16. Are there any additional teacher assessment tools and strategies that have been used? 1) Yes 2) No 16a If yes, what are they? __________________________________________ ___________________________________________________________________ 17. What tools do you use to assess the students? (Please mark ALL the boxes that apply) Name of Tool How frequently? Do you give feedback to the student? 1. Concept Inventory Test 0) Do not use 1) Daily 2) Weekly 3) Monthly 4) By term 5) Annually 1) Yes 2) No 2. Written Exams 0) Do not use 1) Daily 2) Weekly 3) Monthly 4) By term 5) Annually 1) Yes 2) No 3. Quizzes 0) Do not use 1) Daily 2) Weekly 3) Monthly 1) Yes 2) No 103 4) By term 5) Annually 4. Face-to-Face with Students 0) Do not use 1) Daily 2) Weekly 3) Monthly 4) By term 5) Annually 1) Yes 2) No 5. PARLO 0) Do not use 1) Daily 2) Weekly 3) Monthly 4) By School Term 5) Annually 1) Yes 2) No 104 EGYPT STEM SCHOOLS STEM TEACHER DATA COLLECTION FORM Data: Teacher Information Data Collector Name: Teacher Code: Observation Date: Classroom Laboratory School: Subject: Total Number of Students in Class: Total Number of Students Present: Observation Start Time: Observation End Time: Teacher Used English in Instruction Lesson Plan Available: Yes Some No Yes No IMPORTANT INSTRUCTIONS: ● Attach all observation/justification notes, lesson plans and other relevant materials to this document before returning it. ● Include the TEACHER CODE on all pages. ● Obtain a copy of the teacher’s lesson plan for the lesson or laboratory session observed before the observation. ● Meet with the teacher before the observation and arrange with him/her the time and manner of your entry. ● Sit at the very end of the classroom. ● If you were not provided with a lesson plan prior to the observation, it should be noted in the appropriate space at the top of the first page of the tool and Items 1-5 should be left blank. ● Use the blank pages to write down your notes from the observation. ● Do NOT interfere in the proceedings of the session for any reason. ● Avoid making any remarks or using body language to give any indication about the session. ● Thank the teacher for allowing you to observe and remind them that this is NOT a part of the evaluation process. ● Observation should be for an entire session. ● Complete the observation form as soon as you finish the observation. ● Continuously review the form and rubrics before you conduct the observation. ● Do NOT share your observation forms with any one at the school. 105 Data Collection Tool Use your observations to assign ONLY ONE level for each indicator. If you did not observe an indicator you may leave it blank. Use the following scale. 1 — Not Proficient 2 — Not Yet Proficient 3 — Proficient 4 — Highly Proficient Indicator Level Planning 1 2 3 4 1. Lesson objectives are challenging, appropriate and connected to the big ideas of the discipline. 2. Lesson activities are matched to learning outcomes. 3. Lesson objectives and activities demonstrate accurate understanding of the concepts being taught. 4. Lesson activities incorporate different resources beyond the resources textbook. Instruction 1 2 3 4 5. Demonstrates accurate understanding of important concepts in the discipline and provides clear explanations or responses to students. 6. Uses strategies that challenge students to probe for higher order understanding. 7. Clearly communicates the goals of the lesson. 8. Actively connects current information to previous knowledge. 9. Provides opportunities for students to demonstrate understanding. 10. Provides constructive feedback. Classroom Engagement 1 2 3 4 11. Actively ensures the participation of all students in learning activities. 12. Uses different strategies to enable student discussion. 13. Demonstrates knowledge of the academic and developmental needs of his/her students. 14. Acts as a facilitator. 15. Provides instructions or guidelines that include rubrics or other means to demonstrate expectations and/or outcomes. Laboratory (if a lab-based session) 1 2 3 4 16. Models safety protocols and has created a safe laboratory environment 17. Provides lab activity that is academically and developmentally appropriate. Student Behavior (classroom/lab) 1 2 3 4 18. Students are actively and appropriately engaged in the activity. 19. Students use the scientific process in solving problems. 20. Students apply knowledge to a new situation. 21. Students can communicate the goal(s) of the activity. 22. Students demonstrate clear understanding of their work. 23. There is a high proportion of appropriate student-to-student talk. 106 ANNEX IV: DATA COLLECTION SCHEDULE Egypt STEM Schools Project (ESSP) Field Data Collection Schedule Dates Team A Team B Team C Day Count March 21, 2017 Cairo/Maadi Ismailia 6th of October Day 1 March 22, 2017 Cairo/Maadi Ismailia 6th of October Day 2 March 23, 2017 Cairo/Maadi Ismailia 6th of October Day 3 March 26, 2017 PAT (with member from Team C) Stop Order; drawing on select Team B and Team C members to perform limited tasks on an as needed basis Day 4 March 27, 2017 World Learning + 21 PSTEM+ MOE Day 5 March 28 2017 CCIMD + NCER Day 6 March 29, 2017 NCEEE+MOE + USAID + mid-evaluation oral debrief Day 7 March 30, 2017 World Learning Day 8 April 02, 2017 World Learning + MOE + TFI (with member from team B) Day 9 April 04, 2017 Luxor Hurghada Assiut Day 10 April 05, 2017 Luxor Hurghada Assiut Day 11 April 06, 2017 Luxor Hurghada Assiut Day 12 April 09, 2017 Alexandria Dakahleya Kafr El Shiekh Day 13 April 10, 2017 Alexandria Dakahleya Kafr El Shiekh Day 14 April 11, 2017 Alexandria Dakahleya Kafr El Shiekh Day 15 April 19, 2017 MOE/TDC Members of Teams B and C Day 16 April 25, 2017 MSI + TIES Members of Team A Day 17 107 Illustrative STEM School Site Visit Agenda, Evaluation Team Time Key person Venue From To 8:30 9:00 Welcome, Reviewing Schedule & Touring the School Principal Office 9:00 10:00 IT Specialist IT Room 10:00 11:00 Group Discussion 1 Students of G11 Meeting Room 11:00 11:15 Break Principal Office 11:15 12:15 Social Workers Social Workers Office 12:15 1:15 Discussion Group / Teachers STEM Subjects Meeting Room 1:15 1:30 Break Principal Office 1:30 2:30 Group Discussion 2 Students of G11 Meeting Room Monday 10/4/2017 8:30 8:45 Welcome & Reviewing Schedule Principal Office 8:45 10:45 Deputy Deputy Office 10:45 11:00 Break Principal Office 11:00 12:00 Discussion Group / Teachers Non-STEM Subjects Meeting Room 12:00 1:00 Capstone Leaders Meeting Room 1:00 1:15 Break Principal Office 1:15 2:15 BOT Meeting Room 2:15 3:15 Group Discussion 3 Students of G10 Meeting Room Tuesday 11/4/2017 8:30 8:45 Welcome & Reviewing Schedule Principal Office 8:45 9:45 Librarian Library 9:45 10:45 Local STEM Unite Meeting Room 10:45 11:00 Break Principal Office 11:00 12:00 Group Discussion 4 Students of G10 Meeting Room 12:00 1:00 Fab Lab Manager Fab Lab 1:00 1:15 Break Principal Office 1:15 3:15 Principal Principal Office 108 ANNEX V: DATA TABLES AND GRAPHS A. SECONDARY QUANTITATIVE/QUALITATIVE DATA Table A.1: Enrolled Students, by Year, School, and Gender Academic Year Gender October Maadi Alexandria Assuit Luxor Red Sea Kafr El Sheikh Dakahliya Ismailia Total 11-12 Male 147 147 Female Total 147 147 12-13 Male 243 243 Female 119 119 Total 486 238 724 13-14 Male 338 338 Female 209 209 Total 338 209 547 14-15 Male 357 357 Female 300 300 Total 357 300 657 15-16 Male 406 73 33 24 70 64 75 72 817 Female 299 70 12 23 34 72 71 70 651 Total 406 299 143 45 47 104 136 146 142 1,468 16-17 Male 404 144 90 65 118 98 128 146 1,193 Female 299 125 60 38 65 104 129 139 959 Total 404 299 269 150 103 183 202 257 285 2,152 Source: World Learning 109 Table A.2: Number of Graduated Students, by Year and School Academic Year 6th of October School (Male) Maadi School (Female) Total 2013/2014 85 85 2014/2015 92 91 183 2015/2016 121 75 196 2016/2017 116 64 180 (expected) Source: World Learning 110 Table A.3: Student Participation and Success in Local and International Competitions, by Year and School Year Gender Scholarships Number of Participants in Local Competitions Number of Awards/ Achievements in Local Competitions Number of Participants in International Competitions Number of Awards/ Achievements in Local Competitions Misr El Kheir STEP 2013/2014 6th October (Male) 9 3 3 Maadi (Female) 15 6 8 2 Total 9 18 9 8 2 2014/2015 6th October (Male) 4 28 9 25 5 Maadi (Female) 1 12 29 23 10 2 Total 5 12 57 32 35 7 2015/2016 6th October (Male) 2 2 2 0 Maadi (Female) 12 3 3 6 5 Total 12 5 5 8 5 2016/2017 6th October (Male) 7 7 2 0 Maadi (Female) 5 5 1 0 Others (Gender not specified) 26 26 4 1 Total 38 38 7 1 Notes: 1. These are numbers of participants. It may have been the case that students participated in one or more competitions. 2. Some of the local competitions represented the first stage before going on to an international one. 3. Some of the achievements are simply recognitions, not awards. 111 Table A.4: Retention and Loss of Teachers Trained by ESSP (in Numbers) Year of Training Trained and Still in STEM Schools Trained but Left STEM Schools 2012/2013 26 39 2013/2014 42 45 2014/2015 117 31 2015/2016 220 22 2016/2017 297 4 Table A.5: Student Dropout Rate, by Cohort and Gender Cohort Dropout Rates From Grade 10 to 11 From Grade 11 to 12 Males Females Males Females 2011/2012 27.2 2012/2013 10.3 9.2 20.6 2013/2014 3.1 6.9 24.6 15.7 2014/2015 1.4 9.6 4.7 20.2 2015/2016 1.3 2.5 14.7 38.5 Table A.6: Student Dropout Numbers, by Cohort and Gender Academic Year Males Male Dropout Rates* Females Female Dropout Rates* 2011/2012 Grade 10 147 Grade 11 2012/2013 Grade 10 136 119 Grade 11 107 27.2 2013/2014 Grade 10 131 101 Grade 11 122 10.3 108 9.2 Grade 12 85 20.6 2014/2015 Grade 10 138 115 Grade 11 127 3.1 94 6.9 Grade 12 92 24.6 91 15.7 2015/2016 Grade 10 149 120 Grade 11 136 1.4 104 9.6 Grade 12 121 4.7 75 20.2 2016/2017 Grade 10 141 118 Grade 11 147 1.3 117 2.5 Grade 12 116 14.7 64 38.5 Notes: *Bold numbers represent the percent change from grades 10 to 11. Italics identify the percent change from grades 11 to 12. 112 Table A.7: STEM School Teacher Population, by Subject and Gender (2016/17 Academic Year) TITLE GRAND TOTAL Female Male Total Biology Teacher 11 9 20 Geology Teacher 1 10 11 Chemistry Teacher 2 16 18 Physics Teacher 2 19 21 Mathematics Teacher 2 31 33 ALL STEM TEACHERS 18 85 103 Arabic Teacher 2 31 33 English Teacher 9 24 33 French Teacher 5 4 9 German Teacher 3 6 9 Computer Science Teacher 4 7 11 Social Studies Teacher 2 7 9 Home Economics 2 0 2 Art Teacher 7 3 10 Music Teacher 5 4 9 Physical Education Teacher 7 8 15 ALL NON STEM TEACHERS 46 94 140 TOTAL 64 179 243 113 Table A.8: Student Academic Achievement, by Gender and Grade (2016/17 Academic Year) Grade 10 Grade 11 Gender Total Gender Total Female (%) Male (%) Female (%) Male (%) F 0.0 0.0 0.0 0.0 0.2 0.1 C 0.0 0.2 0.1 0.0 0.8 0.4 C+ 0.6 2.6 1.7 1.4 2.5 2.0 B 17.0 23.1 20.3 13.1 17.6 15.5 B+ 54.8 46.8 50.4 54.6 39.5 46.3 A 27.5 26.8 27.1 30.9 38.1 34.8 A+ 0.0 0.5 0.3 0.0 1.4 0.7 TOTAL 100.0 100.0 100.0 100.0 100.0 100.0 Source: STEM School Secondary Data; Grade 10 N=1,033; Grade 11 N=939. 114 Table A.9: Currently Enrolled Students, by Gender, School, and Grade (2016/17 Academic Year) Grade Grade 10 Grade 11 GPA School Gender Total School Gender New Total Schools Maadi & 6th of October Female Male New Schools Maadi & 6th of October Female Male A+ 0.1 0.8 0 0.5 0.3 0.3 2.3 0 1.6 0.9 A 16.5 58.7 27.5 26.8 27.1 26.7 55.3 30.9 37.9 34.7 B+ 54.8 37.5 54.8 46.8 50.4 51.9 33 54.8 39.6 46.5 B 26.1 3.1 17 23.1 20.3 17.8 9.1 12.9 17.4 15.3 C+ 2.3 0 0.6 2.6 1.7 2.8 0 1.4 2.5 2 C 0.1 0 0 0.2 0.1 0.6 0 0 0.8 0.4 F 0 0 0 0 0 0 0.4 0 0.2 0.1 Total 100 100 100 100 100 100 100 100 100 100 No. 774 259 465 568 1,033 675 264 427 512 939 Source: STEM School secondary data. Note: New Schools are the other 7 schools. 115 Figure A.1: Retention and Loss of Teachers Trained by ESSP (Numbers) Figure A.2: Student Dropout Rates, by Cohort and Gender 116 Figure A.3: STEM School Teacher Population, by Subject and Gender (2016/17 Academic Year) Source: STEM School Secondary Data 117 Figure A.4: Currently Enrolled Students, by Grade and Gender (2016/17 Academic Year) Source: STEM School Secondary Data; Grade 10, N=1,033; and Grade 11, N=939. Figure A.5: Grade 10, by School, Grade, and Gender (2016/17 Academic Year) Source: STEM School Secondary data Note: Grade 10 (N=1,033). New Schools are the other six schools. 17.0% 23.1% 20.3% 54.8% 46.8% 50.4% 27.5% 26.8% 27.1% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Female Male Total Grade 10 13.1% 17.6% 15.5% 54.6% 39.5% 46.3% 30.9% 38.1% 34.8% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Female Male Total Grade 11 A+ A B+ B C+ C F 16.5 58.7 54.8 37.5 26.1 3.1 2.3 0 0% 20% 40% 60% 80% 100% New schools Maadi&6October A+ A B+ B C+ C 0% 20% 40% 60% 80% 100% Female Male Total A+ A B+ B C+ C 118 Figure A.6: Grade 11, by School, Grade, and Gender (2016/17 Academic Year) Source: STEM School Secondary Data; Grade 10 Grade 11 N=939. Note: New Schools are the other seven schools 0.3 2.3 26.7 55.3 51.9 33 17.8 9.1 0.6 0 0% 20% 40% 60% 80% 100% New schools Maadi&6October A+ A B+ B C+ C F 30.9 37.9 34.7 54.8 39.6 46.5 12.9 17.4 15.3 1.4 2.5 2 0% 20% 40% 60% 80% 100% Female Male Total A+ A B+ B C+ C F 119 B. PRIMARY QUANTITATIVE/QUALITATIVE DATA (ONLINE SURVEYS) Table B.1: Distribution of Enrolled Students, by Home Governorate Governorate of Home Residence Frequency Percent Alexandria 99 7.01 Aswan 17 1.20 Asyut 62 4.39 Beheira 34 2.41 Beni Suef 20 1.42 Cairo 257 18.20 Dakahlia 59 4.18 Damietta 34 2.41 Faiyum 64 4.53 Gharbia 51 3.61 Giza 258 18.27 Ismailia 23 1.63 Kafr El Sheikh 68 4.82 Luxor 17 1.20 Matruh 3 0.21 Minya 22 1.56 Monufia 22 1.56 North Sinai 10 0.71 Port Said 3 0.21 Qalyubia 65 4.60 Qena 85 6.02 Red Sea 42 2.97 Sharqia 59 4.18 Sohag 22 1.56 Suez 16 1.13 Total 1,412 100.00 120 Table B.2: Students’ Preparatory Schools, by Gender and Location (%) Public School Experimental School Private Arabic School Private English School Total % Number of Students Gender Male 43.6 24.9 19.8 11.7 100 667 Female 46.8 20.3 24.1 8.7 100 675 Total 45.2 22.6 22.0 10.2 100 1,343 STEM School Location Luxor 63.4 14.6 13.4 8.5 100 82 Hurghada 63.0 20.0 11.9 5.2 100 135 Assiut 48.1 21.5 22.8 7.6 100 79 Alexandria 40.7 30.3 15.9 13.1 100 145 Ismaeleya 29.8 25.7 31.7 12.8 100 218 Kafr El￾Sheikh 71.4 15.9 4.0 8.7 100 126 Gamasa 57.3 21.3 12.8 8.5 100 164 6th October 30.4 20.9 31.0 17.7 100 158 Maadi 31.9 25.4 35.1 7.7 100 248 Total 45.0 22.7 22.1 10.3 100 1,355 Source: Enrolled students’ online survey. 121 Table B.3: Percent of Students Recommending STEM Schools to Others, by Gender, School Location, and Grade Yes No Total Number of Students Gender Male 74.9 25.1 100 395 Female 82.4 17.6 100 409 Total 78.7 21.3 100 804 STEM School Location Luxor 78.0 22.0 100 50 Hurghada 71.6 28.4 100 74 Assiut 89.2 10.8 100 37 Alexandria 85.4 14.6 100 82 Ismaeleya 79.3 20.7 100 150 Kafr El-Sheikh 82.1 17.9 100 78 Gamasa 77.0 23.0 100 100 6th October 68.5 31.5 100 92 Maadi 81.2 18.8 100 149 Total 78.7 21.3 100 812 Current Grade Level Grade 10 84.4 15.6 100 435 Grade 11 72.0 28.0 100 328 Grade 12 72.9 27.1 100 48 Total 78.7 21.3 100 811 Source: Enrolled students’ online survey. 122 Table B.4: Satisfaction with STEM Curriculum, Currently Enrolled Students (%) Don't Know 1 Least Satisfied 2 3 4 5 Most Satisfied Total No. of Students Problem solving over memorizing 4.9 10.2 16.7 34.3 24.4 9.4 100 921 Linking theory to application 2.6 7.6 16.7 29.0 30.7 13.4 100 922 Promoting my thinking as a scientist 2.1 8.7 16.4 29.5 27.6 15.7 100 916 Relevance to the community 3.0 8.2 16.6 30.3 26.7 15.3 100 911 Textbooks availability 3.4 10.4 15.3 26.6 27.5 16.9 100 917 Textbook Enhanced learning 4.7 10.1 13.3 28.2 29.6 14.1 100 919 Source: Enrolled students' online survey Note: The question was phrased as follows: On a scale of 1 to 5, where 1 is the least satisfied and 5 is the most satisfied, thinking about the STEM curriculum in your school, how satisfied are you with the following:? Table B.5: Satisfaction with Capstone, Currently Enrolled Students (%) 1 Least Satisfied 2 3 4 5 Most Satisfied Total No. of students Developing my understanding of fundamental concepts 5.9 9.8 31.2 36.5 16.6 100 844 Application of these concepts 7.5 14.6 31.3 31.7 14.9 100 842 Peer learning and group work 3.6 7.0 23.7 38.4 27.2 100 838 Relevance to the community 6.5 10.9 28.3 33.2 21.2 100 834 Percentage assigned to Capstone 7.3 9.6 28.7 31.8 22.6 100 836 Assessment of the capstone 10.0 13.3 30.2 30.0 16.5 100 842 Source: Enrolled students' online survey Note: The question was phrased as follows: On a scale of 1 to 5, where 1 is the least satisfied and 5 is the most satisfied, how would you rate your CAPSTONE experience regarding the following:? 123 Table B.6: Percent Satisfaction with School Facilities, Currently Enrolled Students 1 Least Satisfied 2 3 4 5 Most Satisfied Not Applicable Total Number of students Laboratory Facilities 6.6 15.2 35.7 28.6 12.9 1.0 100 1116 Laboratory Supplies 8.0 17.6 32.5 28.5 12.5 0.9 100 1107 Fab lab 16.5 16.7 22.2 24.9 16.1 3.7 100 1105 Internet connections 48.0 24.0 14.1 7.4 3.1 3.4 100 1111 Laptops 5.5 12.4 21.8 30.3 29.3 0.6 100 1119 Dorms 13.1 24.6 30.4 22.6 6.9 2.3 100 1097 Classrooms 4.0 14.9 34.1 32.7 14.1 0.2 100 1110 Library 7.7 15.8 24.5 27.6 23.3 1.1 100 1120 Source: Enrolled students’ online survey Note: The question was phrased as follows: On a scale of 1 to 5, where 1 is the least satisfied, and 5 is the most satisfied, how satisfied are you with the following:? 124 Table B.7: Percent Satisfaction with STEM Teachers, Currently Enrolled Students 1 Least Satisfied 2 3 4 5 Most Satisfied Total Number of Students Teacher Knowledge of the Subject 5 14.9 32.3 35.6 12.2 100 1,002 Teacher's English Proficiency 13 27.8 30.5 20.5 8.2 100 1,000 Use of Project Based teaching methods 14.8 27.6 33.8 17.2 6.6 100 986 Engaging Student in Classroom discussions 8.1 18.2 31.1 28.5 14.1 100 993 Encourage hands-on activities 14.6 23.9 31.6 18.6 11.3 100 989 Feedback from teacher 19 27.5 28.5 17.7 7.3 100 992 Availability outside the Classroom 16.8 20.0 24.6 20.9 17.7 100 989 Source: The enrolled students' online survey. Note: The question was phrased as follows: Thinking of your teachers for THIS year, on a scale of 1 to 5, where 1 is the least satisfied and 5 is the most satisfied, how satisfied are you with the following: STEM teachers - (Math, Science and Technology Teachers)? Table B.8: Percent Satisfaction with English Teachers, Currently Enrolled Students (%) I Least Satisfied 2 3 4 5 Most Satisfied Total Number of Students Teacher Knowledge of the Subject 7.1 9.2 19.1 32.3 32.4 100 989 Teacher's English Proficiency 7.8 11.5 18.9 27.9 33.9 100 989 Use of Project Based teaching methods 13.3 17.4 26.3 21.4 21.6 100 987 Engaging Student in Classroom discussions 8.1 12.0 19.8 26.7 33.4 100 982 Encourage hands-on activities 9.6 16.6 22.2 21.6 30.1 100 984 Feedback from the teacher 14.6 16.4 21.7 23.0 24.2 100 85 Availability outside the Classroom 15.3 15.5 19.0 23.3 26.9 100 981 Source: Enrolled students' online survey Note: The question was phrased as follows: Thinking of your teachers for THIS year, on a scale of 1 to 5, where 1 is the least satisfied and 5 is the most satisfied, how satisfied are you with the following: (English Teachers)? 125 Table B.9: Percent Satisfaction with Non-STEM Teachers, Currently Enrolled Students (%) 1 Least Satisfied 2 3 4 5 Most Satisfied Total Number of Students Teachers’ knowledge of the subject 4.5 7.0 17.2 30.2 41.2 100 988 Use of project-based teaching methods 16.3 21.2 24.5 21.9 16.0 100 986 Engaging students in classroom discussions 8.9 12.5 21.7 28.1 28.7 100 985 Encourage hands-on activities 15.5 20.5 23.2 21.1 19.7 100 983 Feedback from the teacher 14.6 18.8 22.5 22.4 21.7 100 978 Availability outside the classroom 15.0 14.8 18.4 23.4 28.4 100 982 Source: Enrolled students’ online survey. Note: Thinking of your teachers for THIS year, on a scale of 1 to 5, where 1 is the least satisfied and 5 is the most satisfied, how satisfied are you with the following: Non-STEM teachers - (Arabic, Religion, Social Science)? Table B.10: What are your plans after graduation from university? Graduated Students (n=93) Future Plans % Number Seek employment in Egypt 2.2 2 Seek employment abroad 19.4 18 Self-employment in Egypt 2.2 2 Self-employment abroad 5.4 5 Seek post-graduate degree in Egypt 6.5 6 Not decided yet 15.1 14 Source: Online survey, Graduated students 126 Table B.11: Teachers’ Satisfaction with Teaching in STEM Schools (%) 1 Least Satisfied 2 3 4 5 Most Satisfied Not Applicable Total Number of Students English 2.8 9.2 25.4 21.8 33.8 7.0 100 142 Capstone Project 2.0 7.2 22.9 31.4 35.3 1.3 100 153 Lab/technical Training 5.3 8.3 18.2 22.0 30.3 15.9 100 132 Student Assessment 1.3 4.0 17.9 42.4 33.1 1.3 100 151 Teaching Strategies/Pedagogy 1.3 1.3 15.6 37.0 42.2 2.6 100 154 Mentoring and Coaching 1.5 12.2 19.1 20.6 32.1 14.5 100 131 Design Studio 8.8 5.6 17.6 14.4 19.2 34.4 100 125 Item Writing 4.0 7.2 15.2 21.6 20.0 32.0 100 125 Table B.12: Graduate Online Survey: Source of Financing for University among STEM School Graduates Are you currently enrolled in university? Percent Number Yes 91.4 96 No 8.6 9 TOTAL 100.0 105 No response 1 For those currently enrolled in university, what is the funding source? Self-financed 34.8 31 Scholarship 50.6 45 Partial Scholarship 14.6 13 TOTAL 100.0 89 No response (7) 127 Figure B.1: Extent to which STEM Schools have prepared Graduated Students for University, % (n=92) Source: Online survey, Graduated students Note: The question was phrased as follows: On a scale of 1 to 5, where 1 is the least and 5 is the most, to what extent has the STEM school prepared you for University? Figure B.2: What are your plans after graduation from university? (Graduated Students, n=93) Source: Online survey, Graduated students 128 Figure B.3: Teacher Satisfaction with Training Topics (n=207) Source: Online Teacher Survey 129 Figure B.4: Percent Satisfaction with School Facilities, Currently Enrolled Students (n=207) 130 ANNEX VI: BIBLIOGRAPHY CAPMAS. Statistical Yearbook 2016. Cairo, Egypt: CAPMAS, 2016. El Farargy, Hamada Ahmed. Investigating Project-Based Learning in a STEM School in Egypt: A Case Study. M.A. Thesis, The American University in Cairo, International and Comparative Education, 2016. Ministry of Education (MOE), Egypt. The National Education Indicators (NEIs) for Egypt: Technical Guide. Cairo, Egypt: MOE, January 2009. _____. National Strategic Plan for Pre-University Education Reform in Egypt, 2007/8-2011/12. Cairo, Egypt: MOE, 2007. The QED Group. Technical Proposal: Final Performance Evaluation on the Education Consortium for the Advancement of STEM (ECASE); also referred to as Egypt STEM Schools Project (ESSP) (February 2017). Professional Academy for Teachers (PAT). Memorandum from Dr. Ramadan Mohamed Ramadan, Director of PAT, to HE Minister of Education and President of Board of Directors of PAT, addressing the selection and training of teachers of STEM Schools, 14 November 2013. Rao, D. B. Women as Educators. New Delhi: Discovery Publishers Pvt Ltd, June, 2010. Annual Implementation Plans World Learning. Education Consortium for the Advancement of STEM in Egypt (ECASE), Updated Performance Monitoring Plan (PMP), 2014-2016. _____. Education Consortium for the Advancement of STEM in Egypt (ECASE), Year 5 Annual Implementation Plan Proposed Extension (September 2016). _____. Education Consortium for the Advancement of STEM in Egypt (ECASE): Year 4 Annual Implementation Plan (October 2015-August 2016). _____. Education Consortium for the Advancement of STEM in Egypt (ECASE), Year 3 Annual Implementation Plan (October 2014-September 2015). _____. Education Consortium for the Advancement of STEM in Egypt (ECASE), Modified Year 2 Annual Implementation Plan (April 2014-September 2014). _____. Education Consortium for the Advancement of STEM in Egypt (ECASE): Year 2 Annual Implementation Plan (August 2013-September 2014). _____. Education Consortium for the Advancement of STEM in Egypt (ECASE): Modified Year 1 Annual Implementation Plan (2013). Cooperative Agreement and Modifications of Assistance USAID/Egypt, Cooperative Agreement No. AID-263-A-12-00005, Education Consortium for the Advancement of STEM in Egypt (ECASE), Under RF # 263-12-000020 Science, Technology, and 131 Mathematics Model Schools (STEM). Egypt STEM Schools Project (ESSP) Modification of Assistance No. 2, 2014. Egypt STEM Schools Project (ESSP) Modification of Assistance No. 6, 2014. Egypt STEM Schools Project (ESSP) Modification of Assistance No.1, 2014. Ministerial Decrees Ministerial Decree Number 136 of April 2, 2017 on amending the provisions of Ministerial Decree No. 172 of the year 2014, Regarding establishment of a Stem School Unit. Ministerial Decree Number 219, 2016 (Arabic) Ministerial Decree Number 313 of August 24, 2015 on the establishment of subcommittees for STEM schools at the subnational level. Ministerial Decree Number 172 of 2014 on the establishment of a STEM unit in the Ministry of Education. Ministerial Decree Number 308 of August 27, 2013 on high school graduation examination certificate, STEM schools of science and technology. Ministerial Decree Number 238, 2012 (Arabic). Ministerial Decree Number 382 of October 2, 2012 on admission, academic study, and examination in the STEM school. Ministerial Decree Number 290 of July 24, 2012 on the entire system of STEM schools. Ministerial Decree Number 202 of April 21, 2012 on high school graduation certificate STEM schools of science and technology. Ministerial Decree Number 369 of October 11, 2011 on secondary schools excelling in science and technology system. Project Reports Education Consortium for the Advancement of STEM in Egypt (ECASE). Annual Report, August 2012- September 2013. Washington, DC: USAID, October 31, 2013. _____. Annual Report 2014. _____. Annual Report, September 2016, Quarterly Progress Report, July-September 2016. Washington, DC: USAID, October 31, 2013. ECASE Quarterly Progress Report, July-September 2015. ECASE Quarterly Progress Report, July-September 2014. 132 Egypt STEM Schools Project (ESSP) Quarterly Progress Report, October-December 2016, World Learning, January 30, 2017. Egypt STEM Schools Project (ESSP) Annual Report, August 2012-September 2013. Washington, DC: USAID. Egypt STEM Schools Project (ESSP) Annual Report, September 2016, Quarterly Progress Report, July￾September 2016. Washington, DC: USAID, October 30, 2016. USAID/Egypt. Fact Sheet: Basic Educational, November 2015, USAID, Washington, DC. 133 ANNEX VII: ESSP END-OF-PROJECT PERFORMANCE EVALUATION TIMELINE, JANUARY-OCTOBER 2017 134 135 136 137 138 ANNEX VIII: DISCLOSURE OF ANY CONFLICTS OF INTEREST The Disclosure of any Conflict of Interest can be found on file with the COR. 139 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523