USAID/EGYPT EDUCATION PORTOLIO EVALUATION March 2010 The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. USAID/EGYPT EDUCATION PORTFOLIO EVALUATION FINAL REPORT April 2010 Report prepared for the United States Agency for International Development (USAID), Egypt Mission, Education Office, by the Aguirre Division of JBS International, Inc., in collaboration with the Associate Consultants for Educational Services (ACES), under the Global Evaluation and Monitoring II Contract EDH-E-23-08-00003-00. Its authors are Barbara Hunt, Richard Dye, Adriana Abreu-Combs, Maha El Said, Amal Nasrallah and Adel Saleh Fahmy. USAID/EGYPT EDUCATION PORTFOLIO EVALUATION April 2010 The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. i Table of Contents ACRONYMS ............................................................................................................................................................... iii ACKNOWLEDGMENTS ......................................................................................................................................... v EXECUTIVE SUMMARY .......................................................................................................................................... vi INTRODUCTION AND BACKGROUND ......................................................................................................... 1 Introduction ........................................................................................................................................................... 1 Background ............................................................................................................................................................. 1 Educational Context ....................................................................................................................................... 1 Development Environment in Egypt ........................................................................................................... 1 Program Description ............................................................................................................................................ 1 Methodological Summary .................................................................................................................................... 2 Outline of This Report ........................................................................................................................................ 2 FINDINGS: BY PROJECT ......................................................................................................................................... 2 New Schools Program (NSP) ............................................................................................................................. 2 1. Summary of Project ............................................................................................................................. 2 2. Principal Accomplishments and Challenges ................................................................................... 3 3. NSP: Efficiency, Effectiveness and Sustainability ............................................................................ 5 Education Reform Program EQUIP 1 (ERP1) ................................................................................................ 6 1. Summary of Project ............................................................................................................................. 6 2. Principal Accomplishments and Challenges ................................................................................... 7 3. ERP 1: Efficiency, Effectiveness and Sustainability ....................................................................... 10 Education Reform Program EQUIP 2 (ERP2) .............................................................................................. 11 1. Summary of Project ........................................................................................................................... 11 2. Principal Accomplishments and Challenges ................................................................................. 11 3. ERP2: Efficiency, Effectiveness and Sustainability ........................................................................ 14 School Team Excellence Awards Program (STEAP) .................................................................................. 14 1. Summary of Project ........................................................................................................................... 14 2. Principal Accomplishments and Challenges ................................................................................. 15 3. STEAP: Efficiency, Effectiveness and Sustainability ..................................................................... 16 Girls’ Improvement in Learning Outcomes (GILO) ................................................................................... 17 1. Summary of Project ........................................................................................................................... 17 2. Principal Accomplishments and Challenges ................................................................................. 18 3. GILO: Efficiency, Effectiveness and Sustainability ....................................................................... 20 Technology for Improved Learning Outcomes (TILO) ............................................................................. 21 1. Summary of Project ........................................................................................................................... 21 USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. ii 2. Principal Accomplishments and Challenges ................................................................................. 21 3. TILO: Efficiency, Effectiveness and Sustainability ........................................................................ 23 FINDINGS: CROSS-CUTTING AREAS .............................................................................................................. 24 Training Models ................................................................................................................................................... 24 1. Successful elements in Teacher Professional Development Programs ................................. 24 Efficiency, Effectiveness and Sustainability ..................................................................................................... 26 1. Efficiency .............................................................................................................................................. 26 2. Effectiveness ........................................................................................................................................ 27 3. Sustainability ........................................................................................................................................ 30 Project Design ..................................................................................................................................................... 31 CONCLUSIONS AND RECOMMENDATIONS ............................................................................................. 32 Conclusions .......................................................................................................................................................... 32 Main Findings by Project .............................................................................................................................. 32 Lessons Learned ............................................................................................................................................ 33 Best Practices ................................................................................................................................................. 34 Short-Term Recommendations ....................................................................................................................... 34 Future Strategic Program Conclusions .......................................................................................................... 35 Recommendations for Future USAID Program Strategy .......................................................................... 36 ANNEXES .................................................................................................................................................................. 40 ANNEX I: Egyptian Educational Context ...................................................................................................... 41 ANNEX II: Development Environment in Egypt ......................................................................................... 42 ANNEX III: Matrix of USAID Strategies by Project ................................................................................... 45 ANNEX IV: Components of Peer Observation Conferences ................................................................ 46 ANNEX V: Definitions of Efficiency, Effectiveness and Sustainability .................................................... 48 ANNEX VI: Scope of Work ............................................................................................................................. 49 ANNEX VII: Methodology ................................................................................................................................ 50 ANNEX VIII: List of Persons Interviewed .................................................................................................... 54 ANNEX IX: List of Schools Visited ................................................................................................................ 56 ANNEX X: List of Documents Reviewed..................................................................................................... 58 ANNEX XI: Instruments ................................................................................................................................... 61 USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. iii ACRONYMS ACES Associate Consultants for Educational Services AIR American Institute of Research ALD Administration and Leadership Development BOTs Boards of Trustees CAP Comprehensive Assessment Program CAPS Critical-thinking, Achievement, and Problem Solving Skills test CARE Cooperative for Assistance and Relief Everywhere CETs Community Education Teams CIDA Canadian International Development Agency COTR Contract Officer’s Technical Representative CP Community Participation CSEI Creative Science Education Initiative CTOs Cognizant Technical Officer CYM Community Youth Mapping DNP Democratic National Party DPG Donors Partners Group ECDE Early Childhood Development Education ECE Early Childhood Education EDC Education Development Center ECEP Early Childhood Education Program EGRA Early Grade Reading Assessment EMEP Education Monitoring and Evaluation Program EMIS Education Management Information System EQUIP Education Quality Improvement Program ERP1 Education Reform Program EQUIP1 ERP2 Education Reform Program EQUIP2 EU European Union FOEs Faculties of Education FOS Families of Schools GAEB General Authority for Educational Buildings GDA Global Development Alliance (USAID) GDP Gross Domestic Product GILO Girls’ Improved Learning Outcomes GOE Government of Egypt GS Girls’ Scholarships HED Higher Education for Development HEEP The Higher Education Enhancement Project HINI H1N1 Influenza (swine flu) ICT Information and Communication Technology IL Integrated Literacy LINC Learning Increases with New Connections (Vodafone Foundation Project) LRCs Learning Resource Centers LS Life Skills MAP Management Assessment Protocol MCIT Ministry of Communications and Information Technology MGS Multi-grade Schools MOE Ministry of Education MOHE Ministry of Higher Education MOSS Ministry of Social Sustainability MSAD Ministry of State for Administrative Development USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. iv MSLD Ministry of State for Local Development NAQAA National Authority for Quality Assurance and Accreditation NCEEE National Center for Examinations and Educational Evaluation NEIs National Education Indicators NES National Education Standards NGOs Non-Governmental Organizations NSP New Schools Program PAs Parent Associations PAT Professional Academy of Teachers PSC Personal Services Contract QAU Quality Assurance Unit SBR School-Based Reform SBTEU School-Based Training and Evaluation Unit SC School Construction SCOPE Standards-Based Classroom Observation Protocol SEEP Secondary Education Enhancement Project SG School Governance-BOT SIP School improvement plan SSA School self assessment STEAP School Team Excellence Awards Program STEPS Support to Egyptian Primary Schooling Project STW School to Work TCLs Teacher Learning Circles TDC Technology Development Centre TILO Technology for Improved Learning Outcomes TLC Teacher Learning Circles TOT Training of trainers TPD Teacher Professional Development TRC Teacher Resource Centers TSUs Training Support Units UNICEF The United Nations Children's Fund USAID United States Agency for International Development WB World Bank USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. v ACKNOWLEDGMENTS This study was conducted under the leadership of Barbara Hunt, project team leader, in collaboration with five other evaluation team members: Richard Dye, Adriana Abreu-Combs, Chief of Party (COP) of the Education Monitoring and Evaluation Program (EMEP), Maha El Said, Amal Nasrallah and Adel Saleh Fahmy. In addition to joining as a team member, EMEP’s COP provided guidance and support to the team as needed. We extend our gratitude and appreciation to evaluation team members for their invaluable work in this effort, which required an incredible commitment of time and effort. Their professionalism and unconditional dedication made this study possible. Logistical backstopping was provided by Gehane Mahmoud, to whom we would also like to extend our appreciation. The USAID Contract Officer’s Technical Representatives (COTRs) provided invaluable assistance. We would like to thank Marc Bonnenfant, EMEP’s COTR; and Hala El Serafy, Mary Ishak and Mary Kidwani, COTRs of the projects under evaluation. Their generous support and collaboration each step of the way were crucial and very much appreciated. We would also like to extend our gratitude to Evelyn Rodriguez-Perez, Education Office Director, whose leadership and vision we commend. We appreciate the opportunity she has offered us to contribute to advancing USAID/Egypt’s education goals in the country. Project COPs and their Cairo- and field-based teams were very supportive of the study. We would like to extend our gratitude to Jane Benbow, COP of the Education Reform Program (ERP); Andrea Bosch, COP of the Technology for Improved Learning Outcomes (TILO) project ; and Azza Shafik, acting COP of the Girls’ Improved Learning Outcomes (GILO) project. We would also like to thank Areeg Hegazi and Osama Abdou from the Cooperative for Assistance and Relief Everywhere (CARE) International, and former School Team Excellence Awards Program (STEAP) coordinators for going way out of their way to arrange for site visits in New School Program (NSP) and STEAP schools, both closed projects. We would like to acknowledge the tremendous collaboration the team received from project implementers, from sharing documents, to arranging for site visits, to accompanying the team to remote schools, and to being available to meet and share their honest opinions. The success of this effort is due largely to their unreserved support and genuine feedback. We extend our gratitude and sincerest appreciation to officials at the Ministry of Education (MOE) central, and MOE officials at both idara (local district education office) and mudireya (governorate) levels who were open to this effort and willing to contribute to it. We thank in particular the support we received from Dr. Reda Abou Serie, Deputy to the Minister of Education, as well as his staff. We also acknowledge staff of donors working on similar activities who were open to speak to the team. We thank the education office staff at the World Bank, the Canadian International Development Agency (CIDA), the European Union (EU), and the United Nations Children's Fund (UNICEF). Finally, we would like to thank school principals who extended their welcome and hospitality to the research teams, and school-based staff, teachers, parents, community members, and students who gave the team a warm welcome in each site visited and whose unreserved cooperation made this research possible. The team will never forget their faces and smiles, and their sincere appreciation for the support they are receiving from USAID through the projects. We were welcomed with beautiful songs by students and rounds of applauses by those whose lives are being positively impacted by these interventions. We wish them all much success. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. vi EXECUTIVE SUMMARY This evaluation of USAID Egypt’s Education Portfolio was undertaken by JBS/Aguirre International, in collaboration with the Cairo-based Associate Consultants for Educational Services (ACES). Six educational projects were reviewed, with the purpose of assessing their efficiency, effectiveness and sustainability across a number of impact areas, as a primary source of information for providing recommendations to USAID/Egypt for its next education strategy. Additional information on the activities of other donors, the current education development environment, and work underway by the Ministry of Education (MOE) and others to implement the National Strategic Plan for education was collected as background for the team’s strategic recommendations. USAID has made many positive changes in Egyptian education over the last 10 years, with an array of projects that have contributed to girls’ education, to the application of active learning methodologies in classrooms, to the use of technology in schools, and to educational reform at all levels of the MOE. Although there are always problems to be noted and improvements to be recommended, the team has been impressed by the positive reactions of students, teachers, and parents to the USAID projects. In addition, school administrators and idara (local education district office) and mudireya (governorate) personnel report improved practices as a result of USAID capacity-building efforts, and extraordinary strides have been made at systemic policy levels with the assistance of USAID technical support. Methodological Summary: Drawing on information gathered in the desk review and Meta-Evaluation (Task 1), this study employed a multi-method qualitative design consisting of semi-structured one-to-one and group interviews with key stakeholders at central, governorate and district levels (i.e., USAID staff, project implementers, MOE staff, and donors involved in similar initiatives), school site visits where interviews were conducted at the school level, and classroom observations. Data collection took place over a period of four weeks between February 16 and March 11, 2010. Site visits were conducted in 10 governorates benefiting from USAID interventions. Classroom observations were conducted in six schools for each project with a teacher training component (GILO, TILO, ERP1 and NSP). The team also reviewed project documentation. The study adopted a purposive sampling. The sample included 10 schools per project that intervened at the school level (this excludes ERP2) for a total of 50 schools across10 target governorates. In order to implement the study, interview guides and a classroom observation protocol were developed and can be found in the annexes. Limitations of this study include the constraints imposed by the H1N1 situation, and the high number of projects with different foci and timeframe to be evaluated under one effort. A detailed presentation of this study’s methodology and limitations is included in the annexes. Summary of Conclusions: The team, based on the total data collected, believes that the prospects for significant additional progress on implementation of Egypt’s National Strategic Plan for educational development over the next five years are excellent. It bases that conclusion on the continued high level Government of Egypt (GOE) support for reform, the degree to which that and widespread public support circumscribes any attempts to reverse or revise the plan in ways that would diminish it, and Egypt’s manifest need for the kind of educated, innovative, and skilled human resources needed if the country is to compete successfully in the global economy. The team’s investigations, moreover, indicate that the GOE and MOE’s openness, even eagerness, to receive continued USAID assistance, including involvement in strategic policy development, continues undiminished. Meanwhile, USAID support to the reform process is needed more than ever, especially as other donors in basic education, with the exception of the European Union (EU), USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. vii are cutting back. If USAID wishes to invest on the cutting edge and to capitalize on its large prior investments, both for their own sake and to ensure its legacy in this critical sector, supporting the National Strategic Plan for education is the place to be, with the first priority being completion of efforts to support systemic reforms that are critical to provide a favorable environment for improving educational quality and learning outcomes. The results of the team’s formative evaluation of six basic education pilot projects further confirm that a strong basis for moving beyond the pilot project stage to an emphasis on scaling up has been created. Details of lessons learned and best practices are presented in the body of the report. In the team’s view, the time is ripe for a vigorous but tightly focused new USAID education strategy. The team’s conclusions on what that future program strategy should be can be divided into two parts. The first part comprises needs and opportunities to continue providing technical assistance support in key policy and implementation capacity-building areas, where the promising work begun in recent years has not been and will not be completed during the life of current projects. These include: 1. Teachers: The Professional Academy for Teachers; development of in-service teacher training and professional development capacity; turning the teachers’ cadre into a force for quality improvement; and changes in teacher deployment policies and practices to reduce excessive turnover. In addition to these activities, begun under ERP2 and now being carried forward by ERP1, the team believes that teacher improvement efforts badly need to be supplemented by renewed USAID involvement in the area of teacher education. 2. Organizational and management reforms in the MOE, including implementation of proposed functional decentralization; creation of an education managers’ cadre, similar to the teachers’ cadre, to raise standards and increase professionalism; and review of human resource policies and practices in order to develop a plan to reduce overstaffing and open up opportunities for the Ministry to recruit and develop a new generation of education leaders for its work. 3. Improvement of learning outcomes, particularly in reading and writing, in the early primary grades; current learning difficulties in the early grades undermine the potential for further learning and contribute to the still too-high incidence of effective illiteracy in higher primary grades. USAID projects have addressed this issue in various ways, but a clear, strong national strategy and approach, with the power of the MOE behind it, is needed to convert the experience gained from the pilot projects. 4. A need for further review of current curriculum and assessment systems to create a better match between the curriculum and the learning capabilities of children of different ages and of differing needs and ability levels, and to simplify and focus the assessment system on critical learning measurements. 5. The need for an overall review of MOE educational technology programs and approaches to unify and focus them on improved learning, including their use in classroom teaching. The technical capacities and experience of TILO would be critical in this effort and a logical extension of the policy work that the project is currently doing in cooperation with the Ministry. The second part of the team’s conclusions goes beyond policy and structural reforms to the sphere of action. USAID’s extensive, past and current involvement in basic education has produced an array of tested best practices and lessons learned that is now sufficient to inform and guide a vigorous, but realistic, phased process by the MOE, assisted by USAID, to scale them up to the national level. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. viii Summary of Recommendations: That USAID’s future education strategy focuses on two core objectives: 1. Support the implementation of the Egyptian National Education Strategic Plan. 2. Maximize the return on 10 years of USAID investment in educational reform in Egypt. Three intermediate objectives are recommended. 1. Working with the MOE to fine-tune the enabling policy and operational framework for educational development. 2. Continuing to help the MOE to address systemic problems still inhibiting sustainable school-based reform. 3. Working with the MOE to develop a strategy for scaling-up of best practices of USAID-supported school-based reform models, conducted by replication teams operating at the idara level, utilizing, in part, training and other model assets produced by the pilot projects in schools and associated idaras. Further details of a proposed USAID project to support such an effort are included in the Conclusions and Recommendations section of the report. Before arriving at this scaling up support recommendation, the team considered a range of other possible action possibilities, but decided that, while all would be useful, they did not match the opportunities offered by the scaling up proposal. The options considered included: 1. Limiting future support to technical assistance and supporting additional small pilot projects that are needed to test proposed approaches to policy change and implementation capacity￾building, before final decisions are taken, similar to what is being done in the decentralization area. This could be described as the minimum option. 2. Basing a large scaling up strategy on an existing project, namely TILO. Although the team does not recommend this strategy to be the main one, it sees no conflict between USAID and MOE support for continuing TILO expansion and the larger, national program it is recommending. 3. Developing a new, separate project focusing on the early primary grades. 4. Some combination of the above. Based on the team’s review of the projects currently being implemented and the findings regarding each project, as detailed in Section II, the team makes the following short-term recommendations, presented below by project: GILO 1. An extension of the project to allow needed time to (i) achieve project objectives; (ii) continue the critical work started on reading and writing (for which coordination with ERP1 and its reading and writing program is recommended); and (iii) consolidate and expand current efforts on the decentralization front. 2. Strengthen the focus on gender, by emphasizing it more in training and directing attention to new needs beyond access, e.g. the gap between girls’ and boys’ attendance in upper grades. 3. Rethink and/or modify the plan to reach schools in the clusters to ensure that this activity is not neglected for lack of adequate follow-up and coaching. TILO 1. Establish more effective involvement with technology bodies (the Technology Development Centre (TDC) and the Computer Education Department at MOE). Although the project already has relationships with these entities, they would benefit from more in depth USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. ix collaboration, especially in the context of project expansion in which these entities can and should play a larger role to assure sustainability. 2. Train sufficient supervisors who can provide coaching and follow-up support to teachers in both current and expansion schools, to promote both effectiveness and sustainability. 3. Work in close cooperation with School Based Training and Evaluation Units (SBTEUs) to increase prospects of sustainability. 4. Reach out to the community as partners in harnessing the potential of technology in education while also carrying out existing plans to extend the benefits of such resources to them. 5. Consider putting in place a sustained coaching system to assure that teachers understand and apply what they have been taught, especially regarding integration of technology into the classroom. 6. Consider introducing more content-based training, increasing software options in early grade reading and writing, and providing more software in Arabic. ERP1 1. The team supports the proposed no-cost extension that will offer the project crucial additional time to (i) consolidate its work with the Professional Academy of Teachers (PAT); (ii) provide targeted technical assistance; and (iii) advance the institutionalization of the instruments Standards-Based Classroom Observation Protocol (SCOPE), Management Assessment Protocol (MAP), and Critical-thinking, Achievement, and Problem Solving Skills (CAPS) test in the education system. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 1 INTRODUCTION AND BACKGROUND Introduction This evaluation of USAID Egypt’s Education Portfolio was undertaken by JBS/Aguirre International, in collaboration with the Cairo-based Associate Consultants for Educational Services (ACES). Six educational projects have been reviewed, with the purpose of assessing their efficiency, effectiveness and sustainability across a number of impact areas. Additional information on the activities of other donors, the current education development environment, and work underway by the Ministry of Education (MOE) and others to implement the National Strategic Plan for education was collected as background for the team’s strategic recommendations. The Scope of Work includes three interrelated tasks, of which this is Task Two. Task One was a Meta-Evaluation of all available and relevant evaluations and documents relating to the six projects, as well as a thorough desk review of recent Egypt education sector assessments. Because the Meta￾Evaluation study is to be presented with this report, every effort has been made to avoid duplication between the two reports. The Scope of Work is presented in Annex VI. Also see Annex V for definitions of efficiency, effectiveness and sustainability as used in this study as well as in the Meta￾Evaluation. Background Educational Context Egypt has made great strides in providing access to education, but still has challenges in providing quality education to all of its students. USAID has provided valuable support to the Government of Egypt (GOE) and the MOE in meeting these challenges. Please see Annex I for a brief description of the situation and challenges facing Egyptian education at this time. Also see the Meta-Evaluation for a fuller discussion of the Egyptian educational context. Development Environment in Egypt The environment for education development in Egypt and for USAID’s role in it is positive. There is strong, high-level political support and a clear commitment to transforming the country’s education system for urgent social and economic reasons. A National Strategic Plan based on internationally acceptable standards and processes has been completed and is serving as the lodestone of reform. And generally sound policies and initiatives have created a favorable enabling environment for change. On the other side of the ledger is a less than ideal capacity for implementation. USAID and other donors have been active in supporting the reforms by assisting in building key capacities and providing technical advice, a continuation of which is seen by the Government as critical to success. Obstacles and challenges to success nevertheless are significant. One of the most important is that in spite of encouraging recent progress in decentralization, the education system remains highly centralized. Reform of the MOE’s organization and technical capacities, another key ingredient for success, is proceeding slowly. Teachers’ salaries and status have been raised, but much remains to be done to create the corps of trained and committed teachers that the reforms will require. Nevertheless, compared with the situation ten or even five years ago, remarkable progress has been made and there is reason to expect that trend to continue. See Annex II for a more detailed description of the education development environment. Program Description USAID education assistance in Egypt supports sustained improvement in student learning outcomes by a) improving quality of teaching and learning, b) increasing access to education, and c) strengthening school governance and management. All of these strategies have been addressed by one or more of the projects reviewed in this study. Please see Annex III for a matrix of these USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 2 projects by USAID objectives. Complete project descriptions are provided in the Meta-Evaluation (Task 1). The six projects that have been reviewed are in different stages. Three have been completed. The current status of each of the six is presented below in Table 1. Table 1: Projects’ Starting and Ending Dates Program Start Date1 End Date Girls’ Improved Learning Outcomes (GILO) 2/05/08 4/04/11 Technology for Improved Learning Outcomes (TILO) 9/11/07 9/10/11 Educational Reform Program EQUIP1 (ERP1) 6/15/04 9/10/10 Educational Reform Program EQUIP2 (ERP2) 4/01/04 3/31/09 School Team Excellence Awards Program (STEAP) 9/28/05 8/27/09 New Schools Program (NSP) 1/01/00 7/31/09 Methodological Summary Drawing on information gathered in the desk review and Meta-Evaluation (Task 1), this study employed a multi-method qualitative design consisting of semi-structured one-to-one and group interviews with key stakeholders at central, governorate and district levels (i.e., USAID staff, project implementers, MOE staff, and donors involved in similar initiatives), school site visits where interviews were conducted at the school level, and classroom observations. Data collection took place over a period of four weeks between February 16 and March 11, 2010. Site visits were conducted in 10 governorates benefiting from USAID interventions. Classroom observations were conducted in six schools for each project with a teacher training component (GILO, TILO, ERP1 and NSP). The team also reviewed project documentation. The study adopted a purposive sampling. The sample included 10 schools per project that intervened at the school level (this excludes ERP2) for a total of 50 schools across10 target governorates. In order to implement the study, interview protocols were developed to guide the semi-structured interviews, while allowing flexibility for interviewers to delve into other aspects raised by interviewees that might not have been covered in the instruments. A classroom observation protocol was also designed (see Annex XI, Instruments). A detailed presentation of the study’s methodology, including limitations, can be found in Annex VII. Outline of This Report The following two sections present findings. First, brief reports of the findings for each of the six projects are presented; these are followed by findings regarding several cross-cutting issues, including a review of training models, a summary of findings regarding efficiency, effectiveness and sustainability, and project design issues. The last section presents conclusions and recommendations. FINDINGS: BY PROJECT New Schools Program (NSP) 1. Summary of Project NSP’s main goal was to increase access and enrollment of girls in underserved communities in Minya, Beni Suef, and Fayoum Governorates. To increase access, the project was not confined to building schools, but was a comprehensive school-based reform project with three main components: 1 The official project start dates are listed here; some projects, such as GILO, suffered from delayed start-ups or long procurement periods, as in TILO. The team’s findings reflect the reality of such delays. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 3 1) improve teaching and learning, 2) increase community participation in girls' education, and 3) build schools in deprived areas. The project started in January 2000 with a focus on quality education and access for girls at the primary stage. NSP was extended to May 2005, with a shift in focus towards preparatory schools, especially those that received students from NSP primary schools. Finally, the project was extended a third time to May 2008 with USAID’ Global Development Alliance and Vodafone Foundation Project Learning Increases with New Connections (GDA/LINC) grant to establish school-based Information and Communications Technology (ICT) centers. 2. Principal Accomplishments and Challenges NSP has had two previous evaluations (midterm in 2003 and final in 2008); therefore, this can be considered a follow up to the final evaluation with a view towards assessing sustainability of results as verified in the field. Decentralization NSP was the first attempt at decentralizing school finance. The project provided each school with a trust fund, the interest of which is used by the school to cover maintenance and other school needs. Another financial decentralization is the money generated from ICT centers that is retained at the centers for future updating and maintenance. While interviewees expressed appreciation for the existence of the fund and their ability to use interests to cover expenses, they also noted that a set of regulations on allowable costs and budget lines limits the school’s ability to make the best use of the money. Decentralization hasn’t really decentralized power. We are very much limited and our hands are tied, when it comes to spending for our school. - Board of Trustees (BOT) member NSP is also to be credited for its effective work in activating Parent Associations (PAs), organizing general school assemblies and running PA elections, which laid the groundwork for the creation of BOTs. Community members interviewed confirmed that BOTs are still actively involved in the schools they serve. Gender Equity NSP met the goal of enrolling girls in school and of raising awareness about girls’ education. The project’s final report states that “throughout the project lifetime, NSP achieved a high percentage of girls enrolled (i.e., ranging from 80%-92%) in their schools.” NSP classrooms, including multi-grade classrooms, exceeded the project’s stated goal. NSP’s focus was access to education for girls and empowerment of girls and women in the community. Besides regular schools, there was a major focus on serving out-of-school girls (ages 9 to 14) in multi-grade schools (MGS). These schools enabled the girls to complete primary education, often at an accelerated pace, after which, if successful, they were eligible to enroll in preparatory schools. MGS students proud of the skills they are learning in NSP school USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 4 Women’s participation in PAs and then BOTs is another achievement. Community Education Teams (CETs) advocated for women’s participation and encouraged women’s leadership. Teacher Training It was wonderful. We are still working on the background of that training. All these materials [indicating teacher-made and other materials] we learned all of this from the CARE (NSP) training. - MGS teacher The teacher training component of NSP was a very strong one that offered extensive training and continuous follow-up, plus monthly refreshers to teachers at NSP schools. The training approach in the first phase was mostly direct training supported by the Education Development Center (EDC), Salama Mousa Foundation and members of the Faculties of Education (FOEs). In the second phase, the training approach changed from direct to cascade. NSP developed a cadre of teacher trainers from both teachers and idara (local district education office) supervisors. Training plans were based on training needs assessments and performance problems identified through classroom observation. School-Based Training and Evaluation Units (SBTEUs) were vitalized through the training offered at the schools, while the cluster approach was dropped as the schools did not have enough geographical proximity. Interviewees indicated that the use of a cadre of trained teachers to provide training presented a trade-off between quality and efforts to achieve sustainability through expansion. The effectiveness of these trainings was also questionable as the training time was compressed and many of the teachers in the cadre were promoted into the system. Another challenge was the perceptions of teachers about the quality of the training provided by cadre trainers; teachers preferred to be trained by “experts”. This indicates that there is a need for a change in culture in order to benefit from Training of Trainers (TOT) systems in which teachers are asked to be trainers. Interviewed teachers who were doing training, on the other hand, expressed a need for updating of skills, which might be done through partnerships with FOEs. During the extension period there was minimal training and this focused mainly on technology integration and computer maintenance, under the assumption that SBTEUs would carry on with training new teachers. In many schools visited during the field visits, SBTEUs are indeed delivering training; however, the quality of the training provided is questionable, and may vary from school to school. One trainer explained, The training we received from CARE is very different from the one we are delivering here. We cannot do a workshop here. We do not have the resources. Transfer consists basically of telling them about it. - SBTEU trainer Community Involvement The purpose of the Community Mobilization Component and related advocacy efforts was to produce sustainable community support for education for all children and adults, particularly girls and women. Community involvement in NSP was one of its strongest components. The involvement of the community from the start and the establishment of partnerships with the community created a strong sense of ownership. NSP community involvement was based on the creation of CETs that mobilized the community for girls’ education, carried out community assessment and encouraged communities to donate land for school buildings. This partnership with the community in school construction created strong ties between the school and the community. In the large majority of sites visited, interviewees indicated that these ties remain strong. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 5 Another very important element, that has had really a great impact on us is the community participation work; the fact that the community was involved in land selection gave them ownership, it is something the community owns, they feel a very strong sense of ownership. - Idara Director Establishment of PAs and BOTs for both single grade and multi grade schools “laid the groundwork for the Alexandria Reform Project and the establishment of BOTs nationwide.” (USAID COTR) NSP proved that the community can do wonders, if you put the community on track to find solutions and work toward a specific goal. - USAID COTR Curriculum Development NSPs efforts in curriculum can be summed up in three main accomplishments: 1. Educational Kits: these kits provided teachers with class activities for use with curriculum. ERP1 has adapted these kits for wider use. However, with the changes in the curriculum, teachers reported that they are unable to use them as resources in class. In fact, in many cases teachers considered the kits as resources for extra curriculum activities, for which they had no time. 2. Technology Education Kit: The team saw references to this kit in documents reviewed but did not hear about it or see the kit during the site visits. 3. Curriculum Integration: NSP, in collaboration with the MOE, developed a subject integration guide for teachers of primary four and five. However, this effort was dropped before enough training was provided to support it. Field visit interviews with teachers indicated that they liked the approach but since it was not encouraged by the MOE they had to drop it. 3. NSP: Efficiency, Effectiveness and Sustainability Efficiency • The fact that the project exceeded its targets without any extra funding is an evidence of efficiency that cannot be overlooked. • The use of local trainers from local FOEs is efficient, especially if compared to participant training being offered by other projects. • Using a cadre of trained teachers for expansion is efficient. • Use of SBTEUs for on the job training and refreshers is efficient. • Addition of Private-Public Partnership (PPP) to introduce ICT is efficient. • School construction: Though the introduction of a new low cost model to the General Authority for Educational Buildings (GAEB) may seem efficient, the fact that they only provided for immediate needs and did not think of the long term remains a challenge. We now don’t have a problem with sending our girls to school, but the schools themselves don’t have enough space for them. - Community member Effectiveness • The training model in Phase I was effective, while effectiveness of training in the following two phases is questionable. • Building schools that catered to girls’ needs and provided access to education for girls proved effective in narrowing the gap between boys and girls. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 6 • Establishing multi-grade schools that serve a special population has proved very effective because it took children who were not in the system into school and they have gone on to higher levels. However, questions are raised regarding the adequacy of the facilities in which these schools are operating. • Strong community participation and sense of ownership proved very effective. • Community mobilization around girls’ education increased the enrollment rate and decreased the dropout rate. • Opening the computer lab for community use builds stronger ties to school, and offers the community an opportunity that they would not otherwise have had. • Introduction of computers in MGS with their dire conditions (no gates, no electricity in some cases), and without offering training to teachers is ineffective. Sustainability Most of NSP’s outputs have been sustained. • Teachers observed during field visit still use active learning methodology. • Trust funds for school’s use are still in place and used by schools. • SBTEUs still offer training. • Community involvement with schools is still very active especially after the introduction of ICT. Computer lab is the most successful, because it is not only used with the students but also with the community. Older students use the computer to register at Universities. We charge very nominal fees to the community. We charge low fees to draw people. - School staff • Multi-grade schools still provide girls a second chance. • Schools are generally still in good condition. • Providing training on computer maintenance to some school staff created an “in-house trouble shooter,” in regular NSP primary and preparatory schools. This was not the case, however, in MGS where some teachers reported not receiving training (see effectiveness section above). Not sustained: • School Clusters, because of distance between project schools, is no longer a sustained practice. • Cadre: Most teacher members of training cadres have been promoted to different parts of the system and offer very little help to schools. • Instructional materials provided by NSP are used very little, if at all. • Unless local Non-Governmental Organizations (NGOs) take responsibility for supporting MGS, they may not be sustained.2 Education Reform Program EQUIP 1 (ERP1) 1. Summary of Project ERP1, under the Education Quality Improvement Program (EQUIP 1), began in June 2004 as a companion project to ERP EQUIP 2 (ERP2). It was intended to serve objectives at the school and local level to improve educational quality, while ERP2 worked on policy reform. The project objectives and components changed at least three times, as follows: 2 Although the MGS are supervised by idara personnel, the newer “Girl Friendly Schools”, reportedly supported by Mrs. Mubarak, have in some cases been constructed as little as a block away from MGS. These newer schools offer incentives such as clothing and food, and MGS teachers expressed concerns about losing students to the Girl Friendly Schools. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 7 Objectives and Components: 2004 – 2007 The original structure of ERP1 supported the former Strategic Objective and corresponding activities. However, during implementation, both ERP1 and ERP2 found that the originally proposed structures were not facilitating the necessary integration among components and between the two ERP projects. They therefore proposed to implement their activities under themes central to realizing Quality Education for Lifelong Learning. Thus, in this initial period, ERP1 worked in Families of Schools (FOS) in seven governorates on components organized under the following two integrating themes: • Professional Development: including Teacher Professional Development (TPD); Administration and Leadership Development (ALD); Standards Implementation; and School to Work (STW) • Community Participation, which included Community Participation (CP): Girls’ Scholarships (GS); Multi-grade Schools (MGS); School Construction (SC); School Governance-BOT (SG); Integrated Literacy (IL); Life Skills (LS); and Early Childhood Development Education (ECDE) Objectives and Components 2007 – 2009 After the restructuring of 2007, ERP1 focused on School Based Reform, with an emphasis on Effective Schools. Program components now included Effective Schools; Adult Literacy; Learning, Communications and Information Management; and Other Activities Supporting School Based Reform. Activities that were dropped included School to Work, Life Skills and Early Childhood Development Education. New activities included capacity-building support to the Adult Education Authority, and provision of small grants to mobilize civil society and involve NGOs in support of schools. Strategies for supporting School-Based Reform included an emphasis on developing competent teachers, through continued TPD and the establishment of Teacher Learning Circles (TLC) and Teacher Resource Centers (TRC). In addition, they included a focus on improving school leadership, support of BOTs and increasing capacity building at the idara level. Objectives and Components 2009-2010 With the closing of ERP2 in 2009, ERP1 integrated several activities from ERP2 into its framework, including work on the Professional Academy of Teachers (PAT), M&E databases and work on the national indicators, the focus on the assessment measures Critical Thinking, Achievement and Problem Solving (CAPS) test, Management Assessment Protocol (MAP) and Standards-based Observation Protocol (SCOPE), as well as support to the Quality Assurance Units (QAU), which are implementing accreditation in the idaras. During this period, ERP1 moved away from direct support to schools, focusing strongly on the capacity building efforts at idara and mudireya levels. 2. Principal Accomplishments and Challenges Decentralization The most important thing ERP has done is to train us on how to mobilize resources; training was of high quality. - School Improvement Team (SIP) member ERP1 students elated to have their picture taken USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 8 Findings from interviews indicated great appreciation for capacity-building efforts by ERP1 at both school and idara levels. BOT members, principals and social workers mentioned specific skills they had learned and are continuing to use, such as the ability to do a school self assessment (SSA) and develop an effective SIP. Idara and governorate staff also expressed their appreciation for skills ERP1 had helped them develop, especially in relation to preparation for school accreditation. In response to a request from the MOE, ERP1 teams in the governorates are now traveling to governorates not included in the original project in order to train idara and governorate staff in the units dealing with accreditation: QAU and Training Support Units (TSU). Unfortunately, this work comes across as a “top-down” initiative, in contrast to the previous capacity-building efforts built up from a base of observed school and idara needs in the original seven project governorates. It is doubtful that the current efforts will be sufficient to provide capacity building support adequate for sustainability. In Fayoum, where fiscal decentralization is being piloted, principals and BOT members complained that even though they are receiving a certain amount of money now, there are so many restrictions about its use that they are hamstrung and unable to use the money for things that they need, such as replacing a window frame or repairing the school water system so that water will go to an upper floor. They must apply to the idara for permission to spend most of their resources. They crave “real” decentralization: Decentralization means that as a director I can take decisions without going back to ask permission. - Principal Gender Equity Several interviewees expressed great appreciation for the girls’ scholarships, which provided support for school attendance to many girls who might have been otherwise unable to go to school. In general, comments indicated that there is no longer a need for increased access for girls, but that they need support to remain in school. Many commented that boys are more likely to drop out now because they need to work, and that they need as much or more support now than girls. Training Teachers and school administrators generally expressed satisfaction with training, citing specific examples of ways in which they had changed. These included the ability to use active learning methodologies, comments from principals about new, more collegial environments in schools, and improvements in student attitudes and pass rates. One teacher commented on the way trainers modeled the desired practices, saying, “I felt like a student.” Nevertheless, schools face huge challenges. The most frequently cited concern was the increasing enrollment, which means class size of up to 60 in many schools, even with double sessions. Teachers find it very difficult to continue active learning approaches with large classes and a lack of resources. As a science teacher for instance, I have only one model item to demonstrate; how can I use it with the whole class? We do not have enough resources/materials; this is a major constraint for us. - Science teacher ERP1 used varying training models, sometimes using direct training, as was the case with comprehensive assessment, which was felt to be too complicated to entrust to the TOT model. For much of their training, however, ERP1 used a TOT model, training teachers who were to return to the SBTEU and train others. Some teachers expressed a preference for direct training, while others seemed satisfied with the in-school training provided. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 9 With ERP1 support, SBTEUs were active in most schools visited, and, where there is effective leadership, these appear to be an effective tool for training new teachers who come in each year and thus improving the possibilities for sustainability. However, SBTEU members expressed their own need for continued updating, and some reported having to use their own funds for necessary training supplies. There is also a need to provide incentives for teachers who serve as in-school trainers. One of the success points of ERP is that after our long exposure we can tailor any training to any need of our teachers. But, training in new techniques and new approaches—where can we go to get this? - SBTEU trainer ERP1’s Teacher Learning Circles and Learning Resource Centers received mixed reviews. The Learning Circles are effective when a school leader, either from the SBTEU or a senior teacher in a subject, organizes them on a regular basis. Teachers found such circles useful in planning lessons and solving problems together. Where the leadership is lacking, however, such circles are not happening. In some instances they are only happening in one subject. The LRCs are appreciated by idara staff, who use them for training. Teachers who are close enough reported going and making materials in the “Make and Take” room there. Not surprisingly, however, teachers who are far from the Centers do not get there; team members met several teachers who had never been to a LRC. Re TLCs: The 5th grade curriculum has changed this year, so this circle is very important to analyze the new curriculum. ERP has changed our brains. - Math teacher An interesting synergy between projects was noted in two NSP/ERP1 schools visited. It appeared that the years of strong training and community development provided by NSP established a solid base that ERP1 was able to build on by providing additional training on new topics, such as comprehensive assessment and critical thinking, as well as the additional support to idara staff. This finding is consistent with SCOPE results reported by ERP1, and suggests that if USAID wishes to bring schools to a more solid level of quality, it may be worthwhile continuing project involvement over a longer time period. ERP1 training of supervisors was useful for supporting teachers in implementation of project methodologies. Although supervisors provide varying levels of support, they were reported to be supportive of the new methodologies, so that ERP teachers were not receiving mixed messages from project and MOE staff. Most interviewees expressed appreciation for the reading and writing initiative, and requested more support for this as well as training in general for dealing with students at different levels. Teachers and administrators frequently requested that training be held on non-school days; both teachers and principals expressed concerns about having teachers missing during school time, and some teachers stated that they can benefit more from training provided at a time when they are not also preoccupied with preparing and teaching their classes. They commented that during vacation time, they can attend fully to the training and have more time to reflect on it. We can’t have the teachers missing school. - Principal Several of those interviewed expressed the need for vocational/technical education, and some lamented that the ERP1 School to Work project had been dropped. Around 65% to 75% of students go to vocational education, and only 35% continue to other areas. - Principal USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 10 Also frequently mentioned was the high percentage of illiterate parents; some teachers wished literacy programs were more closely connected with the schools. Community Involvement The training provided to BOT members, social workers and School Improvement Teams has been instrumental in helping sustain community involvement, although keeping community members involved is an ongoing problem. Involvement too often is interpreted as meaning “donations” and not real involvement in school activities. Curriculum Development ERP1 adapted the instructional kits originally developed by NSP, but the curriculum has changed since the revision, and these kits are not frequently used. Teachers are under great pressure to cover prescribed daily lessons in the curriculum and textbooks, and have difficulty finding time to use what they consider “extra” activities. Such kits or instructional materials would be more useful if they were keyed to specific lessons or themes in the curriculum. The new comprehensive assessment plan does provide grades for activities, so there is now more incentive for teachers to use such materials. Several teachers commented that they particularly appreciated ERP1 training on how to develop their own instructional materials. 3. ERP 1: Efficiency, Effectiveness and Sustainability Efficiency The frequent changes in project focus, components and activities were inefficient. In addition, both efficiency and effectiveness were compromised by lack of consistent communication and cooperation between ERP1 and ERP2 projects. It would have been more effective and more efficient to have one project carrying out the educational reform project. One ERP project implementer commented that it is important in such a project that information is able to flow in both directions (policy needs to inform those who are trying out the models as well as vice versa). Effectiveness In spite of frequent program changes, a number of ERP activities appear to have been effective. Many teachers reported changing their behavior as a result of training. Classroom observations showed that most teachers are using group work, although many mistakenly believe that group work is synonymous with active learning. Because this suggests that change is still superficial, many teachers need continued support to move to a deeper level of understanding of the new methodologies. How can we persuade them that active learning means more than rearranging the furniture? - ERP staff member Sustainability The capacity building at the idara level has been effective and bodes well for sustainability of practices. ERP1 worked with clusters of schools; however the clusters appear to be active now only in some areas visited. The clusters formed by ERP1 do not coincide with clusters in the MOE system; for sustainability it would be important to use existing systems whenever possible. Strong leadership is required to sustain many practices, including training by the SBTEUs. Training provided by ERP1 for principals should assist sustainability of program effects. However, it will also be important to support the MOE in providing improved training and certification programs for school leaders. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 11 The fact of the matter is that the schools where we succeeded are those with strong principals. So, leadership should be a focus. - ERP staff member Education Reform Program EQUIP 2 (ERP2) 1. Summary of Project ERP2 was the policy half of the ERP program. ERP2 used a systems approach to build a foundation for policy change and institutional capacity for replicable reform. It worked in the same governorates as ERP1 and focused on the same basic themes: improvements in student learning; decentralization; community participation; and the professional development of teachers. Its role was to work from the top down on developing an enabling environment for reform and to promote standards of excellence throughout the system, while ERP1 piloted school-based reforms in schools and idaras. Like ERP1, ERP2 started out with a scope of work which, in addition to its core functions, included a number of other responsibilities in such areas as introducing English in grades 1-3 and work force development (School to Work (STW) and Community Youth Mapping (CYM)). Another add-on – the Creative Science Education Initiative (CSEI) – came in the second year of the project. Reportedly, during this period USAID had a policy to consolidate program activities in a few large projects to simplify program management and reduce costs. Significant changes in the scope of work occurred in 2005 as a result of re-thinking of program objectives and strategies by the two EQUIP Leaders and again in 2006 and 2007 as a result of two changes in Ministers of Education and MOE policies and adjustments in USAID program priorities. The details are spelled out in two program modifications that took place during this period, as well as the revised ERP2 2007 work plan. In the process, the add-ons were dropped from the program, as well as, regrettably in the team’s opinion, the pre-service teacher education component. While a good deal of work took place between 2004 and 2006, with mixed results, the second ministerial change brought into that key position a person who had been prominent in the highly￾regarded Alexandria school-based reform and de-centralization project. This individual came to office with a commitment to promote application of the principles of that project in the education system, a strong commitment to strategic planning to guide the process of educational reform, and a readiness to look to USAID as a primary source of technical assistance. This opened up an exceptional opportunity for ERP2. As one observer put it, “from that point on ERP2 became essentially an adjunct of the MOE”. 2. Principal Accomplishments and Challenges Strategic Planning ERP2’s role in 2006 in helping create the first national education strategic plan was central. Without ERP2, the strategy probably would not have happened or would have been something of limited value. Instead, a highly professional and comprehensive National Strategic Plan was developed and, with the subsequent blessing of the GOE at the highest levels, became the lodestar of reform. With ERP2’s help, similar strategic plans were developed for the country’s 29 governorates. The MOE’s operating units, including the key idaras, were instructed to adjust their work to the plan’s goals. The donor community was informed that, from that point on, their programs should be related to plan implementation. And a social marketing campaign, developed by ERP2, was launched to build awareness and support in the country. Two key challenges were that understanding and support of the plan in the MOE was limited to the upper levels, with limited buy-in at lower levels, and that the social marketing campaign reportedly did not have a wide impact. Nonetheless, for ERP2 it was a considerable achievement. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 12 National Strategic Plan Indicators After the publication of the National Strategic Plan, ERP2, reportedly working with a small group of people, developed National Education Indicators (NEIs) for each anticipated outcome in the plan. This was later published in 2009 as a technical guide for preparing annual estimates and bi-annual progress reports. The team was unable to find evidence of any use of the indicators or of the preparation of progress reports, though it is possible that some use of them was made in the Ministry for internal planning purposes. The complexity of the tasks outlined in the guide, together with the fact that they apparently were developed without consultation with practitioners or anybody else, leads the team to doubt seriously that they could be employed on a national scale without significant changes and investments in building capacity for their use. This, like the comprehensive assessment program discussed below, would appear to be a good example of ERP2’s tendency to develop policies and tools at too high a theoretical level to be practical in the Egyptian context. Decentralization ERP2’s efforts in this area began with an effort to reproduce in three governorates the decentralization components of the Alexandria reform pilot. These included decentralization both of funding and decision-making authority. Over time, with the MOE, the Ministry of Finance (MOF), and the Ministry of State for Local Development (MSLD) involved, the focus was narrowed to piloting selected fiscal aspects, including funds for schools’ minor and major maintenance, the proportion of school fees to be retained at the school level, school capital expenses, including technology, and buildings and other infrastructure investments.3 The maintenance and school fees items have been approved and reportedly are in the process of being scaled up to the national level, effective in 2011. The last two items, involving significantly larger sums of money, have been fiercely resisted. While this is a good beginning, the key challenge now is to find ways to extend the reform to give school managers authority over other matters critical to school improvement, notably control over hiring and transferring of teachers. Another challenge is to prevent higher-up MOE bodies from wresting control of the decentralized funding from the schools and using the money for their own priorities. Professional development of teachers ERP2’s policy and institutional change efforts in the areas of in-service training and professional development of teachers were arguably its most important accomplishments, though whether they will develop along envisaged lines is uncertain and several years of work will be required to realize their full potential. The results of work in this area include: the creation of a Professional Academy for Teachers (PAT); the development and successful launch of a teachers’ professional cadre; creation of an induction and mentoring system for new teachers coming from the universities; and mechanisms to develop a strong, permanent in-service teacher training capacity via certification of qualified training providers and trainers to conduct new in-service training programs to be developed under the aegis of PAT. Design and implementation challenges are embedded in each stage of the implementation process now getting underway. Standardized Testing ERP2 developed three instruments, initially to assist USAID in measuring the impact of its school￾based reform interventions but subsequently made available to the MOE for wider applications: the MAP for assessing school management, the SCOPE for standards-based observations of classroom teaching, and the CAPS for measuring student achievement. All three instruments reportedly are being used informally by projects and others, but only CAPS is being institutionalized. The others may ultimately be incorporated in the system, but the National Authority for Quality Assurance and Accreditation (NAQAA) is said to be developing other instruments that may take precedence. In 3 A high official of MOE told a member of the team that “decentralization in Egypt is not political, but rather technical and administrative”, a position which, if sustained, limits the extent to which decision-making authorities are likely to be decentralized. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 13 addition, ERP2 worked with MOE on the development of a Comprehensive Assessment Program (CAP) for use by classroom teachers to monitor student performance. This program is widely disliked by teachers, who see it as extra work with no real payoff in terms of improving either learning or their teaching. The program reportedly is now being simplified in response to the criticisms. Organizational Transformation Under this rubric, ERP2 assisted the MOE to assess its organization and personnel in two tasks deemed by the donors and many other observers as critical for assuring success of the reforms. The first was to assess the functions, roles, and responsibilities of all levels of the Ministry and to develop a new organizational structure which would devolve functions to the governorates, districts, and schools in order to implement the reforms more effectively and expeditiously. The second was to develop a plan for reducing the size of the Ministry’s staff to increase efficiency, reduce costs, significantly improve the ratio of teachers to administrators, and open up opportunities and provide incentives to bring fresh blood, in the form of younger and better-trained people, into the Ministry. An inter-ministerial committee consisting of MOE, the Ministry of State for Administrative Development (MSAD), and MSLD was formed to work on the first task. One of the principal challenges to both is resistance of MOE personnel at all levels to changing their responsibilities or their effective tenure in their jobs. Recently, the new Minister is reported to have included positive reference to organizational transformation in one of his public statements, which, if true, could bring renewed attention to these issues. Quality Assurance and Accreditation In 2006 a law was passed establishing NAQAA, an external body for accreditation of all educational institutions in the country. In support of this initiative, ERP2 prepared internal and external review accreditation manuals and training materials, as well as National Effective School Standards, to be the basis for accrediting schools. These were piloted in two governorates, Minya and Alexandria. Subsequently, Quality Assurance Units (QAU) were established in all governorates and idaras, with the help of training and materials provided by ERP1. In 2007, before NAQAA was operational, ERP2 worked with an MOE working group to draft a program that would accredit all 40,000 schools in the country over a five year period. This was again piloted in the same two original governorates, then in all seven USAID-target governorates, and eventually, at MOE’s request, was scaled up to the whole country, with the rollout of the initial accreditation plans concluding in the fall of 2008. Parenthetically, this date corresponds with the opening of year two of the STEAP program, which many say served as a bridge to accreditation. The challenge of this ERP2-assisted initiative is the way in which accreditation targets may have altered the priorities of schools and local District education offices away from teaching and learning to obtaining accreditation. Leadership Development Briefly, this component of ERP2’s work consisted of development and offering of a year-long residency leadership training program to 147 middle managers drawn from the 29 governorates. The graduates were to be considered for senior assignments in the Ministry. The team did not have the opportunity to evaluate the effectiveness of the program, but it doubts seriously that the high unit cost of the program and the diversion of both project and MOE leadership time to it were justified. This view is reinforced by the fact that at the end of the course, no effort was made by MOE to continue it. The same time and money spent on attempting to influence MOE to change its human resource policies to provide openings and incentives to recruit a cadre of younger, better￾trained experts could have had a much greater impact on MOE leadership.4 4 So far as the team could discern, the decision to develop the residency leadership training program was made jointly by MOE and ERP2, presumably with USAID’s blessing. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 14 Education Management Information Systems (EMIS) One of ERP2’s assignments was to help the MOE create a unified database to support a system-wide capacity to collect and maintain timely, accurate, and relevant data to underpin planning and management of the Ministry’s work. The first priority was to “make the existing information mutually intelligible”. After “months of joint committee work”, an EMIS Transition Plan was created which reportedly improved access to the many, disparate, existing databases. Work was also done on designing school databases, but, reportedly, the desired overall system still does not exist. While the team did not have the time or capacity to investigate the EMIS area in detail, the overall impact of these initiatives so far seems to have been limited. 3. ERP2: Efficiency, Effectiveness and Sustainability Effectiveness The overall effectiveness of ERP2 is hard to evaluate, as its product was a mix of the good and not￾so-good. Its work was clearly done carefully and professionally by qualified people. The issues into which it chose or was drawn to enter were, for the most part, important pieces of the reform process, particularly during the last half of the project, and, thus, were within the joint MOE/USAID strategy. One way in which it went wrong was the tendency in some cases, such as strategic plan indicators, the CAP, and quality assurance and accreditation, to design solutions and tools that adhered to international standards, without accompanying them with proposed realistic strategies for staged introduction into the Egyptian context. If ERP had been a unified program, ERP2’s initiatives might have been subject routinely to a reality check by the field-based half of the project.5 Efficiency On balance, ERP2 would have to be judged as inefficient, because key policy and structural changes contemplated but not achieved remain crucial to success in reforming Egyptian education, notably MOE re-structuring, policies and practices governing deployment of teachers, MOE staff quality, overstaffing, and lack of opportunity for trained, young professionals. Sustainability In regard to sustainability, because of the importance of the topics and the absence for now of alternative analyses and tools, the impact of ERP2 is likely to be felt, for better or worse, for some time to come. School Team Excellence Awards Program (STEAP) 1. Summary of Project STEAP began in 2005 and ended in 2009. It ran for two cycles: September 2005-2007 (STEAP I) and September 2007-2009 (STEAP II), with a total budget of $12.9 million. The objective of STEAP was to achieve greater results in student learning in primary and preparatory schools by encouraging high levels of collaboration among stakeholders to implement a series of school-improvement measures, including: active learning; SIPs; community mobilization on behalf of the schools, under the leadership of BOTs; and improvements to the physical plant and equipment. Under criteria established by the project, these measures were to be based on the National Education Standards (NES). Incentives in the form of monetary awards, to be distributed among the school improvement teams, were offered by the MOE for winning teams at the idara, mudireya, and national levels. Besides laying the groundwork for accreditation, a goal of the project was to help the MOE in promoting application of the standards by giving schools a reason to do it on their own. 5 A significant factor, also, was the MOE’s willingness to accept and even prefer state of the art solutions. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 15 STEAP operated in all 27 Egyptian governorates and in both cycles included primary schools and a limited number of preparatory schools. Reportedly, the original plan was to select 1,000 schools, train these schools in project requirements and how to meet them, have them draft SIPs, and then choose the 200 schools with the best SIPs, work the rest of the year with these 200 schools, finally giving awards of US$25,000 each to around 81 schools. By the time the project started, however, there was a new minister; and his deputy decided to work with virtually all schools in the country, a total of 16,000 in all. After this minister was changed, the new one reduced the number of schools to 4,000, which was still four times the original target. In spite of the increase, the project budget was not increased, which required delays in the initiation of the project and the necessity of relying heavily on volunteers. Ultimately, a total of 10,500 schools participated in the project. The announcement made at the eventual start of the project (STEAP 1) specified the objectives as: • Raising awareness among school communities about school-based reforms. • Emphasizing the importance of implementing and achieving targets specified by the National Education Standards. The second cycle of the project (STEAP II) was based on new standards and improved SIP processes and was more tightly focused on helping participating schools prepare for accreditation. Participation was by invitation only. Project volunteers worked to assist schools by conducting periodic site visits during crucial stages of the school improvement cycle to assess degree of progress and provide technical assistance where needed. There was increased attention, as well, given to obtaining private sector support. 2. Principal Accomplishments and Challenges Community Involvement The team’s findings show that the most successful part of the project was the strength of community support and participation in preparing SIPs, raising funds, and improving school facilities. Surprisingly, such participation occurred even in urban areas. Interviewees reported that in some urban communities, up to 70% of the parents were active supporters and an average of 400 parents attended the last General Assembly held during the competition year. With the community and BOT involved, communities in many cases supplied schools with computers and funded physical improvements. Community representatives told the team that STEAP had changed their attitudes and empowered them to achieve goals previously viewed as impossible. Someone called USAID from the US to contribute to his village for the competition. However, some interviewees reported that “The level of community participation achieved during STEAP will not continue,” an observation that the team found to be accurate in many cases. Selection Process The selection process, particularly in the first year, was problematic. Although the idaras were not significantly involved in the project, they were given the responsibility for making the first cuts and then referring the rest to the mudireya and ultimately national levels. In the first cycle, there was little transparency at either the idara or governorate level, leaving the program open to charges of favoritism and, in some instances, changing the rules in the course of the competition. When the STEAP students give a warm welcome to the evaluation team USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 16 results were announced, there was major dissatisfaction among the unsuccessful contestants, much of which remains to this day. The second cycle selection process was conducted with fewer problems. Policy reform Most of the project interventions were relevant to policy reform. STEAP proved that, with incentives, the idea of school-based reform approach could be successfully promoted. The majority of schools also proved during STEAP that they were capable of improving. Some interviewees mentioned the link between STEAP and the initiation of change processes at the school, indicating that the “school is now doing self-assessment, needs assessments, and preparing improvement plans on its own” and saying that “All the improvements we have mentioned are the result of STEAP.” (School staff members) 3. STEAP: Efficiency, Effectiveness and Sustainability Efficiency The multiple changes in the number of schools, project objectives, and criteria negatively impacted the efficiency of the project. Also, there was a lack of coordination with the idaras, which could have provided greater assistance to the project and the schools than they did. The reliance on volunteers, while remarkably successful under the circumstances, further impacted the frequency and quality of project follow-up. Throughout the project there were only six full time and two part-time employees managing 1,000 volunteers. School representatives reported that sometimes the committees which were formed to visit schools did not actually go to them. They told us that a committee would come to evaluate our school, however they did not. I feel that they just selected schools based on old data and not on the current situation. - School deputy Master trainers should be from research centers and universities. The STEAP trainers weren’t qualified; they only had two days of training. - Another school deputy Also, the method of allocating incentives for schools created dissatisfaction. In the first cycle the award was divided among a few at the school (including the school improvement teams that worked on the STEAP application, principals and other staff). Later, this was changed to dividing the award among all persons at the school, and even this way it still caused unhappiness. The project also did not provide any incentives for staff at the idaras and, as a consequence, many of them ended up criticizing it. Effectiveness In spite of the initial program changes, the absence of coordination with some idaras, and lack of transparency in the process of selecting winning schools, especially in the first cycle (2006/2007), a number of STEAP activities appear to have been effective. Findings show that STEAP did indeed set the stage for accreditation and raised awareness about school quality in many schools. The project also promoted the whole idea of using standards to assess gaps and use of tools to plan and evaluate school improvement measures. Site visits by the team showed that most winning schools and a majority of non-winner schools are still preparing and updating their SSAs and SIPs, though partly because of the drive for accreditation which followed on the heels of STEAP. Another indicator is the fact that, in general, schools which participated successfully in STEAP are now more likely to be nominated for accreditation than other schools, even if they did not receive a prize. As one of the interviewees reported, “If it was not for STEAP, none of our schools would be able to submit their papers for accreditation. When NAQAA came on board and started talking about SIP, etc., these were things their schools were already doing and the evidence for that is that the schools that won STEAP awards got USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 17 accredited.” STEAP was also effective in enhancing community involvement and obtaining private sector support, although, as discussed below, generally the level of involvement attained during the life of the project was not sustained thereafter. Sustainability STEAP created capacity in schools to complete and implement SSAs and SIPs. Based on the team’s visits, the use of tools and the skills provided by STEAP training was found to be sustained in most schools. STEAP, as noted, clearly mobilized substantial community and BOT involvement in the vast majority of schools. Team visits, however, disclosed that the level of involvement achieved during the project was not generally sustained. There are two ways to look at this finding. The first is that the kind of excitement and enthusiasm created by a national competition is unlikely to be sustained when things return to normal The other, however, is the fact that STEAP showed that given proper incentives, there is a great deal of potential present there to be tapped. This could be a significant factor should a national scaling up of school-based reform models be launched. The lack of coordination with the idaras worked against sustainability. The idara was not involved in the project; they asked us to help, but mostly we worked as data providers only. - School staff member The financial incentives for winning schools obviously were not sustainable. The trust fund concept developed by NSP might have been an effective alternative. Girls’ Improvement in Learning Outcomes (GILO) 1. Summary of Project Building on the experiences of NSP and other on-going USAID projects, the GILO project focuses on improving the quality of education and learning among girls through the following intermediate outcomes: 1. Expanding equitable access to education for girls 2. Improving educational quality 3. Strengthening school management and governance 4. Strengthening the institutional capacity for decentralized governance Changes in the scope of work since inception include focusing on four rather than seven governorates; changing the 300 schools target to 2,700-3,000 classrooms; and broadening the scope of support on decentralization efforts beyond support to GAEB. These changes were a result of adjusting the project to the reality in which it operates. GILO students collaborate in group assignment USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 18 2. Principal Accomplishments and Challenges Decentralization GILO is continuing ERP2’s efforts in decentralization, which so far is limited to fiscal aspects. Decentralizing recurrent expenses for minor maintenance work at the school level is being piloted in three governorates. Interviewees in Fayoum, where decentralization is being piloted, expressed their appreciation for receiving decentralization funding to be managed at the school level. Some reported, however, that there are a number of restrictions on how the money can be spent, which is undercutting schools’ ability to make spending decisions. This has caused confusion and frustration. Decentralization of funds for major maintenance to the governorate level has been approved by the Ministry of Finance and is scheduled to begin in 2010. Other capital decentralization efforts for major maintenance and construction are under consideration but are facing major opposition. Although the efforts to date include groundbreaking initiatives in the centrally-driven Egyptian education system, these are nonetheless small steps in the larger scheme of education sector decentralization (see ERP2’s section above for more on these efforts). Teacher training Building on the successes and lessons learned from NSP and other projects, GILO has developed a holistic approach to TPD that goes beyond training teachers to training idara supervisors, school administrators, social workers, SBTEU coordinators and other school-based staff (i.e., the school improvement team), thus creating a support system that could sustain new teaching practices introduced by the project. Interviewees spoke highly of the strategy of involving all levels and working through the system to institutionalize school-based reform at both school and idara levels. In a number of sites interviewees testified to the fact that the SBTEU, the unit responsible for supporting professional development at the school level, “was brought to life” by the project to fulfill its important role, although the extent to which its capacity was built is not known and it was clear that the link between SBTEUs and the idara is still very weak, consisting mainly of reporting. Other aspects to be highlighted in this area relate to training delivery modality and strategy. Direct training and follow up, coupled with school-based subject matter coaching (including circles, in-class observations, and lesson planning) delivered by trained senior teachers and/or supervisors, are proving to be effective strategies to solidify concepts and teaching methods, while helping teachers to apply new teaching methodologies to specific subject areas. In the overwhelming majority of schools visited administrators and teachers highlighted the positive effects of this strategy. Coaching has created a learning community in this school. Before we all worked alone, now we have teams and cooperate with each other. - Teacher The coaching schedule is built into teachers’ school schedules (often consisting of weekly events), therefore not adding an extra load for teachers. School-based coaches, on the other hand, have an extra load for which they are not being compensated, although this could become sustainable, considering that coaching and supervision will reportedly be part of the professional development track for senior teachers placed in the new cadre system. While GILO’s training model is strong, some aspects of it can be improved. Teachers in nearly all sites noted that the amount of direct training received is not enough, and that continued follow up will be needed for a longer period of time to ensure sustainability. Some teachers and administrators also expressed concerns regarding training sessions taking place in school hours, which puts a burden on the schools. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 19 Teachers reported positive effects of training on improving teaching practices. Classroom observations indicate that teachers are implementing active learning strategies, but that command of such methods is still somewhat superficial in a number of cases. It was highlighted that students are more participative and motivated to attend class. Interviews with students support these comments. While the majority of teachers noted that it is too soon to see impacts on student learning outcomes, some indicated that retention of information is much higher with the use of active learning methodologies. Community participation GILO has a school leadership capacity training program that targets BOT members and parents. Accomplishments in this area go beyond the number of community members trained. Not only were community members interviewed appreciative of the skills they have attained through GILO training, but they also spoke of the utility of such skills and their ability to apply them, suggesting that efforts in general have been effective in affecting change and may therefore be sustained. In a number of sites, interviewees spoke of BOTs existing on paper only before GILO intervention, and highlighted the capacity of BOTs to mobilize the community and to solve problems in GILO-target schools. The BOT was not active before, now it is active. Now we are more effective, we know our roles and responsibilities. - BOT member The inclusion of women in BOTs is another major accomplishment, especially in communities where gender barriers are high. While BOTs include parents, it was not clear the extent to which other parents are now more involved and co-participants in school activities, although BOT members highlighted the larger number of parents participating in the general assembly as a result of BOT work. Similarly to NSP, GILO has also set up Community Education Teams (CETs), although these were formed after schools had been selected. While the case was made for their importance as a mechanism to identify and address barriers to girls’ education in serving communities, CETs’ sustainability once the project ends is questionable. Gender equity I went to a multi-grade class and there were two girls dressed in galibeya, their hair had been shaved so that they could pass for boys. - GILO staff member While aggregated national figures indicate that gender parity has nearly been reached, disparities still exist in certain areas, in particular rural communities—the problem is “localized,” noted interviewees. A number of them reported that the project has been successful in drawing girls to schools in areas where access was an issue, and in raising awareness about the importance of girls’ education, and women’s participation in community forums, including BOTs (see section above). However, many indicated that the problem is now less of access than it is of keeping girls in schools once enrolled (retention) and making sure they continue on to higher grades. A number of interviewees noted that girls drop out after grade nine. One of the reasons mentioned is the lack of nearby high schools to accommodate these girls, who are not allowed by parents to travel long distances to attend school. Another issue that was brought up in some communities is the reverse gender trend: boys dropping out of school to go to work. Curriculum development GILO’s work with the MOE on the Early Grade Reading Assessment (EGRA) is a promising opportunity to address a pressing need: strengthening reading and writing in early grades. It is believed that the outcome of such an effort will be the design of supplementary materials to accompany the existing curriculum. If this work moves forward, it may require policy change. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 20 Moreover, at the school level, recent training on designing visual aids with resources available in the environment was highly praised by teachers. Other Challenges A number of project staff echoed several internal and external challenges facing the project since inception. The lapse of time (reportedly two years) between project bid and approval, coupled with the changing environment in the country, meant that data on which the project proposal was based were no longer accurate at startup. The GILO experience is a typical example of the problems caused by the protracted proposal bidding process. Furthermore, schools expected to be built by the government and made available for project intervention have not materialized. This has created major problems, including: (i) the time and effort put in by staff in selecting schools for intervention; (ii) working in schools where the need for girls’ education is not pressing (the case in some cohort 1 schools, which were selected using a list provided by GAEB based on outdated information), and (iii) raising expectations and fueling frustration in target communities where construction was promised. One cannot help but wonder why the project was not redesigned before implementation. Another challenge is that GILO is a “numbers-driven project.” Targets are unrealistic at best for a project with a three-year timeframe, but not atypical of an output-driven industry. The project is also spread too thin, with some ad hoc sub-components that could warrant a stand-alone project (e.g., ICT and decentralization support). Change in contractor project management is another issue that could explain delays in implementation (i.e., deployment of the ICT and School Management Information Systems sub-components). Exogenous challenges mentioned include the high turnover of leadership at school, idara and mudireya levels; the increasing number of temporary teachers in benefiting schools; and the failure of the government to build the promised schools. 3. GILO: Efficiency, Effectiveness and Sustainability Efficiency GILO can be rated as efficient in that the project has benefited from and built on other projects’ experiences, having the unique opportunity of improving on field-tested materials, practices, and processes. Overall, however, the project is highly inefficient. Contrary to NSP, GILO is not addressing a pressing need when it comes to expanding coverage of girls’ education. This is so not only because access is generally less of an issue, but also because the project is being implemented in a number of schools/communities where access is no longer an issue; also, the fact that it is operating in some of the same idaras and governorates where NSP was implemented makes it even less efficient. Moreover, some of the project sub-components are significantly delayed (although for reasons outside of the control of project, in most cases). Effectiveness In spite of the challenges mentioned above, the project has proven effective in achieving results in its main components. For instance, teaching practices have improved as indicated by interviewees’ perceptions and as evidenced through classroom observations. Continued support will be needed though to solidify new teaching practices that can in turn lead to improved student leaning outcomes. Furthermore, results of the project’s community participation efforts (and women’s inclusion in particular) cannot be contested. The lack of flexibility when it comes to targets (numbers-driven), however, can put at risk effectiveness, as well as efficiency and sustainability. For instance, plans to expand training to cluster schools (non-target schools) before the project ends has raised concerns over the sustainability, efficiency and sustainability of such an effort given that, reportedly, training to be offered to these schools will be minimal, as will be follow up on the part of the project. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 21 Sustainability Building the capacity of stakeholders at all levels, creating structures to support professional development at the school and idara levels, and building the capacity of master teachers and supervisors to deliver training (coaching in particular) are sustainability mechanisms put in place by the project that could lead to long-term effects after funding support ends. It will be key to continue to strengthen SBTEUs and to improve linkages between these and the idaras and between the latter and mudireyas. Work at these levels will depend on continued cooperation with ERP1. Strong community participation as a result of project efforts is another factor in sustainability. Technology for Improved Learning Outcomes (TILO) 1. Summary of Project In alignment with the MOE strategic plan, the Technology for Improved Learning Outcomes (TILO) project aims at providing an integrated model for introducing technology into school-based reform activities as a means to upgrade the quality of teaching, school management and student learning outcomes. To that end, the project is working on four components, namely: (i) improve the quality of teaching and learning through technology; (ii) develop public-private partnerships to increase innovation and share responsibility; (iii) build the capacity of MOE at all levels for effective management of technology for education; and (iv) measure the impact of technology on teaching, learning and management through effective monitoring and evaluation. The project operates in 11 governorates, originally targeting 188 primary and preparatory schools undergoing School￾Based Reform (SBR), and 85 public experimental Smart Schools. One year ago, TILO began to receive and respond to requests for expansion from a number of governorates (Alexandria, Helwan, and Aswan), raising the number of targeted SBR schools to 292. Additional increases are expected as other governorates plan to request expansion. Field visits and stakeholders’ interviews have shown that there are generally positive views regarding the TILO model, due to the degree of buy￾in and commitment established at different levels in the system, the cascaded training, and availability of training materials and learning resources. 2. Principal Accomplishments and Challenges Decentralization TILO provides a useful model for decentralized decision-making, with its demand-driven approach, in which idaras and schools apply to participate in the project. Selection was based on providing evidence that schools met the project’s selection criteria. Such an approach proved useful in creating a level of commitment, which in turn had a positive impact on the implementation of the model in the selected SBR idaras and schools. This contrasts with the experience of the smart schools that were nominated by the MOE for TILO intervention. However, the links with the idaras are not yet as close and cooperative as they will need to be for effective meshing of idara and project roles in the schools, as well as long-term sustainability. Students delighted to work in pairs in TILO computer lab USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 22 Teacher Training TILO employs an innovative combination of direct and cascade teacher training approaches, in which pools of master teachers are trained in the school (by project master trainers) and are then expected gradually to train the rest of school teaching staff, utilizing TILO training modules. Although the initial training lasts most of a school year,6 these master teachers receive only three days of training in how to train others. Some of the master teachers felt that the time devoted to training of trainers (TOT) was not enough to develop their skills as trainers; their opinions were divided as to how well their training was received by their fellow teachers, and many expressed concerns about the shortage of time available for them to carry out the in-school training effectively. Since the system will depend even more on master teachers to expand the TILO model to other schools, it may be necessary to provide them with refresher or further training and to consider developing a mechanism to support and evaluate their training performance. There is not a clear system of incentives for the master teachers; incentives such as a step on the Teachers’ Cadre, and/or a reduction in their regular workload, are needed to compensate them and assist them in carrying out these critical capacity-building and capacity-maintenance responsibilities. Trained teachers work under the supervision of school administrators and idara staff. School visits and meetings with master teachers indicate that this supervision may not be sufficient; there may be a need to put in place a sustained coaching system to assure that teachers understand and apply what they have been taught, particularly the training related to integration of technology into the classroom. In this connection, it is important to note that no systematic coordination was noted between TILO master teachers and SBTEUs in TILO schools. In addition to training eight teachers per school and then selecting five to become the master teachers, TILO trains three teaching supervisors in each school. Those trained might or might not be SBTEU members; some SBTEU heads mentioned not having attended any TILO training. Since the SBTEU head is responsible for other training being provided in the school, and given the efforts of other USAID projects to strengthen the SBTEU, it is important that TILO develop an explicit system of coordination between those responsible for in-school TILO training and school SBTEUs. Unfortunately, the limited number of TILO master trainers per governorate affects the time they have available to provide on-going technical assistance to master teachers, and relying on the supervisors that TILO trained to provide follow-up training at schools is not an ideal solution. Some supervisors have been trained by the project and others not, evidently under the assumption that the trained ones would be dedicated to TILO schools and would not be transferred. This is unlikely, and would therefore create a need to plan additional training at intervals through the life of the project. In fact, TILO teachers frequently mentioned their need to prepare two lesson plans emphasizing two different teaching approaches, one for themselves, done on the computer, and a traditional one written by hand for their supervisors, who had not been trained by TILO. Administrator Training TILO training for school administrators has been effective in bringing them on board with management of technology investment in their schools. Training them on the SCOPE has enabled them to observe improvement in teachers’ performance, Community participation Although it is not a direct goal of the project to promote community participation in TILO schools, BOTs necessarily play important roles, as they are required to do. In the limited number of schools visited by team members, BOTs were aware of the project in some schools, but not in others. It would be a good idea for the project to provide training for the BOTs, particularly in the role of 6 TILO’s initial training consists of approximately 12 training sessions interspersed over six to eight months during the school year; teachers attend the training for 1-3 days and then return to class to apply what they learned. TILO Master Trainers visit the schools between the training sessions to provide in-class follow-up and support. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 23 technology and the need for the schools’ planning, early on, for the time when they will take over responsibility for managing and sustaining it. In one school visited, the BOT had already made plans to establish a trust fund to assist in maintenance of the computers and facilities over time. TILO staff has indicated that plans are underway to open TILO computer labs for community use for a fee. No schools visited by the team currently have such a program, although one principal had a plan in place and was hoping to be permitted to implement it in the near future. Gender equity There are no specific gender targets for the TILO project. However, TILO’s expansion plan perhaps could help in this area by targeting schools in underserved areas with high girls’ density, in order to bridge gender and digital gaps simultaneously. Curriculum development A major TILO achievement in this area is the Digital Resources package that the project has prepared for teachers for class use. Classroom observations in some TILO schools indicate that the package is a useful supplement to subject-specific curricula. Teachers and students reportedly are enthusiastic about the Digital Resources CDs; however, in teachers' group interviews in different governorates, the team heard recommendations for more: sample lessons, especially for language classes (i.e., Arabic and English); sample lessons in different subject areas; and training to help teachers to effectively integrate digital resource material into their lesson while keeping a balance between use of digital resource material and meeting lesson objectives. In general, teachers seem, at this stage anyway, to be having difficulty doing everything that they need to do, such as exchanging class visits with their master teachers, preparing new kinds of class lessons, and keeping up with their teaching loads. 3. TILO: Efficiency, Effectiveness and Sustainability Efficiency While TILO has benefited from the experiences of previous projects, its efficiency can be maximized by carrying out its plans to build on the successes of technology-based interventions such as the NSP experience in operating school-based ICT centers opened for public use. NSP trained schools on social marketing and micro finance since those schools charged user fees to support their technology centers and increase community understanding of the value of technology in education; such training provided by TILO would also be beneficial. Effectiveness Although TILO’s prime goal is to improve teaching and learning by using technology as a means to facilitate active learning, this goal is still not clear to some beneficiaries at the school level. Some confusion was evident as to whether technology is an end in itself or a means to quality education, particularly in those schools that experienced a gap between training and delivery of the technology. This lack of clarity lessens the effectiveness of the project. Sustainability TILO still has some work to do on helping bring the two technology entities at MOE, the Technology Development Centre (TDC) and Educational Computer department, to the point where they and the technology units at mudireya and idara level will be able to ensure sustainability of the model after the end of the project. Coordination with those entities is taking place and relations are good, so the groundwork has been laid for bringing this about. As the project gears up to expansion, it will be important to further integrate the model into the educational system, and to continue to harness the successful partnerships the project has already made with the private sector. Furthermore, TILO could take advantage of the enthusiasm about USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 24 technology to affect policy change with respect to the appropriate use and management of technology in schools in support of improved learning outcomes. TILO is an important, even key, project for USAID and Egypt in demonstrating that educational technology is now at a stage where it is capable of providing direct, as well as supplemental help to teachers and students in achieving higher levels of learning. The jury is still out, but a promising start has been made. FINDINGS: CROSS-CUTTING AREAS Training Models USAID has requested that the evaluation team assess the different training models used by the projects reviewed in this evaluation. The following responds to that request. The USAID projects reviewed all have successful models for teacher training. However, they have several variations in their use of the key elements of the way training is delivered. Although most interviewees mentioned positive changes in student attitudes, attendance and success rates in passing to the next grade, the real test of successful professional development is improved student learning. In the absence of CAPS or other test results across projects, the team has reviewed these models by comparing them to generally accepted best practices in successful teacher in-service programs.7 The team believes that all of these are applicable and desirable in the Egyptian context.8 1. Successful elements in Teacher Professional Development Programs As with teaching, it is not possible to identify one successful “size fits all” model for teacher training. For example, direct training with in-class coaching might be advisable at the beginning of a training project, but in time a gradual change to TOT support provided by in-school trainers, senior school staff and idara supervisors should be workable and necessary for sustainability. Many schools have been involved in other reform programs, and there are other variations in the circumstances of given schools or idaras that should determine the particular training approach to be used at a given time. Training needs to be tailored to the situation. Successful elements of training are listed below: • Initial training is most successful when it is provided directly by experts to teachers, for a substantial duration, including follow-up and coaching in class on a regular basis. Effective features include: o Training must model the methodologies that teachers are expected to use. o It should include a focus on content—how to apply the methodologies taught within the content teachers are teaching. (That content might include early grade reading and writing, or subject areas at upper grades). o It is beneficial for teachers to attend training with their peers (from their school, or in the same subject or grade level). This encourages the development of a shared culture and teachers’ open reflection about practice with their peers. o Successful training should also include opportunities (and time specifically scheduled) to work with peers in school, in peer observations, teacher circles, etc. Again, the goal is to stimulate reflection, break the habit of teacher isolation, and create a culture of sharing. Peer observations do not always mean visiting a senior teacher; rather, a pair of teachers 7 Garet, M.S., Porter, A.C., Desimone, L., Birman, B.F. & Yoon, K.S. (2001) “What Makes Professional Development Effective? Results from a National Sample of Teachers,” American Educational Research Journal, 38(4), pp. 915-945. 8 It is important to note that the assessment made is based on training documentation provided to the team by the projects, and findings from the interview data. More time and resources would be needed for a complete review. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 25 who are learning something together can visit to help each other work on the new methodologies. (See Annex IV for a framework for peer observation conferences.) o Teachers being trained need supportive supervision from supervisors and principals, who should not only attend the teacher training, but need training on their own in the provision of technical assistance to assist in applying the training in the classroom. • For gradual preparation for sustainability, the TOT model should include: o Teachers trained to become trainers must be experts in practice and need special training on how to train others. If more is invested in the trainers, they can then also serve others in the idara. This is human resource capacity building. o The teachers delivering TOT training need ongoing supervision for a period of time from master trainers from the project, as well as support from school administrators and idara supervisors. o Training materials should be provided to trainers in packages that are easy to use and to duplicate. o The PD plan should be part of the SIP and thus secure funding from BOT or other school resources, rather than expecting the teacher trainers themselves to pay for photocopying or needed materials. o The trainers need incentives, such as a step in the cadre, or becoming certified trainers. • For long-term sustainability, there must be systems in place to provide teachers and TOT trainers with ways of keeping their skills updated. These might include: o Partnerships between schools and local FOEs: professors as consultants o Periodic training provided on key themes by MOE o Establishment of professional networks In Table 2, the training models of three USAID projects are compared to the above framework.9 It should be noted that, although ERP1 provided valuable training support, as was attested to by many interviewees, it used several different training models at different times and for different purposes. For that reason, it was deemed advisable not to include ERP1 training in the chart. Four models are included: NSP Phase 1, NSP Phase 2, GILO and TILO. A review of the chart indicates the following: • NSP: The first phase had most of the desired elements, but in the second phase not enough time was given to the TOT, nor was there enough supervision of the trainers. • GILO: Many of the elements of successful programs are present. There is some doubt as to whether the initial training was long enough; the coaching model in GILO is especially strong. It appears likely that the teachers providing GILO training may not be receiving adequate ongoing support and supervision from project and MOE staff. • TILO: This model also contains many of the desired elements. Only three days of TOT training are provided, which is quite possibly not long enough. Not all relevant supervisors were trained by TILO, and this has created difficulties for teachers. 9 Note that this table shows only the existence or non-existence of desired elements in each training approach. Assessing the effectiveness of those elements as adopted by the project would require more in￾depth evaluation; however, the above bullets under each element could be used as guidelines. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 26 Table 2: Features of Effective Teacher Training in Some USAID Projects Desired elements GILO TILO NSP1 NSP2 1. Initial training (direct) x x x a. Appropriate duration uncertain x x b. Coaching and follow up x x Follow up Follow up c. Collective participation x x x x Features: • Modeling desired methodology x x x x • Supportive system in place: involvement of sufficient supervisors and principals in training x partial x • Training supervisors and principals in supportive supervision x partial x ? 2. Gradual preparation for sustainability a. TOT (Teachers trained to be trainers) x x (3 days) x b. Experts in practice are chosen ? x x ? Features: • TOT includes methodology and content x x x x • TOT includes strategies for training others x x x x • TOT providers receive supervision/support from project, idara and school administration partial partial x partial • TOT providers receive appropriate incentives 3. Long–term sustainability Desired elements: • Partnerships with FOE x • Training systems (update) • On-going supervision and follow up (idara, principals other school administrators) partial partial partial • Regular provision of TOT to novice teachers SBTEU TOT trainers SBTEU SBTEU • Certified pool of trainers exists informal informal informal informal • Leave behind TOT package and training materials ? ? Note: a “?” mark indicates that the team did not have enough information to judge if the feature is present or absent; an “x” means the feature is present. Other judgments are spelled out in words (i.e., partial, informal, uncertain). If no mark is included, it means the feature is not present in that model. Efficiency, Effectiveness and Sustainability 1. Efficiency Efficiency is an important dimension in project evaluation as it reflects many facets in project inputs and implementation. Projects included in this evaluation have demonstrated various levels of efficiency in managing and utilizing project resources (e.g. funds, time, efforts, existing experience, human resources, etc.) to produce results that attain project goals and objectives. Some of the projects evaluated were efficient in making utmost use of project time and collective efforts in creating a culture conducive to education reform, making significant contributions to the policy and systemic changes that the country has witnessed over the last five years or so. Moreover, other projects were efficient in making use of existing organizational mechanisms in reaching out to different groups of stakeholders whose buy-in was very significant in effectively carrying out project interventions. Equally important signs of efficiency were noted in some projects’ adoption of successful business models such as partnerships with the private sector, creation of school revenue generation mechanisms and clustering, which have all contributed to maximizing projects’ inputs, outputs and intended outcomes. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 27 However, from a results-oriented perspective, the efficiency of some projects could be questionable when comparing project accomplishments to level of expenditures and consumption of resources. Although it was outside the scope of this evaluation to review project budgets and expenditures, field work allowed for the observation of some examples of inefficiencies related to managing project resources. These include the production of materials that went unused or under-used, high￾cost activities with little return, poor coordination with existing experiences in the field, under￾utilization of project trained resources, and lack of inter- and intra-project synergies. The restructuring and refocusing exercises that occurred in some of the projects could have contributed to inefficiency since, in addition to the extra time and effort expended by staff members in changing focus, certain project components were dropped after high investments had been made. Efficiency can be improved when realistic mandates are set, expected results are agreed upon, and accountability for results is measured against utilization of resources. 2. Effectiveness An important indication of the effectiveness of teacher training is whether the teachers trained are applying the new methodologies in their classrooms. This section will focus primarily on the results of the 24 classroom observations conducted by the team in schools where teachers had received training from USAID projects. An important caution to bear in mind is that this is an extremely small sample; however, the sample was large enough for the team to see some strong tendencies in the qualitative data. It should also be noted that classrooms observed were not selected randomly.10 The observation instrument11 used in this study consists of a list of 15 Best Teaching Practices observed in the classrooms of effective teachers and most often emphasized in teacher training. Also, observers recorded further detail about each lesson, such as types of questions asked and the nature of teacher-pupil interactions. More information was also gathered during the one-to-one interview the team conducted with each teacher observed. (See Annex XI for the classroom observation and interview instruments.) Table 3 below presents data from the classroom observations undertaken by team members in 24 project classrooms in eight governorates. Observed practices are categorized here in three groups: the first group consists of practices that were observed in most of the 24 classrooms. These include: Having materials ready (24); Evidence of planning (23); Effective climate (23); Use of grouping (21); Classroom displays of printed materials and/or student work (21); A majority of students involved in the lesson (20); and Use of corrective feedback12 (19). These are important aspects of successful classrooms. A second group of items was selected because they are particularly sought after in training, but only about half of the teachers were observed using them. They include: Questions to encourage critical thinking (13) and Teacher modeling of task the students are to do (12). Thirdly, almost no teachers were observed using these two items: Working with students at different levels (3) and Providing Assignments to students of different abilities and needs (2). 10 The sample, however, included schools with different levels of performance given that implementers were asked to provide a list of schools classified by relative performance or other classifications from which the sample could be drawn. Within the purposive sample of 10 schools per project selected by the evaluation team for site visits, each project with a teacher training component was instructed to schedule six classroom observations. Teachers observed in the school sample were therefore selected by the projects and notified of the visits, which allowed them to prepare their lessons for observation (see Annex VII for a full description of the methodology employed in this study, including sampling). 11 The instrument used has been piloted and field tested in a number of similar evaluations carried out by JBS/Aguirre and was reviewed side-by-side with SCOPE and modified accordingly to ensure that the instrument aligned well with the classroom observation instrument developed and used by the Mission. 12 Corrective feedback refers to the response a teacher gives to a student who has provided an incorrect answer, to help him understand his error and arrive at the correct answer. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 28 (Interestingly two teachers who were observed doing both items were NSP multi-grade teachers, who work in settings in which these skills are particularly required.) This finding is important because in their interviews many teachers across the projects requested help in dealing with students at different ability levels or of varying needs, most particularly in reading and writing. When asked how they met the needs of such students, teachers invariably referred to their efforts to provide help before or after class, never in class. Most teachers are working with heterogeneous groups of students; future training is needed to help them learn how to work with a small group of selected students while other students engage in their group work, and how to provide differentiated assignments for some students. Research on highly successful school systems has found that such systems provide substantial training on how to deal with students of differing needs and abilities.13 The team understands that differentiation of instruction has not been a focus of training. The SCOPE instrument was reviewed to ascertain whether differentiated instruction was contained in any of the rubrics. It is present in the rubric for Item 6, but only at the highest level. Reference from SCOPE:14 6. Actively ensures the participation of all students in learning activities irrespective of their sex, achievement level, special needs, giftedness and other differences Rubric for SCOPE Item #6, at highest level (5) The teacher engages all students in learning, and provides planned opportunities to support students with special learning needs and provide gifted students with more challenging learning tasks. The engagement of all students is planned and involves significant learning activities and tasks. All students are involved irrespective of their sex, achievement level, socioeconomic status, and other differences. Additionally, special provisions are made for students with special learning needs and gifted students. In general, differentiation of instruction to meet the varying needs of students is the most difficult thing for teachers to learn, because it requires them to understand the developmental stages of their pupils, to have the ability to plan lessons at different levels, and the time and the resources to do such planning and prepare needed materials. Still, it must be emphasized that in order to provide quality education for all students, it will be important for Egyptian teachers to learn these skills. Most of the teachers were using grouping and most of the students observed in group activities appeared to be interested and participating. However, comments by the observers indicated that classes were still predominantly teacher-centered. Teachers asked most of the questions and often did most of the talking. Classes were teacher directed, in that teachers directed the students to do certain tasks in their groups and asked them to report on their results, and then moved on to other activities. Students were observed asking questions in some classes, but, for the most part teachers asked the questions and students answered. Indications are that many teachers equate active learning with grouping, and understand active learning principles only in a superficial way. It has been observed in many countries that teachers improve gradually, and that it takes time and continued support for them to develop a deeper understanding of the principles of active learning methodologies. It was particularly encouraging to note that a majority of the students was involved in most classrooms; this indicates an effective use of time as well as the use of strategies that effectively 13 McKinsey and Company, 2007. How the World's Best Performing School Systems Come Out on Top, available at : http://www.mckinsey.com/clientservice/socialsector/resources/pdf/Worlds_School_Systems_Final.pdf 14 Standards-Based Classroom Observation Protocol for Egypt: Rubric (English), Section II: Fostering a Collaborative and Equitable Learning Environment, Rubric 6. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 29 motivate students, and considerable research exists showing that student learning is highly correlated with time on task, the time that students spend actively engaged in learning. Teachers, for the most part, gave appropriate corrective feedback, so that students were able to arrive at the correct answer, or at least to understand what the correct answer was. About half of the teachers observed asked at least some questions that required more than literal answers. However, further training on effective question-asking is recommended, especially because of the focus on critical thinking in the educational reform process. Teachers also used modeling only about half of the time, and, of course, some lessons were observed in which there was no task assigned that the teacher could have modeled. Nevertheless, modeling, or demonstrating to the students how to do the task they are assigned, is an important component of effective teaching. It was also observed that students did very little writing in most classes, usually just a few words or sentences at most; however, it should be noted that several classes observed were in science or math. Some teachers did assign written work for homework. In addition, team members noticed an almost total lack of integration of subject matter. This had previously been emphasized in NSP, but teachers commented that the Integration Guide prepared in NSP is now little used. In the TILO program classrooms, it was observed that teachers are generally at early stages of development in their understanding of how to integrate computers effectively into their curricula. Some were using the computer as a substitute for a blackboard, while others had designed lessons using the computer to provide experiences that would otherwise be difficult to simulate in class. Overall, it is to be expected that this development will take time, and teachers will need continued support for optimal development in the effective integration of technology. Table 3. Comparison of Teaching Behaviors More and Less Frequently Observed in Schools Where Teachers Received Training from USAID Projects Item ALL N: 24 GILO N: 6 NSP N: 5 ERP/NSP N: 2 ERP N: 5 TILO N: 6 2: Materials ready 24 6 5 2 5 6 1. Evidence of planning 23 6 5 2 4 6 13. Effective climate 23 6 5 1 5 6 8. Grouping 21 6 4 1 5 6 15. Displays of printed material and/or student work 21 6 5 2 2 6 12. Students involved 20 5 4 2 4 5 11. Corrective feedback 19 5 4 1 3 6 7. Questions to encourage critical thinking 13 4 3 1 3 2 4. Teacher models task 12 3 3 2 3 1 9. Works w. groups at different levels 3 1 2 0 0 0 10. Different assignments based on students’ needs 2 0 2 0 0 0 As previously mentioned, the data were also sorted by governorate and by primary and preparatory levels. A review of the data sorted by governorate did not reveal any noteworthy differences. However, when the data were sorted by primary or preparatory levels, it was somewhat surprising to note that the highest percentage of classes using grouping (100%) was at the preparatory level. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 30 Also, all of the preparatory teachers were reported to have a majority of students involved for most of the lesson, and a higher percentage of teachers in preparatory classrooms and in multi-grades asked questions that might elicit critical thinking, as compared with teachers in primary classrooms. On the other hand, in comparison with preparatory level teachers, primary and multi-grade teachers were more likely to model for students the desired task to be performed, and a higher percentage of them also gave corrective feedback to students. It should be noted that in all schools visited there was great concern about the large class sizes, which in many schools are approaching 60 students per class, even when schools have changed to double sessions.15 Teachers commented that it was difficult to apply the new methodologies with such large groups, and in TILO schools, that there were too many students per computer when a class went to the computer lab. In addition, teachers often mentioned the need for more supplementary teaching materials and other resources for the larger classes. 3. Sustainability The pursuit of continuing flows of development activity benefits to intended populations beyond the completion of interventions has been at the core of a number of the projects reviewed in this study, although more successful in some cases than in others. Rather than repeating what was discussed in the project sub-sections, this summary takes a broader view through the following sustainability lenses: social, institutional, financial and political. A number of the interventions reviewed here include strong community development components that sought to build the capacity of community members to take part in education reform and to become a driving force of social sustainability of project efforts. Examples include the work with BOTs, PAs and CETs. However, although these bodies include parents, results of fieldwork indicate that more effort needs to be put into engaging other parents as co-participants in school activities and education reform. Further harnessing the potential of parents and community leaders to become a voice for improved education is an important opportunity for current and future projects on the path to social sustainability of reform efforts. Overwhelmingly, projects are working through existing systems to institutionalize supported reform practices at school, idara and mudireya levels. Examples include the work with QAUs, TSUs, SBTEUs, school improvement teams, and the support to existing or project-established, reform￾oriented professional development structures. There are different degrees of success in these spheres and in some cases continued support will be needed to achieve sustainability of efforts. At the central level, there are a number of project-initiated initiatives (standardized testing, strategic planning, PAT, just to name a few) that are at varying levels of institutionalization, although sustaining buy-in and closing the gap between policy and implementation will continue to influence sustainability prospects. Finally, the notion of sustainability also rests on political and financial considerations. Continuing long-term effects after initial funding ends will require leadership, political will, and financial resources on the part of stakeholders at the school, idara, mudireya and central levels. Continuing to build support and gaining buy-in at all levels will be key, especially at the lower levels where political changes are less frequent than at higher levels. At the central level, moving away from working with individuals to working with institutions at all levels can be instrumental in ensuring survival of interventions through political changes and in pushing forward the ground-breaking project￾supported reforms initiated. 15 It should be noted that in most instances, team members did not actually observe these high class sizes, because schools were still operating on special alternating attendance systems put in place because of the H1N1 crisis. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 31 Project Design USAID has requested that the evaluation team provide it with comments and suggestions on its project design practices. The following responds to that request. • “Christmas tree” projects are rarely a good idea, as projects usually are more successful when they are tightly focused on clearly defined and related goals. Both ERPs started out with a collection of tasks not closely related to each other or to the core purposes of the project and it was a couple of years before the situation improved. This was particularly harmful because it worked against close cooperation and communication between the two projects, which had more on their hands than they could handle, at least at first. GILO’s original scope was also quite spread out and the targets were unrealistic for such a short project. • Frequent changes in the scopes of work and assignment of a series of additional tasks, such as ERP1’s training support of the MOE’s accreditation goals, over the lives of the projects, however justified, also affected the productivity and focus of the two ERPs. • Anytime the original purposes or context of a proposed activity changes significantly before the procurement process is over, as was the clear case with GILO, the Mission should pause and consider starting over. GILO went to contract with an outdated design, an unrealistic set of targets, an inadequate focus on gender in its training model, and too short a time frame. The project went on to demonstrate that any temporary advantages gained by moving ahead with a deficient design would be more than offset by the long-term grief and extra work entailed in putting things right. • It is important that project design teams first do a reality check on the validity of basic program concepts, relationships, and the general feasibility of the project before they start their work. • As GILO again illustrates, this may not always be done in the earlier stages of projects, even if it should be, especially when the staff is small, the workload large, and there is pressure to move quickly. Even if the purpose and objectives of the new project are clear, there often is a need for follow-up work to contact the key players and institutions involved and verify that they understand the project, are on board with its main features, and that the project can count on their participation and support when the contractor team arrives. Such follow-up work might have helped the Mission to avoid the major disjunction of project design and budget with the ultimate scope of work involved in implementing the STEAP project. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 32 CONCLUSIONS AND RECOMMENDATIONS Conclusions USAID has made many positive changes in Egyptian education over the last 10 years, with an array of projects that have contributed to girls’ education, to the application of active learning methodologies in classrooms, to the use of technology in schools, and to educational reform at all levels of the MOE. Although there are always problems to be noted and improvements to be recommended, the team has been impressed by the positive reactions of students, teachers, and parents to the USAID projects. In addition, school administrators and idara and mudireya personnel report improved practices as a result of USAID capacity-building efforts, and extraordinary strides have been made at systemic policy levels with the assistance of USAID technical support. The conclusions that follow first address current projects, the team’s overall assessment of them, and suggestions the team has for USAID and the projects regarding program changes to be made between now and their termination dates. Following these short-term conclusions are conclusions relating to future program strategy. The same sequencing is used in the recommendations section. The team, based on the total data collected, believes that the prospects for significant additional progress on implementation of Egypt’s National Strategic Plan for educational development over the next five years are excellent. It bases that conclusion on the continued high level GOE support for reform, the degree to which that and widespread public support circumscribes any attempts to reverse or revise the plan in ways that would diminish it, and Egypt’s manifest need for the kind of educated, innovative, and skilled human resources needed if the country is to compete successfully in the global economy. The team’s investigations, moreover, indicate that the GOE and MOE’s openness, even eagerness, to receive continued USAID assistance, including involvement in strategic policy development, continues undiminished. In addition, it is fair to say that, as budget pressures and changing priorities are diminishing the involvement of all other significant donors in basic education, except the EU, USAID is needed more than ever. Finally, USAID, over the years, has invested heavily in basic education in Egypt and, as a result inevitably seeks to capitalize on that investment, both for its own sake and to ensure its legacy in this critical sector. For all these reasons, the team strongly concludes that the work should be continued, with the highest priorities being to finish the systemic reforms process initiated under the current strategy and to progressively scale up the best practices of its school-based reform efforts over the last decade on a national scale. The results of the team’s formative evaluation of six basic education pilot projects confirm, overall, that a strong basis for such scaling up has been created. Main Findings by Project NSP is considered by many to be the example of an exemplary project that paved the way for future projects by engaging in excellent practices to involve communities and parents, by providing education in areas where girls had not been attending, and by offering long term direct training of teachers with strong follow-up. Phase two training was briefer and used a TOT model that appeared to be less effective. Nevertheless, findings indicate that NSP practices are being sustained, especially where there is strong leadership. Staff turnover has been high, but the SBTEUs are instrumental in training newly contracted teachers in several of the schools visited. The NSP trust funds have been important for sustainability. ERP1, though subjected to several iterations in its program design and changes of components, has provided a wide variety of training that has been perceived as useful by school administrators and staff, teachers, parents and students. In addition, its provision of girls’ scholarships and involvement USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 33 of NGOs in school communities was mentioned with appreciation by school staff and parents. ERP’s reading and writing initiative was seen as very much needed. Most recently, ERP1 has taken over several tasks from ERP2, and has expanded its capacity-building training at the idara levels to all governorates. It is not clear that this latest effort will prove to be sustainable. ERP2 focused on policy reform, working closely with the MOE to assist in implementation of an extraordinary array of new policies and initiatives. These included the development of national standards, the development of a professional career path for teachers (the Teacher’s Cadre), the creation of three instruments to use in assessing teacher skills (SCOPE), student learning (CAPS) and school management (MAP). ERP2 also assisted in planning for fiscal decentralization and in the initiation of a school accreditation system. These and other reform initiatives assisted by ERP2 are unprecedented in the Egyptian education context, although some have been more successful than others. STEAP was an awards-based program that mobilized schools and communities, leaving the winners very happy, and the losers often resentful and bitter. The strong community involvement was a surprise and one of the most successful aspects of the project, as some communities raised large quantities of funds and in-kind contributions. Schools reported that participation in STEAP helped them understand how to prepare for accreditation. The project was beset with problems, however, including changing requirements for numbers of schools that could participate, and changing rules about who should receive prize money. In the long run, the cash rewards are not sustainable, though some suggested other possible incentives for any such program in the future. GILO was intended as a project that would succeed NSP, but it had a rocky beginning, with the non-release by the MOE of funds for construction of schools for project intervention, and location of the project in some schools where girls’ access is no longer an issue. Although its focus on gender issues has been a bit weak, its training program is sound, as is its community participation component. The project has introduced EGRA and is reportedly working with the MOE to develop materials to supplement the curricula for early grade reading. This initiative promises to meet a great need. Since ERP2 closed, GILO has also continued with the decentralization work formerly done by ERP2. TILO is a technology project that has created much enthusiasm by offering computers and other technology to schools. It has an innovative training program that includes both active learning methods and the use of technology in classrooms. While TILO’s training approach is strong, it is questionable whether the TOT model provides enough training and support for teachers who are intended to serve as trainers in their schools. In classrooms, it was observed that teachers are using technology but are at an early stage in their efforts to integrate technology into classroom work. Such efforts will take time. TILO has successfully forged public-private partnerships but has not yet worked closely with communities and generally has not worked with school SBTEUs. Support for the project from MOE and MOE technology departments is strong, but coordination with the latter can be improved. Overall, the team found that a climate for change has been produced by these projects. Following are summaries of lessons learned and best practices: Lessons Learned • Leadership: Projects are successful and efforts are sustained where there are strong leaders. • Supervisors: It is essential to train and work closely with all supervisors who will be working with teachers in a project if efforts are to succeed and become sustainable. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 34 • Community involvement: Communities are a strong force behind education reform. However, community mobilization efforts need to go beyond engaging BOTs, to seeking true parent and community involvement in schools. • Partnership with FOEs: FOEs seem to be the missing link; they could be providing quality training in active learning methodology to undergraduates, and could also serve an essential role in providing refresher courses and TPD over the long term for school staff. • NGO involvement: NGOs involved in schools and communities remain there and are important for sustainability • Communication: It is vital for USAID projects to be communicating and cooperating with each other, sharing best practices and solving problems together. A mechanism should be in place to encourage such cooperation. For example, USAID could schedule regular meetings as well as workshops in which project implementers working on similar components could meet and share ideas and work on solutions to problems. Best Practices • TILO’s work with the private sector shows that public-private partnerships can work. • Training approaches that provide adequate direct training, accompanied by coaching and ongoing follow-up are proving successful in changing teacher behavior. Shortcuts (shortening training, not providing follow-up), which are often undertaken in order to increase the number of trained teachers, simply do not work as well. • It is important to work within the system wherever possible, at all levels. Any practice incorporated into the system has a better chance of sustainability. • Building on existing successful models is both effective and efficient. Short-Term Recommendations Based on the team’s review of the projects currently being implemented and the findings regarding each project presented above, the team makes the following short-term recommendations, presented below by project: GILO 1. Extend the project. This would allow needed time to (i) achieve project objectives; (ii) continue the critical work started on reading and writing (for which coordination with ERP1 and its reading and writing program is recommended to avoid duplication and to take advantage of the program’s experiences and lessons learned); and (iii) consolidate and expand efforts on the decentralization front. 2. Refocus the gender approach to address current issues (access seems to be a “localized’ problem; current issues, as indicated by interviewees, include dropouts, and the increasing gender gap in higher grades.) 3. Rethink and/or modify the plan to reach schools in the clusters, to ensure that this activity builds in enough time and resources to provide adequate training and follow up that can in turn lead to sustainability of this effort. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 35 TILO 1. Establish more effective involvement with technology bodies at the MOE (the Technology Development Centre and the Educational Computer Department). Although the project has strong relationships with these entities, it would benefit from more in depth collaboration, especially in the context of expansion in which these entities can and should play a larger role. 2. Train sufficient supervisors who can in turn be tapped for coaching/supportive supervision in expansion schools in addition to providing technical support to teachers. 3. Work in close cooperation with SBTEUs to increase prospects of sustainability. 4. Reach out to the community as partners in harnessing the potential of technology in education while also carrying out existing plans to extend the benefits of such resources to them. 5. Consider putting in place a sustained coaching system to assure that teachers understand and apply what they have been taught, particularly the training related to integration of technology into the classroom. 6. Consider introducing more content-based training, increasing software options in early grade reading and writing, and providing more software in Arabic. ERP1 (reportedly has asked for no-cost extension) 1. Support a no-cost extension that will offer the project crucial time to (i) consolidate the work with PAT; (ii) provide targeted technical assistance; and (iii) advance the institutionalization of the instruments SCOPE, MAP, CAPS in the education system. Future Strategic Program Conclusions The evaluation team’s work has validated the basic USAID decision, taken in 2004, to work simultaneously on policy and other systemic changes, as well as school-based reforms, in the quest for improved learning outcomes. In spite of the sometimes rocky road followed in implementing the decision, it opened the way for USAID to develop programs for the next strategy period that will have major impacts on educational quality and student achievement. The team’s conclusions on what that future program strategy should be can be divided into two parts. The first part comprises needs and opportunities to finish the systemic changes begun under the current strategy through continued provision of technical assistance, training, and other support in remaining key policy and implementation capacity areas. These include: 1. Teachers: PAT; in-service teacher training and professional development; the teachers’ cadre; and changes in teacher deployment policies and practices to reduce excessive turnover. In addition to these continuing activities, the team believes that teacher improvement efforts need to be supplemented by renewed USAID involvement in the area of teacher education. 2. Organizational and other structural reforms in MOE: Implementation of proposed functional decentralization; creation of an education managers’ cadre, similar to the teacher’s cadre to raise standards and increase professionalism; review of human resource policies and practices with a USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 36 view to developing a plan to reduce overstaffing and open up opportunities for the Ministry to recruit and develop a new generation of education leaders for its work. 3. Improvement of learning outcomes, particularly in reading and writing, in the early primary grades, since current learning difficulties undermine the potential for further learning and contribute to the still too-high incidence of effective illiteracy in higher primary grades and preparatory schools. USAID projects have addressed this issue in various ways, but a clear, strong national strategy and approach, with the power of the MOE behind it, is badly needed. In this regard, it is essential that teacher deployment practices focus on staffing the first three grades with the best teachers, since the foundations laid in those grades affect all future learning. 4. A need for further review of current curriculum and assessment systems to create a better match between the curriculum and the learning capabilities of children of different ages and of differing needs and ability levels, and to simplify and focus the assessment system on critical learning measurements. 5. The need for an overall review of MOE educational technology programs and approaches to unify and focus them on improved learning, including their use in classroom teaching. The technical capacities and experience of TILO would be critical in this effort and a logical extension of the project’s current policy work in the Ministry. The second part of the team’s conclusions goes beyond policy and structural reforms to the sphere of action. As anticipated at the beginning of this section, it is the team’s conclusion that USAID’s extensive, past and current involvement in basic education has produced an array of tested best practices that, with the possible exception of reading and writing programs, is now sufficient to inform and guide a vigorous but realistic, phased process by the MOE, assisted by USAID, to scale them up to the national level. Broad details of a proposed USAID project to support such an effort are included in the recommendations section. In arriving at the scaling up support recommendation, the team considered a range of other possible action possibilities, but decided that, while all would be useful, they would not match the opportunities offered by the scaling up proposal, particularly as regards achieving maximum return on investments made over the last decade. The options considered, for the record, include: • Limiting future support to technical assistance, including possible additional small pilot projects in decentralization and MOE restructuring and modernization. • Basing a scaling up strategy on existing projects, especially TILO. • Developing a new project focusing on early education. • Some combination of the above. Recommendations for Future USAID Program Strategy The evaluation team recommends that USAID’s future education strategy focus on two core objectives: 1. Support the implementation of the Egyptian National Education Strategic Plan. 2. Maximize the return on 10 years of USAID investment in educational reform in Egypt. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 37 The first objective reflects the team’s considered view that the prospects for making significant additional progress on achievement of the National Strategic Plan’s objectives are good or better and that continued USAID support is vital to achieving that goal, particularly to complete the systemic changes needed to raise quality at all levels and, thus, create an environment in which improved learning outcomes can be achieved across the system as a whole. The second objective recognizes that USAID’s education programs since 2000, through a series of pilot projects, have focused on school and community-based education reforms. These, collectively, provide a wide range of experience and best practices sufficient, in the team’s view, to provide the basis for moving the Mission’s strategy from supporting school-based reform pilots to using the accumulated knowledge gained, while also benefitting from the improved environment created by past and proposed future systemic changes, to scale up the experiences on a wider scale across the country. These best practices include: 1. Concentration on training and mentoring school and idara leaders, including principals, SBTEU heads, master trainers and teachers in the schools and directors, supervisors, and TSU and TPD heads in the idaras; 2. Where a strong school exists to assume the leadership, adopting a cluster training and mentoring approach; 3. Teacher training models that emphasize long-term coaching and follow-up of trained teachers; 4. Creating a culture of school-idara partnerships; 5. Strong BOT and community involvement to monitor school improvement and generate local and private sector support; 6. Involvement, where feasible, of local NGOs; 7. Partnerships with faculties of education and universities in the governorate. Three intermediate objectives are recommended. 1. Working with the MOE to fine-tune the enabling policy and operational framework for educational development by: a. Developing a plan to make curriculum modifications to reduce the size and level of content to create a better match between the curriculum and the learning capabilities of children of different ages and differing needs and ability. b. Similarly, adapting the comprehensive assessment system to include both formative and curriculum-mastering measurements and to avoid over-loading teachers with excessive reporting requirements. c. Making attainment of greatly improved reading and writing competencies in the early grades a clear GOE and MOE priority and strengthening links between teaching and learning goals of pre-primary and primary schools. d. Developing an administrative cadre for MOE staff and educators to supplement the teachers’ cadre. This would be a critical first step to addressing broader human resource quality and over-staffing issues in the Ministry. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 38 e. Developing and applying better ways to apply the power of technology system-wide to increase the quality and efficiency of teaching and learning. 2. Continuing to help the MOE to address systemic problems still inhibiting sustainable school-based reform, of which the following are viewed as highest priorities: a. Inadequate and inefficient teacher education, training, and deployment (to be addressed by PAT, the Professional Academy of Teachers). b. Poor learning outcomes in the early primary grades, which undermine the potential for learning and create learning deficiencies throughout the education process. c. Serious quality and implementation capacity weaknesses in MOE. d. Low quality and productivity of existing pre-service teacher education programs. 3. Working with the MOE to develop a strategy for major scaling up of best practices of USAID￾supported school-based reform models and conducted by replication teams operating at the idara level, utilizing, in part, training and other model assets produced by the pilot projects in schools and associated idaras. It is recommended that the strategy be idara-based, starting with two idaras in each governorate,16 with the goal of applying the replication strategy in all schools within each idara. A competition would be held to select the initial idaras. When all the schools in an idara have been reached, another idara (possibly the next ranking idara resulting from the competition) would be trained and brought into the program. When sufficient experience has been obtained to warrant it, the project would be further scaled-up, perhaps working with additional idaras in a governorate, and additional replication teams would be formed and trained by the original teams, as needed. The first step after selecting the idaras would be to strengthen them to a level which would assure success and provide a national model for future development of this critical cog in the education system. One element of the strengthening would be for the MOE to assign a few key, proven individuals to each idara and also to agree, as in the case of the pilot projects, to keep the staff in place for the life of the project. Once there is consensus between USAID and the MOE, it is recommended that, before launching, it be discussed and approved as well by the Democratic National Party's (DNP) Policy Secretariat and the Shura Council to ensure that the initiative has broad support in the GOE, thus guaranteeing that changes in the leadership of MOE, if they occur, will not jeopardize it. The following instrumentalities are recommended for implementation purposes: 1. The design and procurement of a single contractor to implement all components of the strategy. A contract is preferred to ensure that the implementer can provide the full range of skills required. Every reasonable effort should be made to assure the uninterrupted work of consultants already working productively with the MOE. 2. Technical assistance and in-country and participant training should be presumed to be required in all components. 3. The use of benchmarked cash transfers to the MOE in support of the scaling up project should be explored. 4. USAID should consider subsidizing a portion of the technical staff costs of replication teams. A possible formula would be to start at a maximum level of 50%, with the 16 An option here would be to pilot the effort in selected governorates first and then, based on experience, to take it nation-wide. Another possibility would be to begin the scaling up project in year two or three of the new strategy, allowing the contractor to concentrate first on the systemic changes needed. These and other design issues could be considered by the design team. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 39 percentage declining by 10% a year over a five-year period, i.e. 40% in year two and so on. 5. A Higher Education for Development (HED) university partnership or partnerships should be considered to help address the need for improvement of pre-service teacher education. It is recommended that the Mission request the visit of a HED staff member to spell out the options17 and, if desired, join with Mission staff in exploring possibilities with the Ministry of Higher Education (MOHE) and local training institutions and universities that might be interested in adding a teacher training program, perhaps in specific, high priority subject areas. Training of future education management specialists might also be considered. The proposed strategy would not foreclose the option of USAID supporting additional replication by existing projects, such as TILO. 17 One possibility would be to use the HED Leader with Associates mechanism to select a consortium of U.S. schools with international experience. The members of the consortium could be assigned to work with, say, three Egyptian institutions in different parts of the country, but under a single cooperative agreement. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 40 ANNEXES USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 41 ANNEX I: Egyptian Educational Context The strong commitment of the government to sector development has resulted in significant progress. Almost universal enrollment in grades 1-9 has been attained. Gross enrollment at the primary level is 96% and at the preparatory level, 93%. Gender parity has almost been attained in basic education (grades 1-9) with girls comprising about 48% of enrollments. A strong policy framework, the National Strategic Plan for Pre-university Education 2007-2010, is in place for guidance and public spending on education is high, 5.9% of Gross Domestic Product (GDP) and 19% of total government spending. Remaining issues to the sector are reviewed here. Access Pre-primary enrollment remains low at 24%. Enrollment at the upper secondary level (grades 10-12) of 76% remains the major access challenge in the system.18 Equity Inequality across income quintiles is lowest for primary enrollment. However, for secondary enrollment, a gap of 30 percentage points exists between the richest and poorest populations levels (74% and 44%).19 According to geographic and gender perspective, the greatest disparities in enrollment and attainment are seen in Upper Egypt. In research using 2005 data, only 40% of the poorest quintile of students completed grade 11 while 91% of the wealthiest fifth did. In comparison of males and females, 27% of girls versus 55% of boys from the lowest quintile completed grade 11 while 92% (girls) and 91% (boys) from the wealthiest quintile did. Only poor boys from urban areas were equally challenged to complete upper secondary school having made only 3 percentage points improvement in completion rates between 1988 and 2005.20 Quality Quality has been much affected by the fast growth in enrollments. The professional education staff consists of a high ratio of administrators to teachers, 0.7: 1. Teachers most often adhere to the rote teaching methodology which does not encourage critical and creative thinking of the students. Pre￾service training remains of poor quality and novice teacher deployment is in disregard to their teaching credentials. Salaries are low driving many teachers to participate in a rampant private tutoring system. There is no professional certification process for school administrators although a recent Teacher Cadre’ law should do much to improve professional development across the sector. Curriculum reform has been a recent focus of the government attempting to rectify the following identified weaknesses: traditional subject-centered teaching-learning processes and assessment techniques, poor use of technology, and failure to meet diverse interests and needs of students.21 Based on grade 9 exams, a tracking system divides students unevenly into academic (33%) and lower quality vocational (57%) streams which limits future education and career choices. Competitiveness to pass the national school-leaving exam at grade 12 drives the current in-class rote memorization method of learning. The education sector remains centrally controlled limiting authority and resources to local actors to address school-level needs. 18 Egypt Ministry of Education Indicators 2007/2008. 19 World Bank (2007) Improving Quality, Equality, and Efficiency in the Education Sector: Fostering a Competent Generation of Youth. 20 Langsten, R and T. Hassan (2009) Education Transition in Egypt: The Effects of Gender and Wealth. 21 Egypt Ministry of Education (2007) National Strategic Plan of Pre-university Education 2007-2012. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 42 ANNEX II: Development Environment in Egypt Development Environment in Egypt The environment for education development in Egypt and for USAID’s role in it is positive. There is strong, high-level political support and a clear commitment to transforming the country’s education system for urgent social and economic reasons. A National Strategic Plan based on internationally acceptable standards and processes has been completed and is serving as the lodestone of reform. And generally sound policies and initiatives have created a favorable enabling environment for change. On the other side of the ledger is a less than ideal capacity for implementation. USAID and other donors have been active in supporting the reforms by assisting in building key capacities and providing technical advice, a continuation of which is seen by the Government as critical to success. Obstacles and challenges to success nevertheless are significant. One of the most important is that in spite of encouraging recent progress in decentralization, the education system remains highly centralized. Reform of the Ministry of Education’s organization and technical capacities, another key ingredient for success, is proceeding slowly. Teachers’ salaries and status have been raised, but much remains to be done to create the corps of trained and committed teachers that the reforms will require. Nevertheless, compared with the situation ten or even five years ago, remarkable progress has been made and there is reason to expect that trend to continue. Other Donors The principal other education donors in Egypt are the World Bank (WB), the European Union (EU), the Canadian International Development Agency (CIDA), and UNICEF. They and other donors coordinate their activities through a Donors Partners Group (DPG), with USAID serving as the informal secretariat. Donor cooperation has been frequent, but donors do not seem to jointly approach the MOE on a regular basis to discuss common concerns. Some donors rely primarily on MOE relationships with key individuals, rather than on an institutional approach. Donor funds for education reportedly are decreasing, apparently due primarily to overall budget reductions. World Bank: The WB’s program in education’s two major activities– the Secondary Education Enhancement Project (SEEP) and the Higher Education Enhancement Project (HEEP) – have either ended (HEEP) or will soon end (SEEP in 2011). A possible new project in an early stage of development would address two key education issues. The first is the streaming of students into general and technical education at an early age, with lack of opportunities for the latter group to get back into the general education stream or other means of access to higher education. The second is the secondary school leaving examination, which reportedly does a very poor job of measuring levels and types of student achievement, as well as providing an adequate basis for judging students’ aptitudes and probabilities of success in different professional areas.22 No new loans appear likely, due to the GOE’s strong preference for grant aid. European Union: The EU is active in all levels of pre-university education, with the emphasis currently on technical and vocational education. Up to 2006, the focus was on basic education, funded through budget support, with conditionalities. In higher education, the current focus is on science and technology and basic research. The EU is the second largest provider of technical assistance in education after USAID. 22 The team recommends that USAID follow the Bank’s efforts to support reform in this critical area, which, if achieved, could have a significant, positive impact on the quality of future teachers and educators coming into the education system. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 43 Canadian International Development Agency: CIDA has been an active and significant player in Egyptian educational development, though on a smaller scale than the larger donors. Their involvement began as collaboration with UNICEF on the community schools program (Support to Egyptian Primary Schooling Project (STEPS1) in CIDA parlance) and continued, beginning in 2005, with STEPS II, a school-based reform program similar to ERP1 but with primary focus on the idaras. They are currently collaborating with ERP1 in developing a database of potential, local training providers for PAT, to use in designing a new in-service teacher training system. A companion project in Early Childhood Education (ECE) was started in 2006, with the goal of supporting the ECE component of the strategic plan, the focus on kindergartens. The main features of this program reportedly have been nationalized. Both programs are reaching their end and, reportedly, CIDA’s involvement in basic education will end with them, with emphasis in the future on small and medium enterprise development. UNICEF: Together with NSP and CIDA, UNICEF pioneered efforts to reach and educate hard to reach children, especially girls. UNICEF’s focus, starting in 1992 in a number of countries including Egypt, was on “community schools”, one room classrooms for children ages 6-14. This program concluded at the end of 2009, though technical assistance continues to be provided to MOE for scaling it up. Currently, they are supporting an ECE program focused on kindergartens initiated in 2007, in collaboration with MOE and the Ministry of Social Sustainability (MOSS). Project Culture: Over the years, the education development scene in Egypt has been characterized by a multitude of internationally-supported pilot projects, sometimes overlapping and duplicative. These projects are convenient for both the MOE and the donors, the MOE because they bring in resources and provide concrete, attractive opportunities for successive Ministers to sponsor initiatives that are responsive to the priorities of the moment, and donors because they constitute manageable pieces of the complex whole of education needs and can be viewed, usually legitimately, as making significant contributions. There is a stage in any developing country when pilots are needed, even crucial, for testing approaches and developing public support for them. But, too often, as in Egypt, they continue to come and go for too long a period of time, without firm decisions and programs to replicate their best practices on a national scale. Meanwhile, the talents of the usually high caliber Egyptian staff that they employ are denied to the education system. Challenges Efficiency: The pre-university public education system in Egypt is highly inefficient. As a result of years of using the MOE as a place to create jobs, the Ministry is seriously overstaffed, with a ratio of teaching to administrative staff of close to 1:1. The quality of education and the level of student learning are low, especially in the critical early grades. Literacy problems remain severe at all levels. Teachers’ low quality pre-service education, deployment policies, and excessive turnover exacerbate the problem, though there are hopes that the new teachers’ cadre, higher teachers’ salaries, and reformed training systems will ultimately have a major, positive impact. Money is not the real problem. Egypt already allocates a significant portion of its budget to education, but does not currently get its money’s worth. The solution rests in a range of systemic policy and management changes to increase educational outputs, most of them already part of the National Strategic Plan. Decentralization. Education finance and decision-making in the Egyptian education system are still highly centralized, although somewhat less so than when the reforms began. Decentralization threatens the power, prestige, and, most important, jobs of a large number of individuals. Commitment to decentralization remains a central part of the reforms, but it will take time and significant changes, including in MOE human resources policies and practices, for it to occur to the degree required to achieve the desired reforms. Implementation: The reform process to date has demonstrated that there are serious deficiencies in MOE implementation capacity at virtually all levels below the senior leadership. The problem is USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 44 not primarily one of resistance, but rather a lack of understanding and skills. One reason for this is that the majority of MOE’s staff is now in its 50s and, while it has a lot of experience in managing the traditional system is ill-prepared to lead a process of change to a whole range of new policies and processes. As a result of this and a general adherence to the principle of seniority, a number of key positions at any one time are filled with people fundamentally unprepared for the new responsibilities that reform requires. The situation is further exacerbated by excessive turnover of Ministry officials at all levels, including school principals, making it difficult to plan and execute long￾term activities. (Again, personnel policies and procedures require fundamental changes.) Governance: MOE establishes and manages change primarily through a system of decrees that, over time, has become a problem in itself. The problem lies in the fact that decrees often are issued without adequate preparation and central review to ensure their consistency with policies and each other. New decrees rarely repeal old ones, often leaving contradictions and uncertainty in their wake. Private Tutoring: The secondary school leaving examination, simply put, determines the future careers of students graduating from secondary education. The importance of the exam and the frequent inability of the schools to provide a level of education that prepares students for it has created an industry of private academies and tutors, many of them teachers, to assist students who can afford it to increase their chances of success. While this is fundamentally a national problem and requires a national solution, it affects the schools, as well as the dedication of some teachers to their jobs. Infrastructure: As a result of population growth and success in achieving high education access rates, there is a serious shortage of classrooms and buildings. This has resulted in class sizes in many schools, particularly in poor urban areas, that are too large for effective education, even with modern methods. Educational technology may offer some help, but it is unlikely to fully resolve the situation. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 45 ANNEX III: Matrix of USAID Strategies by Project USAID education assistance in Egypt supports sustained improvement in student learning outcomes by a) Improving quality of teaching and learning, b) increasing access to education, and c) strengthening school governance and management. According to the matrix above, recent/current education programming has successfully addressed all of the strategies. Several of the programs have had a very specific focus, as for example, the ERP 2, which operated at the policy and system level; therefore USAID was able to address policy objectives. The remainder of the programs was implemented at school level. STEAP focused almost entirely on the school governance and management strategy and made only small contributions to the first strategy through ICT/bases and supplementary materials activities. Although an ICT focus overlays all TILO implementation, its activity focus nearly mirrors that of the GILO project except for the very obvious lack of focus on increasing access to education through girl-friendly activities, which is the main focus of the GILO program. The ERP 1 has the most comprehensive array of activities addressing all three of the strategies. The GILO program, a continuance of the NSP, also implements an array of activities under each education strategy. Several of the most often implemented activities in the USAID portfolio focus on teacher training, ICT/databases, and supplementary materials activities to improve quality of teaching and learning. Increasing access to education was most often accomplished through the use of school construction, multi-grade classrooms, and community efforts. The focus of USAID decentralization efforts in order to strengthen school governance and management was most often done through training/ developing communities, training school administrators and supervisors, and identifying and training different levels of government administration. Matrix of USAID Strategic Objectives and Program Activities* ERP I ERP II GILO NSP STEAP TILO a) Improved Quality of Teaching and Learning Teacher Training: Active-learning Methodologies x x x x Development of TT Cadres/support system x x x Individual Student Assessment x x Curriculum Content x x ICT/databases x x x x Supplementary Materials x x x x CAPS Administration x x SCOPE administration x x x x Other: Literacy, ECDE, School-to-Work x Policy: Student Assessment/Testing x x Policy: Quality Assurance/Standards x Policy: Teacher Professional Development x x Policy: Technology x b) Increased Access to Education Scholarships x Gender-sensitive active learning. x Multi-grade Classrooms, School Construction x x x School Furniture/equipment x x Community Effort x x x c) Strengthened School Governance and Management Community/NGO training/development x x x x x Facilitate school-support groups (e.g. CET, BOT, PTA) x x x Supervisor, School administrator training x x x x x Responsibilities/training at each level: C, G, I, C/S x x x x x School Improvement Plan/School standards x x Data-informed decision-making/knowledge management x x x x Monitoring/Evaluation x x x x x Donations (monetary, in-kind, labor) x x MAP administration x x x Policy: EMIS/databases. x Policy: Decentralization x Policy: Organizational Development x Policy: Strategic Planning x * All activity categories were taken from program documents. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 46 ANNEX IV: Components of Peer Observation Conferences Pre-observation Teacher Observer will if necessary 1. Identify the purpose of the observation. 1. Clarify the purpose of the observation. 2. State the purpose of the lesson. 2. Clarify statements of purpose. 3. Describe what led up to and will follow the lesson. 3. Determine what led up to and will follow the lesson. 4. Translate the purposes into descriptions of observable student behaviors being described. 4. Probe for specific observable student behaviors desired by the teacher. 5. Describe the teaching strategies/behaviors to be used or experimented with. 5. Probe for specific teaching strategies/behaviors to be used. 6. Describe the relationship between the teaching behaviors, students behaviors and desired outcomes. 6. Probe for relation between teacher behaviors, student behaviors and outcomes. 7. Describe the role of the observer. 7. Clarify your own role in the observation. 8. Tell the observer what data you would like collected. 8. Given these outcomes and behaviors, ask what data the teacher would like you to collect. During the lesson During the lesson 1. Teach the lesson 1. Observe and record literally whatever was agreed upon in the pre-observation conference. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 47 Components of Peer Observation Conferences Post-observation Teacher Observer will if necessary 1. Express feelings about the lesson or experiment. 1. Clarify the teacher’s feelings about the lesson or experiment. 2. Recall student behaviors observed during the lesson. 2. Probe for recollection of student behaviors. 3. Compare student behavior observed to student behavior desired. 3. Seek comparison between student desired and actual behavior. 4. Describe own behavior during lesson. 4. Probe for description of teacher behavior. 5. Compare teacher behavior performed with teacher behavior planned. 5. Seek comparison between teacher planned and actual behavior. 6. Ask for data the observer collected (not necessarily the 6th step, but whenever appropriate. 6. Share the data collected about student or teacher behaviors (not necessarily as the 6th step, but whenever appropriate). 7. Make inferences as to the achievement of the purposes of the lesson. 7. Probe for inferences about achievement of the purposes of the lesson. 8. Analyze why the student behaviors were/were not performed. 8. Probe for explanations of why the student behaviors were/were not performed. 9. Summarize what you learned or might do differently. 9. Seek summary of what teacher learned or might do differently. After the Conference 1. Reflect on the conferencing process and how you’re working together. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 48 ANNEX V: Definitions of Efficiency, Effectiveness and Sustainability For consistency and reliability, the definitions of efficiency, effectiveness and sustainability were taken from the World Bank and used for all programs.23 The definitions used for this study are: • Efficiency: Is a measure of how economically resources are converted to results. Resources can refer to funds, expertise, time, or other inputs. • Effectiveness: Is the extent to which the objectives were achieved according to the development interventions, taking into account the relative importance of the objectives. Effectiveness can also be used as an aggregate measure of (or judgment about) the merit or worth of an activity.24 • Sustainability: Is a continuation of benefits from a development intervention after major development assistance has ended. Analysis of sustainability can also project the probability of continued long-term benefits or resilience to risk of the net benefit flows over time. 23 World Bank (2004) Ten Steps to a Results-Based Monitoring and evaluation System by J.Z Kusek and R.C. Rist. 24 An example of this would be the extent to which an intervention has/is expected to attain its major relevant objectives efficiently in a sustainable fashion and with a positive institutional development impact. (World Bank 2004). USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 49 ANNEX VI: Scope of Work Using the information from the desk study, we will evaluate six USAID education interventions. These are: the Girls Improved Learning Outcomes Program (GILO); the Technology for Improved Learning Outcomes program (TILO); the School Team Excellence Awards Program (STEAP); the Education Reform Program (ERP) as implemented by the American Institutes for Research under EQUIP1 and as implemented by the Academy for Educational Development under EQUIP2; and the New School Program (NSP). While several of the activities have concluded or are about to end, this will be a formative evaluation in the sense that the evaluation outcomes are designed to identify lessons learned and “best steps” for improvement, setting the stage for the design of USAID/Egypt’s next education strategy. The evaluation will emphasize efficiency, effectiveness and sustainability, and will assess the impact of USAID activities in the following areas: • Decentralization • Gender equity • Teacher training • Community involvement • Curriculum development • Policy reform The study will use a combination of the Task 1 desk study (including drawing on the findings from CAPS, SCOPE, and MAP); roundtable discussions and one-on-one interviews with key USAID, project and MOE staff; interviews and focus group sessions with principals and teachers; and discussion sessions with community stakeholders. The team will also conduct selected classroom observations and focus group sessions with boy and girl students. USAID has suggested the following questions for each thematic area: 1. What did USAID accomplish in these areas? 2. What was successful and effective? What failed? 3. What still needs to be done in these areas of intervention? 4. What was sustainable? With a view towards identifying “lessons learned,” JBS/Aguirre will add “Why?” and “How?” to appropriate questions and will be including “How can this be made better?” as a standard question in our protocols for educators and, as appropriate, for community stakeholders. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 50 ANNEX VII: Methodology 1. Design Drawing on information gathered in the desk review and meta-evaluation study (Task 1), this study employed a multi-method qualitative design consisting of semi-structured one-to-one and group interviews with key stakeholders at central, governorate and district levels (i.e., USAID staff, project implementers, MOE staff, and donors involved in similar initiatives), school site visits and classroom observations. Site visits were conducted in 10 governorates benefiting from USAID interventions (see Table 1 below), and involved one-to-one and group interviews at the school level with teachers, principals and other key school-based staff, community members and students; it also involved a number of interviews with local officials and local project staff (see Table 1 below). Classroom observations were conducted in six schools for each project with a teacher training component (GILO, TILO, ERP1 and NSP). The team also reviewed project documentation (see Annex X for a list of documentation reviewed). 2. School Sampling Given that resources and time were limited, the study adopted a purposive sampling based on criteria of (a) selection of schools by project covering 10 schools per project; (b) selection of primary, preparatory, secondary and multi-grade schools, oversampling primary schools as agreed given that the major focus of USAID intervention is at the primary level; (c) perceptions by implementers as to the relative performance of schools or other classifications provided by implementers; and (d) relative accessibility and location. Both rural and urban schools were represented in the sample. The sample included 10 schools per project that intervened at the school level (this excludes ERP2) for a total of 50 schools across10 target governorates.25 While the sample of schools per project is relatively small (and therefore not representative), the large number of interviews conducted per site across 10 intervention governorates allowed for robust data to inform this evaluation’s future program recommendations. The team worked closely with project implementers and USAID staff to obtain lists of schools per project from which to draw the sample. The team is thankful for the invaluable support received from USAID staff, and from project implementers from both central and local branch offices in organizing the logistics of the visits in each site. The team would also like to thank CARE staff and STEAP coordinators for their most appreciated support in making visits to NSP and STEAP schools possible, despite the fact that these are closed projects. Support included accompanying the team to schools in remote areas and scheduling all interviews to be conducted per site. As mentioned above, classroom observations were conducted in six schools per project with a teacher training component. Within the purposive sample of 10 schools per project selected by the evaluation team for site visits, each project with a teacher training component was instructed to schedule six classroom observations of full language, science or math classes. Teachers observed in the school sample were therefore selected by the projects and notified of the visits, which allowed them to prepare their lessons for observation. It should be noted, however, that the sample included schools with different levels of performance given that implementers were asked to provide a list of schools classified by relative performance or other classifications from which the sample could be drawn. 25 This figure accounts for three instances where sampling purposively included schools benefiting from two rather than one project. These are two schools with both ERP1 and NSP interventions, and a TILO school that was also a STEAP school; in these cases the sampled school was counted for each project. It is also important to note, however, that given that STEAP was implemented nation-wide, the team found more overlap while in the field. In only one instance was the team not allowed to visit a sampled school (a STEAP participating school in Helwan). USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 51 3. Instruments In order to implement the study, interview protocols were developed to guide the semi-structured interviews, while allowing flexibility for interviewers to delve into other aspects raised by interviewees that might not have been covered in the instruments. Instruments were translated into Arabic to facilitate use when Arabic was required. Following are the instruments developed (see Annex XI): • Interview Guide for School Administrators • Interview Guide for Teachers • Interview Guide for Students • Interview Guide for Community Members and Parents • Interview Guide for Project Implementers • USAID/Ministry of Education Interview Guide A classroom observation protocol was also designed (see Annex XI). The instrument is based on a list of Best Teaching Practices used by teachers and most often emphasized in teacher training. The instrument used has been piloted and field tested in a number of similar evaluations carried out by JBS/Aguirre and was reviewed side-by-side with SCOPE and modified accordingly to ensure that the instrument aligned well with the classroom observation instrument developed and used by the Mission. Data collection took place over a period of four weeks between February 16 and March 11. Research teams conducted school site visits in 10 governorates, generally visiting one school a day, in addition to conducting other interviews at the idara, mudireya and local project implementer levels. The MOE approved the list of schools to be visited and issued notifications to the mudireyas and idaras, and ultimately the schools in the sample. See Annex IX for a list of schools visited. During the school site visits, the teams conducted 42 interviews with principals; 49 group interviews with teachers; 45 group interviews with other key school staff (e.g., school improvement teams, social workers, SBTEU coordinators, and deputies); 22 group interviews with students; 27 group interviews with community members and parents; 36 group interviews at the idara and/or mudireya levels; 19 group interviews with local project staff; and 25 classroom observations (see Table 1 below). Group interviews included between 6-10 persons. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 52 Table 1: Number of interviews per project and governorate during site visits TILO 2 1 1 2 2 2 2 0 ERP 3 1 2 3 4 2 2 1 STEAP 4 2 0 4 4 3 4 3 SUBTOTAL 9 4 3 9 10 7 8 4 GILO 2 1 1 2 2 1 1 1 TILO 2 0 2 2 1 1 0 1 NSP 4 3 2 3 3 1 2 1 ERP/NSP 1 0 1 1 1 1 1 1 STEAP 1 0 1 1 1 1 0 0 SUBTOTAL 10 4 7 9 8 5 4 4 GILO 5 2 3 5 5 3 5 1 SUBTOTAL 5 2 3 5 5 3 5 1 STEAP 3 2 0 3 3 1 2 0 ERP 1 0 1 1 1 1 1 1 TILO/STEAP 1 0 1 1 1 1 1 1 TILO 1 0 1 1 1 1 0 0 SUBTOTAL 6 2 3 6 6 4 4 2 GILO 3 1 1 2 2 1 3 1 NSP 4 2 2 3 2 2 4 1 SUBTOTAL 7 3 3 5 4 3 7 2 NSP; ERP/NSP 2 1 2 1 1 1 1 0 ERP 2 1 1 2 1 1 2 2 GILO 1 0 1 1 0 1 1 1 SUBTOTAL 5 2 4 4 2 3 4 3 TILO 4 3 1 4 4 1 1 1 ERP 2 1 1 2 2 1 2 2 STEAP 1 1 0 1 1 0 1 0 SUBTOTAL 7 5 2 7 7 2 4 3 TOTAL 49 22 25 45 42 27 36 19 Minya Fayoum Alexandria Cairo, 6th October, Giza and Helwan Project Beni Suef Qena Aswan Group interviews Idara/Mudireya Group interviews local project staff Group interviews teachers Group interviews Students Classroom observations Group interviews school staff Interview principals Group interviews parents/BOT In addition to the site visits, a list of key stakeholders to be interviewed in the Cairo area was put together with the help of USAID staff and project implementers, as well as the Egyptian evaluation team members. This list included project implementers, USAID staff, MOE central staff, and donors working on similar initiatives. The list of MOE central staff for interviews was approved by the MOE (as was the list of schools to be visited). Some 52 people were interviewed individually or in small groups in the Cairo area from the four latter categories, in addition to three other group interviews conducted at TILO, GILO and ERP1 headquarters with other principal project staff. A list of key stakeholders interviewed in the Cairo area can be found in Annex VIII. 4. Field procedures and data quality control Data collection was undertaken by a team of six evaluation team members, three Americans and three Egyptians, divided in teams of two persons each. The research team was trained by the evaluation team leader. Training consisted of a one-and-a-half day workshop in which the evaluation team leader provided an overview of the study objectives. Interview guides and observation protocol items were reviewed and practiced. An overview of the education context in Egypt was provided by one of the Egyptian team members. In the case of the observation protocol, the evaluation team leader used simulations to prepare researchers to observe in classrooms. Observers were instructed to score “virtual” observations consisting of field notes from previously observed lessons as well as videotaped lessons. Agreement USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 53 among raters at the end of the half day training in the observation instrument stood at 82.22 percent across the three teams taking part in the training. In the field, observers, teamed up in teams of two persons each, kept running narratives and completed the observation protocol together after the lesson for each class observed. After each observation a follow-up interview was completed with each teacher observed. Most interviews were conducted in Arabic, which was spoken by three of the six team members. Any bias that emerged through their interpretations was managed through post interview discussions and analysis, and by the fact that the Egyptian team members were education specialists doubling as consecutive interpreters. Additional quality control measures included the development of a field manual, which provided a reference guide to the field procedures developed in the training and implemented in the field. 5. Data analysis Given the short period of time available after data collection to conduct data analysis and write the report, a rigorous system of data recording was adopted to ensure data was recorded and shared with all team members on a daily basis during data collection. In-depth weekly analysis sessions were held to identify patterns and record findings based on the evaluation’s key areas. Results of these sessions were recorded, shared with the team, and built on in each subsequent analysis session. Observation data was coded and analyzed to determine patterns and findings across programs. Descriptive statistics (crosstabs and frequencies) were used to look at results by project, by grade and by governorate. This was put side by side with analyses of the qualitative data collected during one-to-one interviews with teachers observed to allow for a more complete picture of results and trends. Given the small number of observation conducted per project, illustrative data was reported in a qualitative rather than quantitative manner. 6. Limitations A major limitation of this study is the HINI situation. Due to the outbreak of the disease, schools in the country adopted more than one shift and/or limited the number of students per class to avoid high class density, which is believed to increase the chances of spreading the virus. These policies created a more favorable but “artificial” situation for classroom observations in that it did not allow for observation to take place in “real” class conditions. High class density was cited by a number of teachers as one of the obstacles to the implementation of student-centered methodologies, so one wonders if observation results would have differed had the research taken place in “real” conditions. The H1N1 situation also constrained and condensed project-based teacher training efforts given the pressure put on teachers’ time. Reportedly, teaching requirements in terms of time increased, which in turn translated into less time available for participation in professional development activities. The high number of projects with different foci and timeframe to be evaluated under one effort is another limitation of this study. Although this is a big picture evaluation, the final product could have benefited from studying each project more in depth, had time and resources allowed. Needless to say, some of these interventions, if not all, would have warranted a stand-alone evaluation, especially because projects were in different stages of implementation with some of them already closed. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 54 ANNEX VIII: List of Persons Interviewed MOE Central Name Affiliation Dr. Reda Abou Serie First Deputy to the Minister of Education Mr. Tarek Abdel Aziz Policy and strategic planning unit Mr. Amr Zein El Abedin Policy and strategic planning unit General Nabil Helmy General Authority for Educational Building (GAEB) head Engineer Hatem Zaghloul General Authority for Educational Building (GAEB) deputy head Dr. Salah Elewa Head of MOE TDC Eng. Osama El Baz Former MOE TDC Eng. Rehab Lasheen MOE Projects Officer Mrs. Suzanne Marzouk First Undersecretary Mr. Ayman Helmy Ministry of Communications and Information Technology (MICT) Pre-University Track Leader Dr. Hassan El Bilawy Former Advisor to the Minister of Education Dr. Ahlam El Baz Associate Professor of Education/Evaluation, National Center for Examinations and Educational Evaluation (NCEEE) Dr. Naguib Khozam Director of NCEEE Dr. Mahmoud Abdeen Director of PAT Dr. Zaher MOE consultant Ms. Amina Ghanem Deputy of the Minister of Finance Dr. Hamdy Ali Head of Social Education Department Dr. Amin Abu Bakra Head of Basic Education Department Dr. Ines Labib Head of Primary Education Department Dr. Mohamed Abu Dahab Quality Assurance and Accreditation Unit Mrs. Nagat El Gohari MOE General Education Departments Dr. Nagwa Sadek TDC, MOE Eng. Kamal Seddik Project Manager, MCIT Eng. Ahmed Tameen Project Manager, MCIT Note: In addition to interviewing MOE Central staff, the team also conducted a number of group interviews with local officials at idara and mudireya levels during the site visits conducted in 10 target governorates. Donors Name Affiliation Mr. Mahmoud Gamal El￾Dine Education Office, World Bank Silvia Crescimbeni Program Manager – Education, EU Inas Hegazi UNICEF Andre Saint Pierre Paul DeGiacomo Mr. Geri Burkholder Head of Aid, Canadian Embassy, CIDA Support to Egyptian Primary Schooling Project (STEPS II) Project Field Manager, CIDA Early Childhood Education Program (ECEP) Project Manager, CIDA USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 55 Project Implementers Name* Affiliation Monika Schaffner Former COP, STEAP Andrea Bosch COP, TILO Areeg Hegazi Education Program Manager, CARE Azza Shafik Acting COP, GILO Dr. Jane T. Benbow COP, ERPI Susan Ross Former COP, NSP John Gillies Vice president and Director, EQUIP2, ERP2 Howard Williams Former Deputy COP, GILO Corinne McComb Director, Keys for Effective Learning Laila Iskandar Director, CID Consulting Nevein Dous Data Management and Communications Director, ERPI Hany Attalla Director Decentralization, GILO Diane Prouty Director TPD, GILO Yasser Youssif ERPI Robert LaTowsky Director of M&E, Information, and Communications, GILO Liz Warfield Former Director of Education Department in USAID/Egypt Jane Schubert American Institute of Research (AIR) Ms. Ebtisam Abdullah ** Former STEAP Coordinator Mrs. Mona El-Manshy** STEAP master trainer in Damietta governorate *These are persons who were interviewed individually. Please note that group interviews with other key project staff were also conducted at GILO, ERP and TILO headquarters and local offices. **These persons were interviewed by phone. Other STEAP coordinators and trainers were interviewed during site visits in the governorates included in the sample. USAID Key Staff Name Affiliation Mary Kidwani COTR, TILO; STEAP Mary Ishak COTR, NSP Hala El Serafy COTR, NSP; ERP1; ERP2; GILO USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 56 ANNEX IX: List of Schools Visited Week of visit: February 21-25 Governorate School Name Idara Project Cairo Khaled Ebn El Waled East Nasr City TILO Giza El Tahrir Preparatory School for Girls Om El Moemeneen Primary School North Giza El Agouza STEAP STEAP 6th of October El Nasr Primary School 6th of October Primary (7th district) Abu Nomros 6th of October STEAP STEAP Helwan El Salam Primary school first shift Basatin & Dar Esalam ERP El Salam Primary school second shift Basatin & Dar Esalam ERP Sakr Korash Preparatory school first shift Basatin & Dar Esalam ERP Aly Abn Aby Talbe second shift Basatin & Dar Esalam ERP Zein El Abeden Helwan TILO Fayoum Al-Zahoor Al-Maamlia primary school Fayoum NSP Kuta Multigrade Youssef El Sedek NSP Al-Saaha Primary School Fayoum ERP Al-Basyooniya Mixed Preperatory Fayoum ERP Al-Agmeen Primary Ibshawaay GILO USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 57 Week of visit: February 28-March 4 Governorate School Name Idara Project Alexandria Mostafa Mosharafa (morning shift) EL Montazah TILO Omar Makram (afternoon shift) EL Montazah TILO El Andalos Wa El Hegaz (morning shift) EL Montazah TILO Aziz Abaza (afternoon shift) EL Montazah TILO Isis Secondary for Girls East Idara ERP Karmooz Boys Primary West Idara ERP El Zahraa East Alexandria STEAP Beni Suef Al-Gafadoon Primary Mixed al-Fashn GILO Al-Gamhood Preparatory Mixed al-Fashn GILO Ali Mubarak Experimental school Beni Suef TILO El Hakim primary school Beba NSP Tarshoub New Prep school Beba NSP Aswan Edfo Boys Primary Edfu STEAP El Hay El Sakani Edfu STEAP Abu Bakr Al-Sadeeq Kom Ombo ERP Balana Compound Primary Nasr El Nuba TILO Shaterma Primary Nasr El Nuba TILO Week of visit: March 7-11 Governorate School Name Idara Project Beni Suef El Awawna new primary Ehnasia ERP Bani Ghoneem El gedida Prim El Wasta TILO Abo El- Makarem Multigrade Ahnasya NSP Sharhy Prep school Ahnasya NSP El Emam Beba STEAP Qena Salam Bihajr Danfeeq Primary Naqaada GILO Abu Bakr Al-Sadeeq Qena GILO Madina Al-amaal Primary Qena City GILO Qena Girls Preparatory Qena City GILO Awlaad Hamadi New Primary in Abu Diyaab Sharq Dishna GILO Minya Dalqam Basic Education Samalout GILO Al-Awr Primary Samalout GILO Elmostakbal-primary school for girls Abu Qorqas NSP Bani Mousa Multigrade Abu Qorqas NSP Bani Ebid New primary school Abu Qorqas NSP USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 58 ANNEX X: List of Documents Reviewed REFERENCES Egypt Ministry of Education (2008) Development of Education in Egypt, 2004-2008, A National Report. Cairo. Egypt Ministry of Education (2007) Integrated Document of Effective Schools Standards. Cairo. Egypt Ministry of Education (2007) National Strategic Plan for Pre-university Education 2007-2012. Cairo. Ginsberg, M, N. Megahed, M. Elmeski, and N. Tanaka (2008) Reforming Education Governance and Management in Egypt: National and International Dynamics (draft). Cairo. Langsten, R and T. Hassan (2007) Basic Educational Attainment in Egypt: Trends and Determinants. American University in Cairo, Cairo. Langsten, R and T. Hassan (2009) Education Transitions in Egypt: The Effects of Gender and Wealth. American University in Cairo. UNDP (2004) Egypt Human Development Report: Choosing Decentralization for Good Governance. Institute of National Planning, Cairo. UNESCO (2003) Education Reform in Egypt 1996-2003: Achievements and Challenges in the New Century. Cairo. By Project Education Reform Program (ERP1)/EQUIP 1 Bernbaum, M. (2007) Education Reform Program: Rapid Assessment Summary. Cairo. Ginsberg, M, N. Megahed, M, A. Abdellah, and A. Zohry. (undated) Active-Learning Pedagogies as a Reform Initiative: The Case of Egypt. Research paper prepared under the Education Quality Improvement Project 1: School and Classrooms. American Institutes for Research and US Agency for International Development. Washington D.C. Ginsberg, M, and N. Megahed (2008) Education Reform Program: Support in the Area of Professional Development: Documentation Research. Monitoring and Evaluation Division. Cairo. USAID (2007) Education Reform Program/EQUIP 1: Progress Report on Original Program Description. June 2004 – July 2007. American Institutes for Research, Washington D.C. USAID (2007) Education Reform Program/EQUIP 1: Revised Program Description. SO 22: Sustained Improvement in Student Learning Outcomes (SILO). American Institutes for Research, Washington D.C. USAID (2009) Education Reform Program/EQUIP 1: Quarterly Report January-March 2009: SO 22: Sustained Improvement in Student Learning Outcomes (SILO). American Institutes for Research, Washington D.C. USAID (2009) Education Reform Program/EQUIP 1: Quarterly Report April-June 2009: SO 22: Sustained Improvement in Student Learning Outcomes (SILO). American Institutes for Research, Washington D.C. USAID (2009) Education Reform Program/EQUIP 1: Quarterly Report July-September 2009: SO 22: Sustained Improvement in Student Learning Outcomes (SILO). American Institutes for Research, Washington D.C. USAID (2009) Education Reform Program/EQUIP 1: Quarterly Report October-December 2009: SO 22: Sustained Improvement in Student Learning Outcomes (SILO). American Institutes for Research, Washington D.C. USAID (2009) Egypt: Education Reform Program Educational Quality Improvement Program 1(EQUIP 1) Technical Application. American Institutes for Research, Washington D.C. USAID (undated) ERP Summary: What Parents Think About Education. Cairo. USAID (undated) Formula for Calculating Financial and Human Resources for Schools Quality Improvement. Cairo. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 59 Education Reform Program (ERP2)/EQUIP 2 American Institutes for Research (Undated) ERP Family of Schools 2009. List of All ERP Constructed Schools. List of LRCs Established by ERP 2009. Cairo. Ginsberg, M. and N. Megahed. (2008) Education Reform Program: Support in the Area of Governance and Management. Cairo. USAID (2009) Education Reform Program - EQUIP 2, Policy and System Change: Final Report 2004-2009. Academy for Educational Development, Cairo. USAID (2006) Scope of Work: ERP 2. Cairo. Girls Improved Learning Outcomes (GILO) Linan-Thompson, S. (2009) Upper Egypt EGRA 2009: Results and Implications. Girls’ Improved Learning Outcomes (GILO). Power-point. Cairo. Research Triangle Institute (2009) Girls at the Center of Education Reform in Egypt. At http://www.rti.org/brochures/egypt_girls_education.pdf USAID (2008) Girls Improved Learning Outcomes (GILO) Quarterly Report, Inception to June 2008. Research Triangle International, Cairo. USAID (2008) Girls Improved Learning Outcomes (GILO) Quarterly Report, July-September 2008. Research Triangle International, Cairo. USAID (2008) Girls Improved Learning Outcomes (GILO) Quarterly Report, October-December 2008. Research Triangle International, Cairo. USAID (2008) Girls Improved Learning Outcomes (GILO) Quarterly Report, January-March 2009. Research Triangle International, Cairo. USAID (2008) Girls Improved Learning Outcomes (GILO) Quarterly Report, April-June 2009. Research Triangle International, Cairo. USAID (2008) Girls Improved Learning Outcomes (GILO) Quarterly Report, July-September 2009. Research Triangle International, Cairo. USAID (2008) Girls Improved Learning Outcomes (GILO) Quarterly Report, October-December 2009. Research Triangle International, Cairo. USAID (2008) Scope of Work: Modification #03: Expanding Equitable Access to Education for Girls. Cairo. USAID (undated) Story of Change: Change through Improved Classroom Management. Cairo. New School Program (NSP) Egypt Ministry of International Cooperation (2006) Project Evaluation Report No. 16: New School Program. Cairo. USAID (2009) New School Program, Inclusive of the School LINC GDA: Final Report. American Institutes for Research, Washington, D.C. USAID (2003) New Schools Program: Mid-term Evaluation. Aguirre International, Washington. D.C. USAID (2008) New Schools Program: Final Evaluation. American Institutes for Research. Washington, D.C. School Team Excellence Awards Program (STEAP) Schaeffner, M, A.K.A. Badran, and A. El-Baz (2008) Site Visit Guide to Evaluate Egyptian Schools for Quality. Cairo. USAID (2009) School Teams Excellence Awards Program II: Quarterly Report April-June 2009. Creative Associate International, Cairo. USAID (2009) School Team Excellence Awards Program II: Final Report. Development Associates. Cairo. USAID (undated) Scope of Work: STEAP 1. Cairo. USAID (undated) Scope of Work: STEAP II. Cairo. USAID (undated) Success Story: Trainer of Trainers and the Teacher of Teachers. Cairo. Technology for Improved Learning Outcomes (TILO) USAID (2009) Feedback Survey Report on TILO. Creative Associate International, Cairo. USAID (2008) Technology for Improved Learning (TILO) Annual Report 1 September 2007-September 2008. Creative Associate International, Cairo. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 60 USAID (2009) Technology for Improved Learning (TILO) Annual Report 2 October-September 2009 FY09- Q8. Creative Associate International, Cairo. USAID (2010) Technology for Improved Learning Outcomes: Armada Retreat. Power-point. USAID (2009) Technology for Improved Learning (TILO) Quarterly Report Year 3, Q1 October-December 2009. Creative Associate International, Cairo. USAID (undated) TILO: Building Blocks for Improved Learning. Power-point. USAID (2010) TILO: Summary of Private Sector Leverage. Creative Associate International, Cairo. By Instrument Critical-thinking, Achievement, and Problem-solving (CAPS) USAID (2008) CAPS Assessment (draft), National Center for Education Evaluation and Examinations and American Institutes for Research. Cairo. Management Assessment Protocol (MAP) USAID (undated) ERP: MAP Rubrics, MAP Protocol. Cairo. Standards-based Classroom Observation Protocol for Egypt (SCOPE) USAID (undated) ERP: SCOPE Rubrics, SCOPE Protocol. Cairo. Note: during the site visits and interviews the team also received a number of soft and hard copy documents that are not listed above either because they had to be returned within a short timeframe (the case of borrowed hard copy documents), or because there was limited time between field work and report writing to add these to the above list. USAID/Egypt Education Portfolio Evaluation Aguirre Division of JBS International, Inc. 61 ANNEX XI: Instruments School Administrators Aguirre Division of JBS International, Inc. January 29 2010 GEM II BPA Number EDH-E-00-08-00003-00 Page 1 1 INTERVIEW GUIDE FOR SCHOOL ADMINISTRATORS Interviewers: ____________________________________ ___________Date ____________________ Administrator Name(s): ______________________________________________________________________ Years of experience: _____Years working in this school: ______ Highest degree: _________________ Sex: (1) M (2) F School Name: _______________________________________________ Village/Town/City: ________________________ Governorate: _______________________________ Location of school (1) urban area (2) rural area What other schools are nearby? Type of School: (1) Elementary/Primary (2) Preparatory (3) Secondary (4) Multigrade USAID intervention(s) in school in past 10 years: (1) ERP-EQUIP1 (2) ERP-EQUIP2 (3) GILO (4) TILO (5) STEAP (6) NSP (For each project, indicate beginning year / ending year) Instructions: Start by thanking the administrator(s) for permitting the school visit and for giving time for this interview. Explain that interview results are confidential and will be analyzed along with many others in order to assess a variety of education programs in Egypt. Thus, some of the questions may not be applicable to this particular school. If the interview includes both the School Director and Deputies, please obtain the relevant data for the others. Please adjust verb tense as necessary for projects that are ongoing or have finished. Note: If a question has already been answered in an earlier response, there is no need to ask it again. USAID PROGRAM EXPERIENCE (If more than one USAID program in the school, please repeat the questions for each program.) 1. Overall, would you characterize your school’s experience with the USAID program(s) as: 1) All positive; 2) More positive than negative; 3) Equally positive and negative; 4) More negative than positive; 5) All negative; 6) Don’t know/no answer (Record each relevant project and its rating.) 2. In what ways has the program been particularly successful? Please describe the particular benefits to your school from participating in this program. 3. What were some of the obstacles or problems with implementation of this program in your school? In what ways could the experience(s) be improved? 4. Have any efforts been undertaken to sustain this program after the USAID activity ends? If yes, please describe the efforts. (Probe the question of how the project’s technical and training support will be (or has been) sustained and by whom, at the end of the project.) School Administrators Aguirre Division of JBS International, Inc. January 29 2010 GEM II BPA Number EDH-E-00-08-00003-00 Page 2 2 5. (If more than one USAID project in school) How would you characterize your school’s experience of having participated in more than one USAID project at a time? Why? (Request rating used in item 1 above, for the whole experience with multiple projects.) (Probe for synergy: In your view, have the programs complemented each other?) SCHOOL MANAGEMENT 6. Does your school have a Board of Trustees? a) If yes, please describe the process of selection. ` b) Do you feel that having a Board of Trustees has been beneficial for your school? Why or why not? 7. Please tell us about your School Improvement Plan. a) Please describe the process of its development, and who was involved. b) Please outline the main objectives of your most recent plan. Have they changed over time? c) Do you feel this process has been beneficial to your school? If yes, please describe improvements you have noted. (Has your plan had an impact on the instructional program?) d) Do you have suggestions for improving the process? 8. Has your school been involved with any of these assessment instruments: School Self Assessment Instrument / CAPS / MAP /SCOPE? a) If yes, please comment on the assessment process. b) Did you receive feedback on any of these assessments? c) Do you think these tools are effective in measuring the impact of interventions on students, teachers and school management? 9. Does your school have a School Training and Evaluation Unit? If so, please describe your experience with it. 10. Please comment on your use of School Report Cards and the School Data Bank. GENDER EQUITY 11. How many students are there in this school? Boys: Girls: Total: 12. (If it is a mixed-gender school) Is there a difference between teaching girls and teaching boys in your school? If yes, please describe the differences. (Probe: Please describe girls’ involvement with school activities such as monitoring, school police, assisting with morning lines, etc.) 13. Has the USAID program targeted girls specifically in your school? If so, do you believe the girls benefited from this program? a) If no, why not? b) If yes, please describe how they benefited. 14. What are your suggestions for improving gender equity in your school? School Administrators Aguirre Division of JBS International, Inc. January 29 2010 GEM II BPA Number EDH-E-00-08-00003-00 Page 3 3 PROFESSIONAL DEVELOPMENT: FOR ADMINISTRATORS 15. Have you received any training as a school administrator as a part of the USAID program? (If not, please skip to question 18.) 16. If yes, which USAID program(s) funded the training and who conducted it? 17. Please describe the training you received (List by topic, duration or number of sessions of each topic). 18. Please describe any changes in your management system since you received this training? 19. (If the administrator received training in supervision and evaluation of staff) Do you evaluate the teachers in your school? If yes, please describe any changes in your evaluation system since you received this training. 20. Do you feel you have been better able to manage the school, supervise and support your staff since you participated in this training? If yes, please describe what you do differently now. 21. What were the most useful aspects of the training you received? 22. Was this training consistent with other training you have received (from USAID or MOE or other)? If not, please describe the differences. 23. What further professional development opportunities do you feel would be useful to you? 24. Do you have opportunities to meet with administrators from other schools to discuss school issues? If yes, please describe. 25. Have you noticed changes in the support you or teachers receive from district or governmental supervisors since the inception of the USAID project(s)? If yes, please describe. PROFESSIONAL DEVELOPMENT: FOR TEACHERS 26. How many teachers are at your school? (Regular classroom teachers / Support staff) 27. What is the average class size? The typical range of class sizes? 28. Have any of your teachers attended USAID-sponsored training? a) If yes, approximately how many teachers have attended events? On what topics? b) Was this training school-based, or done at the district, governmental or national level? c) What was the approximate duration of these events for most teachers? (single talk or workshop / series of classes over course of several months / training events for more than 1 year) d) Did your teachers attend these events with others from your school (e.g., by grade or subject)? 29. Based on your observations, do you think this teacher training was effective? Why or why not? School Administrators Aguirre Division of JBS International, Inc. January 29 2010 GEM II BPA Number EDH-E-00-08-00003-00 Page 4 4 30. What specific practices have you observed as improved in teachers who received the training? (Probe: planning,; use of active learning/student-centered methods; stimulating students’ problem solving skills; cooperative learning; evaluation; using technology to facilitate learning, etc.) 31. What other professional development activities has the project introduced? How effective are these in your opinion? 32. If you or your teachers have participated in training from more than one USAID project, were different models or approaches to training used? If yes, please describe. In your opinion was one approach more effective than others? 33. Can you offer any suggestions for how the training might be improved? 34. Has your school lost teachers who had received training from the USAID project? If yes, why? How has that impacted the effect of the teacher training in your school? Do you have any suggestions for solving this problem? 35. Has the school or district made any plans to continue the teacher training when the USAID program ends? If yes, please describe the efforts. 36. What additional training do you think would be beneficial for your teachers? CURRICULUM AND INSTRUCTIONAL MATERIALS 37. Please describe changes to your curriculum, and the way it is taught, in recent years. What is your opinion of these changes? 38. Has the project provided instructional materials to your teachers? If yes, please describe. a) If yes, do you think these have been effective in improving teaching and student learning? b) What additional instructional materials do you think your teachers need in order to teach the curriculum effectively? 39. Do you use technology in support of your curriculum? If yes, please describe. 40. How many computers do you have in your school? a) Where are those computers located? (computer lab, classrooms, teachers’ room, library) b) Please describe access individual students and classes have to computers (length of time each day or week, where, student-computer ratio) c) Are there particular subjects in which students use computers more intensively for learning purposes? 41. Do you feel that the computers have been used effectively to support your curriculum? (Probe if not mentioned in previous section: Please comment on training your teachers have received in how to use technology to improve teaching and learning.) 42. What do you consider to be the major constraints that impede teachers’ and students’ effective use of technology in your school? School Administrators Aguirre Division of JBS International, Inc. January 29 2010 GEM II BPA Number EDH-E-00-08-00003-00 Page 5 5 For Participants in TILO (or other projects with a technology component such as NSP): 43. What have been the greatest successes of the technology component of the project? 44. What suggestions do you have for improving it? 45. What plans do you have for maintaining and updating the computers and software after the conclusion of the project? COMMUNITY INVOLVEMENT 46. Has the school’s relationship with community changed since the project? How? 47. How have parents and community members been involved at your school this year? (Probe for items below if they are not mentioned.) Community leaders visit school and give talks Routine meetings with individual parents Meetings of Board of Trustees Volunteering in classrooms Volunteering in school functions Assistance with school repairs or renovations Support in fundraising (cash or kind: money, materials, equipment, etc.) Participation in developing School Improvement Plan Other (please describe) 48. What percentage of your school’s parents typically participates in school activities? 49. Do you find that most of your parents are aware of your school’s participation in USAID projects? If yes, are they supportive of the project(s)? 50. Please name ONE thing that parents or community members could do (that they are not doing now) that would significantly help your school. STUDENTS 51. Please describe the impact on students of the USAID project(s) in which you have participated. (Probe: do you think student learning has improved as a result of your school’s participation in this project? Why or why not?) 52. Have you received results of standardized testing (national tests, CAPS)? If yes, is there evidence that student learning has improved in your school in recent years? DECENTRALIZATION 53. Since your school’s participation in the USAID-sponsored program, has decision-making for the school been more locally oriented? In which areas? School Administrators Aguirre Division of JBS International, Inc. January 29 2010 GEM II BPA Number EDH-E-00-08-00003-00 Page 6 6 54. Can you describe the overall decision-making process, and indicate which of the following are involved? The national government The governorate A school oversight committee (BOT) The school’s staff (administrators and teachers) Parents and students 55. Can you offer any suggestions of how decision-making in your school might be improved? POLICY REFORM 56. Since your school’s involvement with USAID programs, have there been any changes in your relationship with the Egyptian Ministry of Education? If yes, please describe. 57. Has any teacher in your school taken the Cadre placement tests and been licensed? a) If yes, please comment on the process. b) If no, what is your opinion about placing teachers in the Cadre? 58. Did your school participate in STEAP? a) If yes, please describe what this participation involved for your school. b) Did you use the ESQM (Egyptian School Quality Manual) in connection with STEAP? If yes, have you continued to use it? How? c) Did your school win a STEAP award? d) Please comment on the process followed for award selection. e) Do you feel participation in this process was beneficial for your school? If yes, in what way? 59. Since your school’s involvement with USAID programs, have you had any opportunity to provide feedback on education policy reform within your community? If yes, please provide examples. 60. Can you offer any suggestions for how education policy might be reformed now? CONCLUSION 61. Is there anything else you would like to tell me about your experiences with the USAID program that I might not have asked? Thank you very much for your time. Classroom Observation Aguirre Division of JBS International, Inc. January 29 2010 GEM II BPA Number EDH-E-00-08-00003-00Page 1 11 CLASSROOM OBSERVATION INSTRUMENT (To be completed after observation of the lesson) Observers: Date: Teacher’s Name __________________________ Grade/Subject of Lesson: Male: ___________ Female: __________ School Name: ____________________________ Village/Town/City: _________________________________ Governorate: ____________________ Type of School: (1) Elementary/Primary (2) Preparatory (3) Secondary (4) Multigrade USAID intervention(s) in school in past 10 years: (1) ERP-EQUIP1 (2) ERP-EQUIP2 (3) GILO (4) TILO (5) STEAP (6) NSP Indicator Observed [1] Not Observed [0] Instruction 1. The teacher shows evidence of prior class planning. The teacher has prepared materials for the class period. 2. Materials related to the class activities are ready. The students have materials related to the class activity. 3. The teacher explains the goal and purpose of the class lesson to the students 4. The teacher models the task that the students are to perform (e.g., predicting before reading, preparing an outline for a report, solving an algebraic equation, etc.) 5. The teacher begins the class activity with questions that review previous lessons and draw on the prior knowledge of the students. 6. The content prepared by the teacher is consistent with the purpose and goal of the lesson. 7. The teacher asks questions of different types to encourage students to answer using critical thinking skills. Grouping and Feedback 8. The teacher facilitates work in groups when appropriate. 9. The teacher works with student groups at different levels of difficulty based on the knowledge of the students. 10. The teacher gives different assignments based on the learning needs of the students, when appropriate. 11, The teacher provides students with corrective feedback and positive support as necessary. Classroom Management 12. The teacher organizes class time so that a majority of the students are involved in schoolwork during the observation. 13. The teacher promotes an effective classroom climate through positive and respectful actions, attitudes and gestures. 14. The teacher reminds the students of the classroom rules and expectations, when necessary. Use of Physical Space (Grades 1-6) 15. The classroom is decorated with printed materials and/or students’ original work. Classroom Observation Aguirre Division of JBS International, Inc. January 29 2010 GEM II BPA Number EDH-E-00-08-00003-00Page 2 21 Comments on Lesson Observed Please describe here examples of items you observed in these categories. Include useful quotes from lesson to illustrate, when possible. Items related to form: Item 1. Evidence of planning (mention if teacher showed you a lesson plan. Comment if lesson involved integration of subject areas.) Item 2 Materials: brief description (If computers or other technology used, describe how they were used and by how many students) Item 3 Describe goal and purpose of lesson Item 4 Teacher models activity for students: brief description Item 7 Provide examples of levels of questions Item 8. Brief description of grouping (were students actually working together? Was cooperative learning observed?) Item 9 Teacher works with groups at different levels: Brief description, including any evidence of teacher use of evaluation as basis for grouping Item 10 Differential assignments: Brief description, including any evidence of teacher use of evaluation as basis for differential assignments Item 11 Examples of feedback (especially when student gave incorrect or partial answer). Also note whether students gave feedback or assisted each other. Item 12 Use of Time: Comment Item 13 and 14 Comment on climate, discipline: Did students appear to be familiar with classroom routines? Items 15 Brief description of type of items in classroom Other 1. Did teacher make use of student experiences in lesson? 2. Student participation: Did students ask questions? Who did most of the talking: teacher or students? 3. Gender: did teacher ask questions of both boys and girls? 4. Please describe any observation of teacher assessment or evaluation of students. 5. Did students engage in writing? Provide brief description (length? Was writing original?) Please add any additional comments you wish to provide. Community/Parents Aguirre Division of JBS International, Inc. January 29 2010 GEM II BPA Number EDH-E-00-08-00003-00 1 Page 1 INTERVIEW GUIDE FOR COMMUNITY MEMBERS AND PARENTS Interviewer(s): ____________________________________ Date ____________________ No. of people in panel:____________ Males: ____________ Females: ______________ School Name(s): __________________________________ Village/Town/City/District: _________________________________ Governorate: ______________ Type of School: (1) Elementary/Primary (2) Preparatory (3) Secondary (4) Multigrade USAID intervention(s) in school in past 10 years: (1) ERP-EQUIP1 (2) ERP-EQUIP2 (3) GILO (4) TILO (5) STEAP (6) NSP Instructions: Start by thanking the participants for giving time for this group interview. Explain that interview results are confidential and will be analyzed along with many others in order to assess a variety of education programs in Egypt. Change tense as necessary for USAID programs that have finished. PARTICIPANTS (Note: Although group is intended to be a mixed group of parents and community members, some education personnel may be members of community groups (e.g., a Board of Trustees) and so may also attend. Many groups will include participants from more than one school.) Please tell us a little about yourselves: 1. How many of you are: (Preliminary information may be gathered by a show of hands. Also pass around sheets on which participants can provide the information below.) INFORMATION ABOUT PARTICIPANTS Category School M F TOTAL Parent Community Member Teacher Administrator MOE supervisor Other (specify) USAID PROGRAM EXPERIENCE 2. Are you familiar with USAID programs in your school (provide program names)? Number who say Yes: Number who say No: Community/Parents Aguirre Division of JBS International, Inc. January 29 2010 GEM II BPA Number EDH-E-00-08-00003-00 2 Page 2 3. Overall, would you characterize your experience with USAID programs as: (Write number who raise hands in each category) All positive More positive than negative Equally positive and negative More negative than positive All negative Don’t know/no answer 4. What have been particularly successful aspects of USAID programs in your school(s)? 5. What have you perceived as weaknesses? In what ways could the experience(s) be improved? 6. Have any efforts been undertaken to continue project activities after the USAID program ends (ended)? If yes, please describe program activities you expect to be continued: ___________________________________________________ GENDER EQUITY 7. (If it is a mixed-gender school) Is there a difference between the way boys and girls are taught in this school? If yes, please describe the differences. 8. Has the USAID program targeted girls specifically in your school? If yes, how? 9. If so, do you believe the girls benefited from this program? a) If no, why not? b) If yes, please describe how they benefited. 10. What are your suggestions for improving gender equity in your school? PROFESSIONAL DEVELOPMENT 11. Have teachers in this school received special training provided by this project? 12. If yes, have you noticed any changes in your children’s school experience since the training began? If yes, please describe the changes. 13. Have you yourself attended any special training or workshops for parents or community members? If yes, please describe the training and who provided it. 14. What were the best things about the training you received? What suggestions would you have for making it better? 15. Please describe any further training you think would be useful for you. 16. Has the school or district made any plans to continue the teacher training or other training when the USAID program ends? If yes, please describe the efforts. Community/Parents Aguirre Division of JBS International, Inc. January 29 2010 GEM II BPA Number EDH-E-00-08-00003-00 3 Page 3 CURRICULUM 17. Has this project provided instructional materials for students to use in class or at home? If yes, what kinds of materials? Do you think they have been useful? 18. Has the project provided any books or other materials for parents? If yes, on what topics? Do you think they have been useful? 19. Has this project provided or assisted with Adult Literacy programs in your community? If yes, please describe. Does your community need more Adult Literacy services? 20. Has the project provided computers or other technology in support of the curriculum? If yes, please describe. (Probes: Do you think that these have been effectively used in teaching and learning at the school? Do you think that students have benefited from using technology in their learning?) For schools that benefited from a technology component in a USAID project: 21. Has your community benefited from the project’s Information and Communication Technology Centers (ITC) in the case of NSP or Community Learning Centers (CLC) in the case of TILO or ERP? If yes, please describe. Do you have recommendations for improving these centers and the way in which they are used? 22. Have plans been made by the school, government, or community to continue providing instructional materials and maintaining and updating equipment after the USAID project ends? COMMUNITY INVOLVEMENT 23. Have you been involved at your school in the following ways this year? (Read items and request show of hands.) Community leaders visit school and give talks Routine meetings with individual parents Meetings of Board of Trustees Volunteering in classrooms Volunteering in school functions Assistance with school repairs or renovations Support in fundraising (cash or kind: money, materials, equipment, etc.) Participation in developing School Improvement Plan Other (please describe) STUDENTS 24. Please describe the impact on students of the USAID project(s) in this school. a) Do you feel the students are happier coming to school? b) Do you think the students are learning more than they used to? Please explain. c) What other changes in your children’s attitudes or behavior do you notice? Community/Parents Aguirre Division of JBS International, Inc. January 29 2010 GEM II BPA Number EDH-E-00-08-00003-00 4 Page 4 25. Have you received results of standardized testing (national tests, CAPS)? If yes, is there evidence that student learning has improved in your school in recent years? DECENTRALIZATION 26. Since your school’s participation in the USAID-sponsored program, has decision-making for the school been more locally oriented? 27. Can you offer any suggestions of how the decision-making process might be improved? POLICY REFORM 28. Does your school have a School Improvement Plan? a) If yes, please describe the process of its development. Who was involved? b) What are the major goals of the most recent plan? c) Do you feel this process has been beneficial to your school? d) Do you have suggestions for improving the process? 29. Does your school have a Board of Trustees? If yes, please comment on its role and effectiveness. 30. Did your school participate in STEAP? Yes /No a) If yes, please describe what this participation involved for your school. b) Did your school win a STEAP award? c) Please comment on the process followed for award selection. d) Do you feel participation in this process was beneficial for your school? 31. Since your school’s involvement with USAID programs, have you had any opportunity to provide feedback on education policy reform within your community? If yes, please provide examples: 32. Can you offer any suggestions for how education policy might be reformed now? CONCLUSION 33. Is there anything else you would like to tell us about your experiences with USAID program (s) that we might not have asked? Thank you very much for your time. Project Implementers Aguirre Division of JBS International, Inc. January 29 2010 GEM II BPA Number EDH-E-00-08-00003-00Page 1 1 INTERVIEW GUIDE FOR PROJECT IMPLEMENTERS Interviewers: ____________________________________ Date ____________________ Respondents’ Names: __________________________ Organization_________________ Positions___________________________________________________________________ Male: ________ Female: __________ Interview regarding USAID intervention(s): (1) ERP-EQUIP1 (2) ERP-EQUIP2 (3) GILO (4) TILO (5) STEAP (6) NSP Instructions: Start by thanking the individual for giving time for this interview. Explain that interview results are confidential and will be analyzed along with many others in order to assess a variety of education programs in Egypt. If more than one individual is present in the interview, please obtain the relevant data for the other(s). BACKGROUND Please tell me a little about yourself: 1. How many years in total have you been in this position (including this year)? 2. Please indicate which UAID projects you are or were involved with. Ask the following questions for each relevant project (adjust tense as needed for projects that are finished: PROJECT: 3. For how long have you been involved with this project? 4. Please describe your specific responsibilities in the implementation of this project. 5. What is the primary area this project is intended to impact? 6. What are the project’s main components? 7. Has the project objective/scope of work been changed since the inception of the project? How? Why? How was this change agreed upon? How has this change impacted the project? Project Implementers Aguirre Division of JBS International, Inc. January 29 2010 GEM II BPA Number EDH-E-00-08-00003-00Page 2 2 8. Please describe the major activities this project has undertaken under each component: (Request data re number of sites, number of individuals trained, training curricula, buildings constructed, equipment purchased and other relevant information. If possible obtain copies of reports showing this information.) 9. Please describe the division of responsibilities between implementers at national and governorate levels. (If implementer at governorate or district level: Please describe your communication with your counterparts in other parts of the country.) 10. Who are your implementing partners at the MOE (central and district level)? How involved are they? (design, implementation, monitoring, reporting, etc.) Have they been generally supportive or not? How? 11. How does this project affect Egypt’s efforts to decentralize education? Please explain. 12. How does this project assist in improving gender equity in Egyptian education? Please explain. 13. Does this project include an in-country or participant training component? If yes, please describe. (Ask questions below separately for in-country and participant training.) a) Target population (teachers, administrators, MOE personnel, etc) b) Location of training (national/governorate/district/school) c) Focus of training d) Design of training d) Delivery of training (e.g.,cascade, core of trainers, in-class coaching) e) Duration of training for each participant (single session, # of weeks or months, # of years) f) Evaluation and monitoring system for training f) Overall accomplishments and shortfalls of this component 14. What effect has this project had on curriculum development in Egypt? Please explain. 15. Please describe any instructional material produced by this project. Who was involved in the development of this material? How and where is it used? 16. How does this project facilitate the involvement of parents and community in the educational system? 17. How does this project contribute to educational policy reform? 18. Please comment on the efficiency of this project (management, financial, use of resources). 19. (If not mentioned above) Please outline the monitoring and evaluation component of this project. Have any of the M&E tools developed by ERP EQUIP2 for USAID been adapted and used in this project? 20. Have you found it possible and productive to work together with implementers of other USAID projects? Please explain. (Do you believe there are synergies between some of the projects?) Project Implementers Aguirre Division of JBS International, Inc. January 29 2010 GEM II BPA Number EDH-E-00-08-00003-00Page 3 3 21. Do you believe this project has been effective? Why or why not? If yes, please indicate whether it has improved: a) Teacher practices b) Student learning c) Management and supervisory styles d) School climate e) Other (please describe) f) Technical and training support to the school from the District and Governorate education offices 22. What do you believe have been the most successful aspects of this project? 23. What have been the least successful aspects? Please explain. 24. In your view, what have been the greatest challenges to successful implementation of this project? 25. What do you believe have been the most important lessons learned from this project? 26. What suggestions do you have for improving this project? 27. What changes would you recommend in the design of a future project with similar goals? 28. In relation to the goals of this project, what do you believe will still remain to be done at its conclusion? 29. How do you believe this project will be sustainable in the long run? (probe: financial, social, institutional, technical, political sustainability) 30. Have any efforts already been undertaken to sustain this program after the USAID activity ends? If yes, please describe the efforts. 31. What elements of the project have been or are in the process of being taken to scale? (If the project has ended, probe whether progress since the end of the project has met expectations or not.) CONCLUSION 32. Is there anything else you would like to tell us about your experiences with the USAID program(s) that we might not have asked? Thank you very much for your time. Students Aguirre Division of JBS International, Inc. January 29 2010 GEM II BPA Number EDH-E-00-08-00003-00 Page 1 1 INTERVIEW GUIDE FOR STUDENTS Interviewer(s): ____________________________________ Date ____________________ School Name: __________________________________ Village/Town/City/District: _________________________ Governorate: ______________ Type of School: (1) Elementary/Primary (2) Preparatory (3) Secondary (4) Multigrade USAID intervention(s) in school in past 10 years: (1) ERP-EQUIP1 (2) ERP-EQUIP2 (3) GILO (4) TILO (5) STEAP 6) NSP Instructions: Start by thanking the students for giving time for this group interview. Explain that we are visiting schools that are in special projects and we want to find out whether those projects are helping students. Anything they tell us will be kept confidential. (Adjust wording as needed to age of students.) PARTICIPANTS Please tell us a little about yourselves: 1. Please tell us what grade you are in: Grade levels M F Total Ask the following if appropriate for the age of the students. USAID PROGRAM EXPERIENCE 2. Have you heard teachers or parents talking about (provide program names)? Number who say Yes Number who say No 3. What do you know about this program? 4. Do you think it has made a difference in your school? If yes, what? ___________________________________________________ GENDER EQUITY 5. (If it is a mixed-gender school) Is there a difference between the way boys and girls are taught in this school? If yes, how is it different? (Probe: Are boys and girls involved in different activities?) Students Aguirre Division of JBS International, Inc. January 29 2010 GEM II BPA Number EDH-E-00-08-00003-00 Page 2 2 6. Do you know whether (project name) is especially meant for educating boys or for educating girls? If yes, do you think it does help them? How? 7. Do you have any suggestions that would help boys or girls learn better in your school? PROFESSIONAL DEVELOPMENT 8. Have you noticed any changes in the way your teachers teach now as compared with when you were younger? If yes, what are the changes? 9. In your classroom, do you sometimes work with other students? 10. Do your teachers give different assignments to some children? 11. If you don’t understand something or need help with your schoolwork, what does your teacher do or say? (Is it easy for you to ask for help?) CURRICULUM 12. Do you have books and materials in your school that you think are interesting? If yes, for which subjects? Do you sometimes take them home? 13. Do you have computers in your school? If yes, have you received any training at school on how to use them? How often do you get to use them? What do you do when the teacher takes you to the computer lab? ￾Which subject teachers take you often to the computer lab? Do you usually have a chance to work on the computer yourself? What do you use it for? (If TILO, Do your teachers bring a computer to class sometimes? If yes, how is this different from going to the computer lab? In your opinion what is the difference between classes taught using a computer and classes not using computers? COMMUNITY INVOLVEMENT 14. Do parents or other adults sometimes come to your school to talk to your class? 15. Do your parents come to school to meet with your teacher sometimes? How often? 16. Do your parents help out in the school? If so, how? Students Aguirre Division of JBS International, Inc. January 29 2010 GEM II BPA Number EDH-E-00-08-00003-00 Page 3 3 DECENTRALIZATION 17. Do you have a student union in your school? If yes, what does it do? If not, how do students help decide about what happens at the school? POLICY REFORM (Older students) 18. Do you know whether your school has a School Improvement Plan? If yes, what is it? Who made the plan? What changes have you seen at the school? 19. Do you know whether your school has a Board of Trustees? If yes, what do they do? 20. Did you ever hear of STEAP? a) If yes, please tell us what you know about it. b) Did your school win a STEAP award? c) If yes, how did your school get chosen? d) Was this a good thing for your school? CONCLUSION 21. What do you think is the best thing about your school? 22. What do you wish you could change about your school? 23. Is there anything else you would like to tell us about your school that we might not have asked? Thank you very much for your time. Teachers Aguirre Division of JBS International, Inc. February 14 2010 GEM II BPA Number EDH-E-00-08-00003-00 Page 1 1 INTERVIEW GUIDE FOR TEACHERS Interviewers: ____________________________________ Date ____________________ No. of Teachers in panel: Male: ___________ Female: __________ School Name: ____________________________ Village/Town/City: _________________________________ Governorate: ____________________ Type of School: (1) Elementary/Primary (2) Preparatory (3) Secondary (4) Multigrade USAID intervention(s) in school in past 10 years: (1) ERP-EQUIP1 (2) ERP-EQUIP2 (3) GILO (4) TILO (5) STEAP (6) NSP Instructions: Start by thanking teachers for giving time for this interview. Explain that we are not evaluating teachers or their schools. Interview results are confidential and will be analyzed along with many others in order to assess a variety of education programs in Egypt. Please change verb tense as necessary for programs that have finished. Note: If a question has already been answered in an earlier response, there is no need to ask it again. PROFESSIONAL DEVELOPMENT 1. *How many of you have attended in-service training in the past 3 years? (show of hands) Males: Females: 2. *Please describe who provided these events. (ask for show of hands) 1) USAID program(s). Which? 2) Provided by Governorate or by Directorate (Moderia) or (Idara) 3) Other 4) Don’t know 3. *What were the topics of training you received from USAID program(s)? 4. *Do you think this teacher training was effective? If no, why not? If yes, please provide us with some examples of how you have used this training in your classroom. (Probe: active learning/student-centered methods; stimulating students’ problem solving skills; evaluation; dealing with children of varying abilities and needs) 5. *Have you been involved in any other professional development activities, such as teachers’ learning circles or centers of excellence? If yes, how effective were these activities? Teachers Aguirre Division of JBS International, Inc. February 14 2010 GEM II BPA Number EDH-E-00-08-00003-00 Page 2 2 6. *If you received training from more than one USAID program, please comment on any differences you noted in the models and approaches used in the training. (Did you find one approach more useful than others? Why?) 7. How is the training you received contributing to your professional development (re: promotion, incentives, pay grade, etc. (from the teacher cadre exam)? 8. Did the training present the same teaching principles and approaches as training you have received from the MOE or other projects? If not, please explain. 9. *Can you offer any suggestions for how the USAID training might be improved? 10. *Do you receive support to assist you in applying new strategies or dealing with classroom issues? (Has this changed since your participation in the USAID program(s)?) If yes, from whom (describe support received): School administrator District supervisor Project trainer Other teachers Other CURRICULUM (Include questions 11 to 15 only for teachers in whose classrooms a lesson was observed. If group interview, start with question 16.) We’d like to ask you some questions about the lesson we observed today. 11. *What were your objectives for the lesson observed today? 12. *How did you feel your lesson went? (Is there anything you would do differently if you were to present this lesson again?) 13. *Are the children always seated as they were today, or do you vary the seating sometimes? If yes, please explain how and why you might group students. [record way students were seated] 14. *How will you evaluate your students on the lesson they did today? 15. Does your curriculum provide provisions for dealing with children of varying needs and abilities? If not, how do you deal with children of varying needs? (For all teachers) 16. *Has the project provided you with adequate and appropriate instructional materials? If yes, please describe. If not, what do you see as needed? 17. *Do you use technology in support of your curriculum? If yes, please describe. 18. *Do you have computers in your classroom? In your school? If yes, please describe how you and your students use them. (How often? In which subjects?) Teachers Aguirre Division of JBS International, Inc. February 14 2010 GEM II BPA Number EDH-E-00-08-00003-00 Page 3 3 (*For Participants in TILO (or other projects with a technology component such as NSP): 19. Do you feel that the computers have been used effectively to support your curriculum? 20. What have been the greatest successes of the technology component of the project? 21. What suggestions do you have for improving it? GENDER EQUITY 22. *How many students are in your classes? What is the ratio of boys to girls? (This information may be gathered on a piece of paper that is passed around.) 23. *(If it is a mixed-gender school) Do boys and girls behave differently in your classrooms? If yes, please describe. 24. *Has the USAID program targeted girls specifically in your school? If so, do you believe the girls benefited from this program? a) If no, why not? b) If yes, please describe how they benefited. 25. *What are your suggestions for improving gender equity in your school? STUDENTS 26. *Please describe the impact on students of the USAID project(s) in which you have participated. (Probe: do you think student learning has improved as a result of your participation in this project? Why or why not?) 27. *Have you received results of standardized testing (national tests, CAPS)? If yes, is there evidence that student learning has improved in your school in recent years? COMMUNITY INVOLVEMENT 28. *What percentage of your parents typically participate in school activities? In what ways are parents and community members involved in your school? (Probe for items below if they are not mentioned.) Community leaders visit school and give talks Routine meetings with individual parents Meetings of Board of Trustees Volunteering in classrooms Volunteering in school functions Assistance with school repairs or renovations Support in fundraising (cash or kind: money, materials, equipment, etc.) Participation in developing School Improvement Plan Other (please describe) What else do you wish parents or community members would do? Teachers Aguirre Division of JBS International, Inc. February 14 2010 GEM II BPA Number EDH-E-00-08-00003-00 Page 4 4 29. *Do you feel that having a Board of Trustees has been beneficial for your school? In what ways? USAID PROGRAM EXPERIENCE (If more than one USAID program in the school, please repeat the questions for each program. Include programs that have now ended.) 30. *Overall, would you characterize your experience with the USAID program(s) as: 1) All positive; 2) More positive than negative; 3) Equally positive and negative; 4) More negative than positive; 5) All negative; 6) Don’t know/no answer. 31. *In what ways has this program been particularly successful? 32. *In what ways could the experience be improved? 33. *Have any efforts been undertaken to continue project activities after the USAID program ends? If yes, please describe. POLICY REFORM 34. *Have you been involved with any of these assessment instruments: CAPS / MAP /SCOPE? a) If yes, please comment on the assessment process. b) If yes, did you receive feedback on any of these assessments? c) If yes, do you think these tools are effective? 35. Have you seen the new national standards? If yes, have they affected the way you teach? 36. *Have any of you taken the Cadre placement tests and been licensed? a) If yes, please comment on the process. b) If no, what is your opinion about the Teachers Cadre? 37. *Have you participated in the development of your school’s School Improvement Plan? a) If yes, please describe the process of its development. b) Whether yes or no: Do you feel this process has been beneficial to your school? c. Do you have suggestions for improving the process? 38. *Did your school participate in STEAP? (show of hands) a) If yes, please describe what this participation involved for your school. b) Did your school win a STEAP award? c) Please comment on the process followed for award selection. d) Do you feel participation in this process was beneficial for your school? 39. Can you offer any suggestions for how education policy might be reformed now? DECENTRALIZATION 40. *Can you describe how teachers are involved in the decision-making process in your school? Teachers Aguirre Division of JBS International, Inc. February 14 2010 GEM II BPA Number EDH-E-00-08-00003-00 Page 5 5 41. Can you offer any suggestions of how the decision-making process might be improved? CONCLUSION 42. *Is there anything else you would like to tell us about your experiences with the USAID program that we might not have asked? Thank you very much for your time. USAID/MOE Aguirre Division of JBS International, Inc. January 29 2010 GEM II BPA Number EDH-E-00-08-00003-00Page 1 1 USAID/MINISTRY OF EDUCATION INTERVIEW GUIDE Interviewer(s): ____________________________________ Date ____________________ Respondent’s Name: __________________________ Organization_________________ Position___________________________________________________________________ Male: ________ Female: __________ Interview regarding USAID intervention(s) : (1) ERP-EQUIP1 (2) ERP-EQUIP2 (3) GILO (4) TILO (5) STEAP (6) NSP Instructions: Start by thanking the individual for giving time for this interview. Explain that interview results are confidential and will be analyzed along with many others in order to assess a variety of education programs in Egypt. If more than one individual is present in the interview, please obtain the relevant data for the other(s). BACKGROUND Please tell me a little about yourself: 1. How many years in total have you been in this position (including this year)? 2. Please indicate which USAID projects you are or were involved with. (Circle projects above) Ask the following questions for EACH project circled above: (Please adjust verb tense as needed for projects that have finished.) PROJECT NAME: 3. What is the primary area this project is intended to impact? 4. What are its principal components? 5. For how long have you been involved with this project? Please outline your responsibilities for the project. 6. Has the project undergone any change in focus or approach during its implementation? a) If yes, please describe the changes. b) Why were the changes made? Please describe the process for making the changes. c) If no, do you think the original project design is adequate to achieve desired results? 7. Does this project have an effect on Egypt’s efforts to decentralize education? Please explain. USAID/MOE Aguirre Division of JBS International, Inc. January 29 2010 GEM II BPA Number EDH-E-00-08-00003-00Page 2 2 8. Does this project assist in improving gender equity in Egyptian education? Please explain. 9. Does this project include an in-country or participant training component? If yes, who is/are the provider(s)? Please outline the training model. (Ask questions below separately for in￾country and participant training.) a)Target population (teachers, administrators, MOE personnel, etc) b) Location of training (national/governorate/district/school) c) Focus of training d) Delivery of training (e.g., cascade, core of trainers, in-class coaching) e) Duration of training for each participant (single session, # of weeks or months, # of years) f) Overall accomplishments and shortfalls of this component (including Central, Governorate, and District capacities to support the schools) 10. What effect has this project had on curriculum delivery and provision of instructional materials or technological resources to support the curriculum? Please explain. 11. Does this project facilitate the involvement of parents and community in the educational system? If yes, how? 12. Does this project contribute to educational policy reform? Please explain. 13. Please comment on the efficiency of this project (management, financial, use of resources) 14. Do you believe this project has been effective? Why or why not? If yes, please indicate whether it has improved: a) Teacher practices b) Student learning c) Management and supervisory styles d) School climate e) Other (please describe) 15. Please comment on synergies you noted between this and other USAID projects. Did projects work together on some issues? How might such cooperation have been improved? 16. Please describe the monitoring and evaluation component of this project. Do you have suggestions for its improvement? 17. What do you believe have been the most successful aspects of this project? 18. What have been the least successful aspects? Please explain. Were there aspects you felt were unsuccessful? (Some approaches that just didn’t work?) Why? 19. What do you believe have been the greatest challenges? And the greatest accomplishments? 20. What do you believe have been the most important lessons learned from this project? (What would you do differently if you were to do it again?) 21. What suggestions do you have for improving this project? USAID/MOE Aguirre Division of JBS International, Inc. January 29 2010 GEM II BPA Number EDH-E-00-08-00003-00Page 3 3 22. In relation to the goals of this project, what do you believe will still remain to be done at its conclusion? 23. Which elements of the project have been or are being taken to scale? (If the project has ended, probe whether progress since the end of the project has or has not met expectations.) 24. Do you believe the accomplishments of this project will be sustainable in the long run? (probe: financial, social, institutional, technical, political sustainability) 25. Have any efforts already been undertaken to sustain this program after the USAID activity ends? If yes, please describe, including any specific plans to ensure the maintenance and updating of school buildings, computer equipment and software provided by the project. CONCLUSION 26. Is there anything else you would like to tell us about your experiences with this USAID program that we might not have asked? Thank you very much for your time.