This report was produced for review by the United States Agency for International Development. It was prepared by Creative Associates International, Inc. EGYPT TECHNOLOGY FOR IMPROVED LEARNING OUTCOMES (TILO) PROJECT FINAL MONITORING AND EVALUATION REPORT Contract Number EDH-I-00-05-00029-00 Task Order No. 06 Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 2 Table of Contents EXECUTIVE SUMMARY ...................................................................................................................... 3 ACRONYMS ............................................................................................................................................ 7 OVERVIEW ............................................................................................................................................. 8 APPROACH AND METHODOLOGY ................................................................................................. 10 1.1 Introduction .........................................................................................................................10 1.2 Subjects ................................................................................................................................11 1.3 Instruments ..........................................................................................................................13 1.4 Data Collection Process .......................................................................................................17 1.5 Limitations of the Study ......................................................................................................20 RESULTS AND FINDINGS ..................................................................................................................22 2.1 Impact Level Results ...........................................................................................................22 2.1.1 Impact Intended Result 1: Improved Student Learning ......................................................22 2.1.2Impact Intended Result 2: Sustainable Decentralized Management of Use of Technology for Teaching/Learning at the School and Idara Levels ................................................................29 2.2 Outcome Level Results ........................................................................................................41 Outcome Intended Result 1: Increased Student Engagement with IT ..........................................41 Outcome Intended Result 2: Changed Classroom Practice (Teachers) ........................................44 Outcome Intended Result 3: Increased Local Support for IT in Schools .....................................53 Outcome Intended Result 4: Enhanced School Management of IT .............................................57 Outcome Intended Result 5: Enhanced MOE Capacity for Management of Education Technology Resources .................................................................................................................59 2.3 Output Level Results ...........................................................................................................60 Outputs Intended Result 1: Training (Teachers) ..........................................................................60 Output Intended Result 2: IT Infrastructure and Digital Content (School) ..................................62 Output Intended Result 3: Community Awareness and Public-Private Partnerships ...................63 Output Intended Result 4: MOE Capacity Building Program ......................................................64 CONCLUSIONS AND RECOMMENDATIONS .................................................................................66 APPENDICES ........................................................................................................................................69 Appendix 1: TILO Performance Monitoring Plan (PMP) ............................................................69 Appendix 2: TILO Follow Up and Support Plan .........................................................................75 Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 3 Executive Summary Background The Technology for Improved Learning Outcome (TILO) project was funded by the United States Agency for International Development (USAID) as a Task Order under the Assistance to Basic Education-Basic Education (ABE-BE) IQC. TILO was a six-year project implemented in nine governorates in Egypt from September 2007-September 2013, with a total ceiling of $31.2 million. The project was managed by Creative Associates International (Creative) and was implemented in collaboration with partners Keys to Effective Learning, PalTech and Seward Inc. TILO was designed to enhance the quality of teaching, learning and school management through the effective use of technology in schools. Creative worked with Egypt’s Ministry of Education (MOE) and other governmental and community-based entities to develop and implement a holistic, integrated model for introducing technology into school-based reform activities with the aim of improving learning outcomes in the intervention schools. TILO’s work encompassed four main components: Component 1--Improve the quality of teaching, learning, and IT management in targeted schools. Component 2—Establish public-private partnerships to support TILO objectives and other innovative educational technology interventions. Component 3--Build capacity for effective management of technology for education at all levels of public education administration. Component 4--Monitor and evaluate how TILO activities are impacting improvements in teaching, learning, and management in targeted schools. TILO interventions originally began in in 277 schools in nine governorates: Greater Cairo (including Cairo, 6th of October and Helwan), Giza, Alexandria, Beni Suef, Assiut, Minya, Fayoum, Qena, and Aswan, to provide interventions in 192 SBR Primary and 85 TILO Smart Schools (TSS). In 2011, the project was extended for two years with a mandate to expand its interventions to SBR Preparatory schools that would receive graduating TILO primary schools students. The TILO Prep school model was extended to 127 schools, increasing the total number of TILO schools receiving interventions to 404 schools. Monitoring and Evaluation Framework TILO’s Monitoring and Evaluation (M&E) component was designed to capture the progress, impact and outcomes of the project according to the activities laid out in the scope of work. TILO’s M&E process was based on a conceptual framework and a Performance Monitoring Plan (PMP) designed to monitor the project’s intended results at three levels: impacts, outcomes and outputs. As per the TILO PMP, M&E measurements followed a structured process, and examined 40 performance indicators related to 11 primary results at the three levels noted below: Level 1: Impact:  1 - Improved student learning  2 - Sustainable decentralized management of use of technology for teaching / learning at the school and Idara levels Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 4 Level 2: Outcomes  1 - Increased student engagement with IT  2 - Changed classroom practice (Teachers)  3 - Increased local support for IT in schools  4 - Enhanced school management of IT  5 - Enhanced MOE capacity for management of education technology resources Level 3: Outputs  1 - Training (Teachers)  2 - IT infrastructure and digital content (School)  3 - Community awareness and public-private partnerships  4 - Ministry of Education (MOE) capacity building program M&E Methodology M&E activities were conducted through baseline and annual measurements over four years of the project for SBR Primary, TSS Prep and SBR Prep schools. Although the project worked in nine governorates, the M&E assessments were conducted in only seven governorates: Greater Cairo (including 6th of October, Helwan and Giza), Alexandria, Beni Suef, Minya, Fayoum, Qena, and Aswan. Assiut was not included due to the very small scale intervention in that governorate, and Giza was included within Greater Cairo. Interventions were carried out in two cohorts. Cohort 1 schools (in Alexandria, Beni-Suef and Greater Cairo) began interventions in 2008. The TILO team learned from the successes and challenges faced during this period and made adjustments to the technology and training models before rolling out the interventions to Cohort 2 schools (in Fayoum, Minya, Qena and Aswan) in 2009. Of the 277 SBR Primary and TSS Prep schools, 57 were selected for the M&E activities, as well as 33 of the 127 SBR Prep schools. TILO used a combination of eight tools developed by TILO as well as two instruments developed by USAID. Some changes were made in the structure of measurements and the tools used over the course of the project, to accommodate for changing circumstances on the ground. Measurements were conducted on an annual basis over a four year period, from October 2008 – April 2012. The findings, conclusions and recommendations are summarized below. Impact Result 1: “Improved Student Learning” Two variables - “performance over time” and “training” were examined in detail. The performance over time variable was analyzed annually from the baseline until one year after the end of interventions in TILO schools (referred to as the “sustainability year.”) The training variable was analyzed at the baseline and by looking at the average mid-year marks of students during the period of intervention. Two categories of TILO teachers were analyzed: those taught by TILO Trained Teachers (who were directly trained by TILO Master Trainers) and those taught by TILO Locally Trained Teachers (teachers at TILO schools who were trained by the TILO Trained Teachers). The major findings are as follows:  Performance Over Time: The academic performance of TILO students in both SBR Primary schools and TSS Prep schools showed significant improvement over the course of the intervention as measured by their average mid-year marks. Even the results of the measurement conducted at the “sustainability year” showed improvement compared to the baseline. These findings indicate Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 5 that the overall model and approach of the TILO interventions was sound and has potential for sustainability.  Training: In SBR Primary schools and TSS Prep schools, students trained both by “TILO Trained Teachers1 ” and by “TILO Locally Trained teachers2 ” improved significantly in their average mid￾year marks compared to the baseline. In fact, in SBR Primary schools, the average mid-year marks of students of TILO Locally Trained teachers were even higher than those taught by TILO Trained teachers. As experts in their specific subject areas, TILO Trained teachers could provide more targeted support to their colleagues than TILO Master Trainers were able to provide to them. This suggests that the TILO’s cascade model for training (training some teachers directly and having those teachers train others in their school) worked well. Impact Result 2: “Sustainable Decentralized Management of Use of Technology for Teaching/Learning at the School and Idara Levels” This result evaluated the use of technology for teaching and learning as well as decision-making, and the sustainable use of resources for maintaining and supporting education technology in schools. The main findings below relate to both SBR Primary and SBR Prep schools:  Use of technology at school and Idara levels: Both schools and Idaras significantly increased their use of technology for making data-driven decisions. Schools reported a more effective use of technology over the course of the TILO interventions, and were satisfied with the support received from their school management, Idaras and Muderiyas. A main reason cited was the inclusion of school managers, Idara and Muderiya supervisors in the TILO trainings, which allowed them to better cater to the needs of the schools. Teachers reported both using the resources they were provided and taking the initiative to provide e-resources themselves. This is a good indicator of teachers’ engagement and interest in using technology in their classrooms and a positive sign for their sustaining these practices after the end of the project.  Perceptions of technology: Both teachers’ and students’ perceptions of technology use in the classroom improved significantly over the TILO project. Teachers reported greater levels of confidence and competence in using technology and stated that using technology had changed their role to facilitators rather than lecturers. Students commented that their teachers’ use of technology helped them to feel freer to work better both independently and in teams. Teachers, on their part, reported that students were more engaged, attentive and collaborative. These findings suggest that the technology provided through the TILO approach was successful in engaging students’ attention and desire to remain in school.  Integrating technology into the teaching-learning process: While technology was certainly useful to teachers in their teaching, it was clear from the results that the teachers were able to use the technology in appropriate ways because they had first been trained on how to teach well. In the TILO model, technology was not considered an add-on but was consciously integrated into the curriculum at different grade levels. It was also included only after teachers had been trained on basic pedagogical techniques. Teachers reported very positively about the trainings they received on topics such as student-centered teaching, classroom management, etc. They reported that they adopted active learning, creative thinking, and problem solving strategies in the classroom when 1 TILO teachers trained by TILO Master Trainers 2 TILO teachers trained by TILO Trained Teachers Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 6 working with technology. This finding suggests the soundness of the TILO model in terms of using technology as a tool to enhance learning, rather than an end in itself.  Sustainable management of resources: While TILO was successful in increasing the effective use of technology at the school level, control of financial resources was not decentralized and financial allocation of technology resources was conducted primarily at the central level. This will make it difficult for Idaras and schools to plan financially for the proper maintenance and support of the technology in their jurisdictions.  Phasing of interventions in cohorts: The M&E results showed that Cohort 2 schools performed better than Cohort 1 and showed greater improvement in all categories evaluated through the M&E process. This corroborates the concept of working in phases – starting with a pilot, making necessary adjustments, and then scaling out to the remaining target areas. After piloting in Cohort 1 schools, the TILO team made adjustments to both the technology and training models, and the findings show that these adjustments made a difference not just to the smooth implementation of interventions in Cohort 2 but also to the results of these interventions. Recommendations  Replicate the Student Marks Study in 2014 and 2015 in TILO schools which have not received further interventions in order to see if the impact of the TILO activities have been sustained.  Include other relevant stakeholders (such as Idara and Muderiya education officers as well as school principals) in interventions involving training of teachers. This makes a big difference in terms of securing their buy-in and their engagement during the intervention, as well as their interest in and ability to continue to support the initiatives after the end of the project.  Consider ways to decentralize decision-making about the allocation of financial and technology resources to ensure the sustainability of TILO interventions.  Use a phased approach to implementing interventions so that adjustments can be made to address any problems identified in a pilot before the model is rolled out widely. Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 7 Acronyms BL1 Baseline Measurement in October 2008, (Cohort 1 schools from 3 governorates) BL2 Baseline Measurement in October 2009 (Cohort 2 schools in 4 governorates) BoT Board of Trustees (of schools) CAPS Critical Thinking, Achievement, and Problem Solving (CAPS) Test KS Kid-Smart, an IBM HW and SW package introduced within TILO through a PPP M1 Year 1 Measurement for Cohort 1 Schools in April 2009 M2 Mid-Project Measurement in April 2010, Cohort 1 and 2 schools in 7 governorates M3 End-Project Measurement in April 2011, Cohort 1 and 2 schools in 7 governorates M4 Sustainability measurement in April 2012 MOE Ministry of Education of Egypt PI Performance Indicator PMP Project Monitoring Plan RC Reading Companion, an IBM software package introduced within TILO as PPP SCOPE Standards-based Classroom Observation Protocol for Egypt (SCOPE) T1 TILO Data Collection Tool, Teacher Technology Portfolio Rubric T2 TILO Data Collection Tool, Teacher Focus Group Protocol T3 TILO Data Collection Tool, Student Focus Group Protocol T4 TILO Data Collection Tool, Teacher Questionnaire T4++ TILO Data Collection Tool, A combined Teacher Questionnaire introduced at M4 T5 TILO Data Collection Tool, Videotaped Classroom Observation evaluation form T6 TILO Data Collection Tool, School Director Interview Protocol T7 TILO Data Collection Tool, Idara / Muderiya Director Interview Protocol TDC Office of Technological Development Centre of the MOE Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 8 OVERVIEW The USAID-funded Technology for Improved Learning Outcome (TILO) project was implemented by Creative Associates International, with partners Keys to Effective Learning, PalTech and Seward Inc. TILO was designed to enhance the quality of teaching, learning and school management through the effective use of technology in schools. Creative worked with Egypt’s Ministry of Education (MOE) and other governmental and community-based entities to develop and implement a holistic, integrated model for introducing technology into school-based reform activities with the aim of improving learning outcomes in the intervention schools. The TILO intervention originally began in in 277 schools in 9 governorates: Greater Cairo (including Cairo, 6th of October and Helwan), Giza, Alexandria, Beni Suef, Assiut, Minya, Fayoum, Qena, and Aswan, to provide interventions in 192 SBR Primary and 85 TILO Smart Schools (TSS). In 2011, the project was extended for two years with a mandate to expand its interventions to preparatory schools which would be receiving TILO primary schools students after they had completed Grade 6. The TILO extension introduced the Prep school model to 127 schools, increasing the total number of TILO schools receiving interventions to 404 schools. Monitoring and Evaluation (M&E), the fourth TILO component, aimed to monitor the progress of the project and provide regular feedback to the project management on the performance of the project activities. This was done over the life of the program and across its geographical locations to allow the management to make informed decisions for adapting and improving implementation as the project progressed. TILO’s M&E process was conducted over four years based on a conceptual M&E framework and a Performance Monitoring Plan (PMP) approved by USAID that were designed to monitor the project’s intended results at three levels: impact, outcomes and outputs. Project activities were regularly assessed through baseline and yearly measurements, where information about the project implementation was acquired, analyzed, interpreted, and conclusions were drawn and recommendations presented to the project management. See Appendix 1: Performance Monitoring Plan (PMP). Impacts, Outcomes and Outputs The Monitoring and Evaluation (M&E) Advisors and the TILO team developed a comprehensive M&E framework and a Performance Monitoring Plan (PMP) which was designed to measure and report project performance and progress towards the achievement of TILO's goals and to provide evaluation feedback to improve planning and management decisions. The PMP aimed to monitor the outputs of TILO's activities being implemented as well as to evaluate outcomes achieved and long-term impacts. Output measurements showed the implementation of activities, while outcome measurements illustrated to what degree the anticipated results of these activities were being achieved. Impact assessments showed the degree to which the overall goal of the project was realized. As per the TILO PMP, M&E measurements followed a structured process based on assessing 40 performance indicators related to the eleven intended results in the three levels noted below: Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 9 TILO Monitoring and Evaluation Conceptual Framework The ultimate impact of TILO on teaching and learning depended on a series of intermediate results or outcomes which were achieved by implementing project activities according to plan. The TILO PMP (see Appendix 1) presents performance indicators, sources of data, methods of data collection, people responsible for data collection and frequency of data collection that were used to measure progress and impact of project activities. TILO measured progress on 40 performance indicators based on the 11 results noted above. TILO collected data on outputs quarterly and made baseline and annual measurements on outcomes and impact throughout the project. This final M&E report provides a final comprehensive assessment of the project’s performance in achieving the intended results as stated in the PMP. The report provides an overall summary of the methodology adopted by the M&E plan in monitoring project performance along the life of the project, offers a full account of the intervention’s results, challenges, lessons learned during implementation, and makes recommendations for future projects. IMPACT •1. Improved student learning •2. Sustainable decentralized management of use of technology for teaching / learning at the school and Idara levels OUTCOMES •3. Increased student engagement with IT •4. Changed classroom practice (Teachers) •5. Increased local support for IT in schools •6. Enhanced school management of IT •7. Enhanced MOE capacity for management of education technology resources OUTPUTS •8. Training (Teachers) •9. IT infrastructure and digital content (School) •10. Community awareness and public￾private partnerships •11. MOE capacity building program Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 10 APPROACH AND METHODOLOGY 1.1 Introduction Measurements for monitoring TILO activities consisted of several studies conducted at different times based on the purpose of each study. The studies monitored the performance of TILO teachers, students, and school management along the 40 performance indicators identified for assessing the 11 intended results at levels of output, outcome and impact. At the impact level, six performance indicators were analyzed related to two intended results. A major study was conducted at the end of the project activities which measured the change in students’ learning performance by the change in the mid-year exam marks of students in three school grades in five subjects over four school years in both SBR Primary and TSS Prep schools. This study was specifically introduced to address the impact level intended result “Improved Student Learning.” The second impact￾level result, “Sustainable decentralized management of use of technology for teaching / learning at the school and Idara levels,” was addressed by through various studies and will be further explained over the course of this report. At the outcome level, 15 performance indicators were analyzed related to five intended results. A multi￾component longitudinal study was conducted over the life of the project which monitored the change in teaching and learning at TILO schools along a number of dimensions. It assessed teachers’ performance in implementing reform-based teaching practice, their attitudes and confidence in using technology and implementing modern methodology of teaching and learning, and the change in their students’ learning behavior to match teachers’ shift to using reform-based practices and technology to improve teaching and learning. This study included several measurements, starting with baseline measurements followed by three annual measurements which concluded with a “sustainability” measurement one year after the completion of the TILO intervention in primary schools. The sustainability measurement used several tools including the Standards-Based Classroom Observation Protocol for Egypt (SCOPE) instrument (see details below), as well as teachers’ and students’ surveys, and focus groups and interview protocols for school management. At the output level, 19 performance indicators were analyzed related to four intended results. M&E information was gathered through document review activities that took place as needed based on the reporting cycle adopted by TILO. Data was regularly obtained for M&E purposes through TILO reporting cycles about TILO’s delivery of training programs, number and types of participants at different times and locations, status of provision of hardware and software to schools, activities related to building up relationships between TILO and the community and establishing Public Private Partnerships with relevant entities in the community. This process also took into consideration TILO’s efforts in building the capacity of the MoE staff to provide continuity and sustainability to TILO schools after the conclusion of the interventions. Implementing the TILO PMP over the life of the project was a dynamic process that adapted to changes in the implementation strategy, particularly as a result of the TILO extension in its fourth year to include prep schools, changes within the MOE, and in response to the situation in Egypt following the January 2011 revolution. The implementation of the TILO monitoring plan (M&E activities and measurements, data sources, data collection frequency, tools, personnel and the process for managing data collection runs) was modified a number of times and adjusted as needed to accommodate changing circumstances Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 11 over the course of the project. A detailed table of all the M&E measurements, studies and their status in reference to the PMP is presented below in the data collection section. Over the course of the project, certain data collection tools were discontinued and others were added to adapt to changing implementation circumstances (explained below). Two major studies were also introduced to meet the needs of new activities that were introduced over the course of project implementation, the Student Marks Study to address the change in student learning outcomes, and a qualitative study on SCOPE administration and performance. Additionally, TILO’s extensive work on the Public-Private Partnership (PPP) component necessitated additional M&E work to track those partnership activities, in particular the Discovery Channel Global Education Partnership (DCGEP) and the IBM partnership introducing Reading Companion Software and Kid-Smart units to selected TILO schools. The M&E activities for the PPP component were always planned in coordination and collaboration with the PPP partners and primarily addressed teachers’ and students’ confidence in, attitude to and satisfaction with using the technology brought to their classes through the PPP partners. These are all further described in Section 1.2 on Instruments. 1.2 Subjects Subjects for the M&E activities fell into the following categories: Idara and Muderiya Directors, school principals, teachers of core subjects, and students of different grades. All subjects from SBR Primary, TSS Prep and SBR Prep schools were selected from a representative sample of the project schools. Fifty￾seven of the 277 SBR Primary and TSS Prep and 33 of the 127 SBR Prep schools were selected for the M&E interventions. It is important to note that although the TILO intervention covered nine governorates (Greater Cairo including Cairo, 6th of October and Helwan, Giza, Alexandria, Beni Suef, Assiut, Minya, Fayoum, Qena, and Aswan), the M&E reports address changes in only seven governorates. The reasons are twofold: First, one of the Governorates, Assiut, was never included in the M&E activities because the TILO intervention there began after the other eight governorates and was only limited to 4 schools, which was insufficient for inclusion in the M&E sample; Second, the project interventions had consolidated Cairo, 6th of October and Helwan governorates together under Greater Cairo but had considered Giza separately. However, for M&E purposes, Giza was subsumed under Greater Cairo due to its proximity. Cohorts 1 and 2: TILO’s activities for SBR Primary and TSS schools were conducted in two phases, with two cohorts of schools.  Cohort 1 began TILO interventions in 2008 in Alexandria, Beni-Suef and Greater Cairo. The TILO team learned from the successes and challenges faced during this period and made adaptations to both the technology and training models before rolling out the interventions to Cohort 2 in 2009.  Cohort 2 began interventions in 2009 in Fayoum, Minya, Qena and Aswan. Accordingly, the M&E measurements for these schools followed the intervention schedule for each cohort (see Table 2 in Section 1.3.1). TILO interventions in SBR Prep schools began in 2011 and measurements for these schools were conducted in October 2011 (baseline) and April 2012 (mid-line). A detailed account of the school sampling methodology applied in the project measurements is addressed in Section 1.3.2 below. Within each governorate and school in the sample, principals, managers, and teachers were included in the subject population based on their posts at the time of data Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 12 collection for the baseline, prior to the start of the interventions. The selection of teachers in each school was conducted through a stratified random sampling technique. The aim was to randomly include one teacher of each of the six subjects: Math, Science, Social Studies, Arabic, English, and Computing, out of the pool of eight teachers identified to become “TILO Master Teachers” in their schools. Selection of students to fill out students’ surveys was also conducted randomly and was carried out on the day of data collection at the school with the help of the school administration. The number of subjects who responded to different tools at schools varied as follows:  1-2 School Managers (including the Principals);  6-8 teachers;  Up to 15 students responding to surveys and or focus groups;  1-2 high-level Idara and Muderiya MOE managers per Idara or Governorate, including Director of Muderiya “Modir Muderiya” and Director of Idara “Modir Idara.”  All students in grades 3, 4 and 5 in SBRs and 7, 8 and 9 in TSSs responded to the Students Marks study. From SBR Primary and TSS Prep schools, 1,664 subjects were targeted for participation from 57 TILO schools in seven governorates to respond to all the tools in the longitudinal study that spanned four years. Table 1a shows the distribution of the categories of respondents who participated in this study from each governorate. During the extension phase of the project, 226 teachers from 33 Prep model schools in seven governorates were targeted for participation in the baseline and midline study. Table 1a below has details. Table 1a: Number and Type of Subjects by Governorate # Governorate # and Type of Respondents from SBR Primary and TSS Schools # and Type of Respondents from Prep Model Schools Schools MOE Directors School Managers Teachers Students Schools Teachers 1 Alexandria 7 2 14 468 238 7 52 2 Aswan 6 2 6 36 37 3 20 3 Beni Suef 12 2 24 147 84 6 36 4 Greater Cairo 15 2 30 214 105 5 35 5 Fayoum 4 2 4 26 38 5 32 6 Minya 9 1 9 54 52 4 29 7 Qena 4 2 4 24 37 3 22 Totals 57 13 91 969 591 33 226 Additionally, 775 subjects provided data for the DCGEP studies, 95 for IBM Reading Companion (RC) and 472 for the IBM Kid-Smart (KS) studies as indicated in Table 1b below. Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 13 Table 1b: Number and Type of Subjects for PPP M&E activities PPP Partner Program Instrument used for Data Collection Participants # Type DCGEP SCOPE 50 Teachers DCGEP Teacher Survey 162 Teachers DCGEP Student Survey 563 Students IBM RC Teacher Survey 30 Teachers IBM RC Student Focus Groups 65 Students IBM IBM / HMH Kidsmart Teacher Survey 142 Teachers IBM IBM / HMH Kidsmart Student Focus Groups 330 Students Finally, in the Student Marks Study, the average mid-year exam marks for five subjects from students in three grades (for both SBR Primary and TSS prep schools) were gathered over four school years, yielding 211,000 marks which were used in the data analysis for this study. For this study, students’ marks were obtained based on the whole class as the unit of selection. 1.3 Instruments The TILO PMP included a number of data collection tools designed to accommodate the diversity and the large number of intended results to be assessed for monitoring the project activities. Eight of these instruments were developed by TILO M&E advisors and were labeled TILO Tools T1-T7 and T4++. The other tools, Critical Thinking, Achievement and Problem Solving (CAPS) and Standards-Based Classroom Observation Protocol for Egypt (SCOPE), had been previously produced, piloted and used by USAID projects in Egypt. A full description of each of these tools follows in this section. 1.3.1 TILO Instruments/Tools Originally, seven TILO instruments were designed to gather both quantitative and qualitative data related to the project’s intended results. The instruments were set to be administered for baseline and yearly measurements within a longitudinal study along the project lifetime. All TILO tools were produced in English and translated to Arabic for administration. Most of the seven TILO tools were used for the first three years of the project (from October 2008 – April 2011). However, when the TILO extension was approved by USAID (to focus on prep schools), the Project Monitoring Plan was modified to fit the new extension scope. Several of the TILO tools were combined into an eighth tool, T4++, which was administered for the October 2011 and April 2012 measurements, along with the USAID-SCOPE tool (described in Section 1.2.2). The eight TILO tools are described below. 1.3.1.1 T1, Teacher Product Rubric Developed to assess the desired outcome of increasing teachers’ performance in integrating technology into student-centered methods (outcome level results), this tool assessed the content of Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 14 teachers’ portfolios of ICT integrated teaching materials produced and/or used in their classrooms (PI3 3.2). This tool was administered once in the first year but was discontinued later due to logistical difficulties. 1.3.1.2 T2, Teacher Focus Group Protocol This tool was developed to acquire data on two levels: at the impact level by capturing “sustainable decentralized management use of technology for teaching/learning at the school and Idara levels” (PI 2.3), and at the outcome level by capturing “changed classroom practice” (PI4.1). T2 was administered as planned for baseline and yearly measurements until the Year 3 measurement in April 2011. During the TILO extension period, T2 was rolled into the T4++ instrument. 1.3.1.3 T3, Student Focus Group Protocol T3 was developed to collect information from students about two TILO results at the outcomes level: increased student engagement with IT (PI 2.1 and 2.2) and changed classroom practice (PI 6.1). T3 was administered as planned for the baseline and for three consecutive yearly measurements and discontinued during the extension period. The PIs 2.1, 2.2 and 6.1 were later assessed through the SCOPE classroom observation protocol. 1.3.1.4 T4 TILO Teacher Questionnaire / Survey T4 was designed to elicit information from TILO teachers about their classroom practice (outcomes level result, PIs: 4.1, 4.2, 5.1, and 6.1). T4 was administered as planned for baseline and the first three yearly measurements (through April 2011), and then T4++ was introduced for the measurements during the extension period. Information on students during the extension period was collected through the SCOPE observation tool. 1.3.1.5 T5, TILO Class Video Observation Tool T5 was designed to acquire information about increased student engagement with IT (outcomes level result, PI 1.1, level of technology integration in student learning activities). Information was obtained through video recording of TILO classes and having trained educators watch the video recordings to assess the level of technology integration in the lessons by responding to 14 items on T5. This tool was administered only once during the first year, but was discontinued later as the cost of administration was too high and the availability of sophisticated equipment such as video cameras with directed microphones for clear voice videotaping was limited. In addition, the activity was distracting for students during the duration of the lesson. The SCOPE class visit protocol was used afterwards as a substitute. 1.3.1.6 T6, TILO Technology Management Assessment Protocol T6 was designed to acquire information from TILO School Principals about the desired impact of “sustainable decentralized management of use of technology for teaching/learning at the school and Idara levels” (impact level result, PIs 2.1 and 2.3 a, b); and information about three results at the outcomes level: increased local support (PIs 7.1 and 8.1); and enhanced school management of IT (PI 10.1). T6 was administered as planned for the baseline and three yearly measurements, but was discontinued during the extension period. During the extension period, necessary data about management of technology in schools was obtained through teachers’ reporting on the T4++ instrument. 1.3.1.7 T7, Idara and Muderiya Interview Protocol This tool was designed to acquire information about how MOE Management at the Muderiya and Idara levels supported technology in education and how they perceived TILO’s intervention in their schools. T7 was administered for the baseline and two yearly measurements, but was discontinued later due to the unstable situation at the Idara and Muderiya levels after the January 2011 revolution. 3 Performance Indicator Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 15 1.3.1.8 T4++ During the TILO extension, T4 was modified to T4++ to accommodate the new mandate and extension granted by USAID for implementing the TILO prep school model. The questionnaire which had included 26 controlled and short response items was expanded to 35 items addressing teachers’ perception of and confidence in using technology in teaching, attitudes towards using technology, levels of satisfaction with their management’s support for technology, mastering reform-based pedagogy in their classrooms, and using technology for teaching and learning. The T2 and T6 tools were also consolidated into the T4++ tool. This tool was administered for the October 2011 and April 2012 measurements. 1.3.2 USAID-Produced Instruments The TILO PMP also included two USAID-produced measurement tools: CAPS and SCOPE. The two instruments were used within the M&E framework to particularly assess the project’s impact on teachers and students. 1.3.2.1 CAPS The CAPS (Critical Thinking, Achievement and Problem Solving) test was originally selected to acquire information about the impact level result, improved student learning (PI 1.1). CAPS was administered only once through the USAID ERP Project in May 2010. However, the students’ scores on the tests were never submitted to TILO from the MOE, as a result of changes in management at the National Center for Examinations and Educational Evaluation (NCEEE) which put the exercise on hold. As a result, TILO decided to devise another method for assessing the impact of the intervention on the academic performance of TILO students: The Student Mid-year Marks study. This study is described in more detail in Section 1.2.6. 1.3.2.2 SCOPE The Standards-Based Classroom Observation Protocol for Egypt (SCOPE) was designed to “assess teachers’ use of practices that are characteristic of the standards and reform-based teaching methods outlined in the National Standards for Education in Egypt, and to also measure student behaviors that reflect the development of problem solving and critical thinking skills.”4 SCOPE was included in TILO measurements to assess changes in classroom practice as a result of teachers’ integration of technology in classes (outcomes level result PI3.1). SCOPE was administered as planned for all TILO measurements that took place throughout the life of the project. Additionally, it was used as a substitute for some of the TILO tools that were discontinued during the extension period in October 2011 and April 2012 (such as T3 and T5). 1.3.3 Documents Review A set of documents was included within the PMP as a means of acquiring information at the three levels of results. These documents were: field reports, regular project reports, and the TILO database. 1.3.4 Additional Data Collection Tools and Studies Introduced Ten additional data collection tools were introduced during the course of the project to address the changing needs of the project. Adaptive changes in the M&E plan were not limited to modifying tools; 4 ERP SCOPE IV Report, 08 Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 16 there were also a number of studies designed to demonstrate the monitoring of TILO activities that were introduced to the intervention during the implementation period. 1.3.4.1 DCGEP instruments The Discovery Channel Global Education Partnership (DCGEP) partnered with TILO under the PPP component to provide educational videotapes in a number of subjects that were translated into Arabic, approved by the MOE and mapped to the Egyptian curriculum. The videos are an excellent resource, used in TILO classrooms as a tool for improving learning outcomes. Data related to the implementation of the DCGEP intervention was collected, translated to English, digitized and submitted to DCGEP for analysis. No data analysis or reporting was required from the TILO M&E team in this respect. Three tools were devised for DCGEP M&E purposes--a modified SCOPE observation practice, a student survey and a teacher survey. The surveys were produced by DCGEP and translated to Arabic by the TILO M&E team while the SCOPE items were used as is, except for some modifications to the way the observation was conducted in class and the scoring of teacher performance. The DCGEP team, in collaboration with the TILO M&E Advisor, conducted training for the new “DCGEP SCOPE” and the survey data collectors to ensure the reliability of data collected. 1.3.4.2 IBM IBM partnered with TILO under the PPP component to provide TILO schools with access to IBM’s web-based Reading Companion program and the KidSmart Young Explorer hardware units. In addition to the hardware units, Houghton Mifflin Harcourt, the owners of the Kidsmart software agreed to donate lifetime licenses to TILO MOE schools. The software was installed on all TILO equipment in schools. Four tools were created to collect data about these partnerships. TILO designed and used a teacher survey and a student focus group protocol for both Reading Companion and KidSmart. IBM tools became part of TILO tools and were administered during data collection runs with the rest of the data collection activities. The data was especially useful to IBM, since TILO schools were the first to use IBM’s resources in Egypt. 1.3.5 SCOPE Qualitative Study Three tools were introduced to respond to the Standards-Based Classroom Observation Protocol for Egypt (SCOPE) data requirements for the Qualitative Study: a trainers’ focus group protocol, a TILO Governorate team leader interview protocol and a SCOPE supervisor focus group protocol. These tools were used only once, in September 2012, to obtain data on the context of the SCOPE measurement in April 2012 to support the data analysis of the quantitative data obtained through administering SCOPE observations for the sustainability year measurement. 1.3.6 Students Mid-year Marks In the absence of CAPS data, TILO designed a new study to obtain and analyze students’ mid-year marks in five subjects across four school years for three school grades at the SBR primary level (grades 3-5) and TSS prep level (grades 7-9). The analysis was then used to draw conclusions about changes in students’ performance in TILO schools. No new or TILO-specific data collection tools were introduced for the Student Marks Study. Schools’ records of student marks for mid-year exams were gathered from schools and digitized for analysis. Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 17 1.4 Data Collection Process At the beginning of the project, a full data collection plan for the project was developed within the PMP and the M&E plan. The TILO M&E Advisors and COP participated in making decisions regarding sampling strategies, administration intervals for data collection tools, and criteria for data collector selection, training and handling. They also identified and approved all budgetary requirements for the data collection and other M&E-related activities. During project implementation, data collection runs were planned collaboratively between the M&E Advisor and TILO senior management to agree upon the timing, duration and specific activities to be conducted. A comprehensive plan for each M&E run was then produced, along with timetables for school visits, meetings with subjects, names of subjects and data collectors, communication channels and reporting lines during the runs, and of course deadlines for data submission by data collectors. The process of data collection was a yearly exercise that generally took between two to three weeks and was always conducted near the end of March or beginning of April. This timeline was established so that teachers and schools would have completed the intervention activities for the school year and could demonstrate their implementation of activities they had learned from TILO. During the data collection runs, all data collection teams’ activities were monitored closely and a daily reporting cycle by team leaders was put in place for resolving any field problems that may arise during the process. Data collection runs were monitored by the TILO M&E Advisor in coordination with the TILO governorates’ Team Leaders. Upon completion of data collection work in the field, data documents were sent to the M&E Advisor at the central TILO office for analysis and reporting. A report on the data collection process was issued at this stage, highlighting issues faced during the data collection process and recommendations for resolving any pending issues. Following the data collection, data processing was conducted for approximately 4-5 weeks. During this stage, data documents were counted, coded, and stacked. Data processing teams conducted the data entry process, where all data (quantitative and qualitative) were digitized in predesigned tables on excel sheets and/or SPSS files. Electronic data files produced during this stage were reviewed for data cleaning and to produce the final data files for analysis. For data analysis, collective tables and graphs were produced out of the data tables to respond to the various study questions for each run. Results of the analyses were then reported in formal reports issued by the M&E Advisor and submitted to TILO HQ and USAID. 1.4.1 Data Collection Runs Seven major measurement runs were conducted over a span of four years, between October 2008 and April 2012.  Measurements for SBR primary school were conducted from October 2008 or 2009 through M4, April 2012.  Measurements for TSS Prep schools in Cairo were conducted from October 2008 or 2009 through M3, April 2011.  Measurements for SBR Prep schools were conducted for one school year, from the baseline in October 2011 to the mid-line in April 2012, during the TILO extension period. Table 2 below shows the details of these runs. Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 18 Table 2: TILO M&E measurements / Data Collection Runs Dates, Measurement / Data Collection Runs, Activities # Data collection tools used Comments October 2008, BL1 BL1, Baseline 1. Measurement in 3 governorates: Cohort 1 SBR Primary and TSS schools. 6 tools T1 was not used for baseline (no teachers’ portfolios available). CAPS was not used as it was not available. T2-T4 and T6-T7 and SCOPE were used. April 2009, M1 M1, Year 1. Measurement in 3 governorates: Cohort 1 SBR Primary and TSS schools. 1 tool Only SCOPE was used as training of teachers had taken place. Due to delay in installation of hardware and digital resources, other T instruments were not used. October 2009, BL2 BL2, Baseline 2. Measurement in 4 governorates: Cohort 2 SBR Primary and TSS schools. 6 tools T1 was not used for baseline (no teachers’ portfolios available). CAPS was not used as it was not available. T2-T4 and T6-T7 and SCOPE were used. April 2010, M2 M2, Year 2 measurement. Third measurement for Cohort 1 and second measurement for Cohort 2 – SBR Primary and TSS schools. DCGEP Pre measurement in SBR Primary schools. All tools 3 tools All tools T1 – T7 were used in this run. 2 surveys and DCGEP SCOPE. October 2010 IBM Reading Companion and Kid￾Smart. First measurement in SBR Primary and TSS schools. 4 tools Specific tools were designed to measure the IBM Reading Companion and Kid-Smart. April 2011, M3 M3, Year 3 measurement. Fourth measurement for Cohort 1 and third measurement for Cohort 2 – SBR and TSS schools. DCGEP Post measurement. All tools 3 tools All tools T1 – T7 were used in this run. surveys and DCGEP SCOPE. Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 19 IBM Reading Companion and Kid￾Smart. Second measurement in SBR Primary and TSS schools. 4 tools Specific tools were designed to measure the IBM Reading Companion and Kid-Smart. October 2011 BL for Prep schools: TILO Teacher Survey Pre-Intervention Measurement and SCOPE 1 tool T4++ and SCOPE April 2012 M4, Year 4 measurement. Sustainability measurement for Cohorts 1 and 2 SBR Primary schools. SBR Prep schools mid-line measurement Students Mid-Year Marks Study 2 tools 2 tools Schools Marks reports T4++ and SCOPE Cairo TSS schools were not included in this measurement. T4++ and SCOPE Students’ mid-year marks September/October 2012 SCOPE Qualitative Study (Supplementary report on SBR Primary and Prep schools) 1 tool SCOPE 1.4.2 Sampling In planning for the project M&E activities and measurements, TILO decided to use the school as the unit of sampling for all measurements. A sampling ratio of 20% was adopted for selecting a representative sample of TILO schools for the project’s M&E interventions based on a stratified random sampling plan. Following this strategy, data collection for all measurements took place in 90 schools in seven governorates. The 90 schools were a combination of SBR Primary schools and TSS prep schools (all Cairo schools were TSS schools) as well as SBR Prep schools. Table 3 shows the TILO M&E school sample distribution in the seven governorates. It should be noted that the governorate of Assiut, although it was part of the TILO intervention, was not part of the M&E plan. Assiut was included in TILO interventions later in the project timeline and it only had four schools. The logistics and resources that would have been required to include the four schools in the M&E measurements could not be justified. Table 3: Sample SBR Primary and TSS M&E Schools in Target Governorates Governorate SBR Primary Schools TSS Prep Schools SBR Prep Schools Total Alexandria 6 1 7 14 Aswan 6 0 3 9 Beni Suef 11 1 6 18 Greater Cairo 0 15 5 20 Fayoum 4 0 5 9 Minya 9 0 4 13 Qena 4 0 3 7 Total # of Schools 40 17 33 90 Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 20 1.4.3 Data Collectors A total of 60 data collectors participated in conducting TILO measurements over the lifetime of the project. These were carefully chosen by the M&E Advisors based on their professions and their previous experience participating in similar tasks. Data collectors were separated into two groups: a) 31 TILO Tools data collectors, and b) 29 SCOPE MOE Supervisors. The 31 TILO Tools data collectors were all experienced data collectors who lived in or near the seven governorates included in TILO M&E measurements. They were interviewed by the M&E Advisor prior to their selection for the task. These data collectors were considered to be experts in administering data collection tools such as surveys, interviews, and focus groups to different populations and age groups. They participated efficiently in conducting six TILO data collection runs that took place over five years in the 57 SBR Primary and TSS schools in seven governorates. During this period, they successfully performed around 360 data collection activities and administered 360 interviews, 1000 focus groups and administered the filling out of 4,000 questionnaire forms. The SCOPE group consisted of 29 ERP SCOPE-trained MOE Supervisors who lived and worked in the seven governorates. They were chosen from non-TILO Idaras to maximize their objectivity and the fairness of the classroom observation process. The MOE Supervisors who participated in the TILO SCOPE observations were experts in administering the SCOPE classroom observation protocol, using its rubric and reporting on the observed teachers both quantitatively and qualitatively. They participated efficiently in conducting around 2,000 SCOPE classroom observations of TILO teachers during six TILO data collection runs that took place over five years in the 57 SBR Primary and TSS schools in seven governorates. Prior to beginning data collection activities, both the TILO Tools and SCOPE data collectors in all the governorates received a one-day orientation conducted by the TILO M&E Advisor on the project’s objectives and how to administer the tools assigned to them. All data collection activities at TILO schools were carefully coordinated by the TILO office with the involvement of the MOE Muderiyas and Idaras. 1.5 Limitations of the Study There were certain limitations to the M&E measurements and analyses conducted that are noted below:  Selection of Control schools: The selection of Control schools for the SBR Primary and TSS Prep school interventions were out of the project’s control. SBR Primary schools were selected by the MOE and were some of the best schools in the Idaras. The TILO M&E team was not able to ensure that these schools were truly comparable to the TILO schools or representative of the MOE schools’ norm in the target areas. In contrast, as it was TILO’s mandate to provide support to underprivileged areas, many of the TILO schools were some of the most underserved in the Idaras. For the TSS Prep school intervention, it was found that some of the Control schools received interventions from other sources, which may explain their particularly high level of improvement over time. These factors made it difficult to easily compare the results of TILO and Control schools at both the SBR Primary and TSS Prep levels.  Student Marks Study Data Sources: The data analysis of the Student Marks Study was bound by specific research questions that focused on the “school year” and “training”. Measurements were taken for the other variables (“governorate,” “subject,” and “grade”) but the baseline data Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 21 was not disaggregated by these variables. Therefore, it was not possible to show improvement over the course of the project by these variables.  Changes in Tools and Instruments: Some changes were made in the structure of measurements and the tools used over the course of the project. Certain tools were withdrawn after the M2 or M3 measurements due to a variety of reasons (see Methodology section for details) and the SCOPE and T4++ tools were used for the M4 (April 2012) measurements. Due to these changes, information is not always provided for all three categories of schools. Some examples are listed below: o SBR Prep schools were only measured during the project’s extension period, from 2011-2012. Therefore, any measurement comparisons presented before this period do not include SBR Prep schools. o TSS schools were only measured through M3. Therefore, any comparisons of 2008/2009 baseline measurements to M4 (April 2012) are only provided for SBR Primary only. o For certain indicators, the information at M4 is presented, but since the baseline data were collected using a different tool, it is not possible to show improvement over time. Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 22 RESULTS and FINDINGS Assessing whether the TILO project was attaining its intended results was part of a regular process that accompanied project implementation and was conducted regularly through annual measurements throughout the project’s lifetime. A full assessment of TILO’s performance and a descriptive analysis of each result as per the project’s PMP will be presented in this final M&E report. This section presents the findings for the 11 “intended” results that were identified for achieving the project’s goal, “improved learning outcomes through integrating technology”. For the purpose of this report, TILO’s 40 performance indicators (PIs), which are identified within the PMP, are used to guide the explanation of the findings, which are listed in clusters relating to each performance indicator. The overall findings and results are presented, and recommendations are made for future projects. 2.1 Impact Level Results Two results were assessed at the impact level:  Improved student learning, and  Sustainable decentralized management of use of technology for teaching/learning at the school and Idara levels. 2.1.1 Impact Intended Result 1: Improved Student Learning Improved student learning was intended to be assessed by examining one performance indicator, the increase on Critical Thinking, Achievement and Problem Solving (CAPS) test scores at TILO schools (impact PI 1.1). However, as indicated in previous sections, CAPS results could not be used for TILO purposes and therefore a substitute indicator was introduced: the “the increase in TILO students’ mid￾year exam marks.” A Student Marks Study was conducted at the end of the project which examined TILO’s impact on students’ average mid-year marks along five independent variables: performance over time, training, governorate, subject, and grade. The two variables that were examined in detail for improvement over time were “performance over time” and “training.” The performance over time variable was analyzed at the baseline, during each year of intervention, and one year after the end of intervention in the TILO schools (referred to as the “sustainability year.”) The training variable was analyzed at the baseline and by looking at the average mid-year marks during the period of intervention. This analysis examined two categories of TILO teachers: those taught by TILO Trained Teachers (who were directly trained by TILO Master Trainers) and those taught by TILO Locally Trained Teachers (teachers at TILO schools who were trained by the TILO Trained Teachers). As noted in the limitations section of this report, it was not possible to disaggregate the baseline data for the “governorate,” “subject,” and “grade” variables within the available resources of this study. Summary of Findings The results of the study revealed that the academic performance of TILO students in both SBR Primary schools and TSS Prep schools showed improvement over the course of the intervention and even at the end of the sustainability year. Additionally, both students trained by TILO Trained Teachers and by TILO Locally Trained teachers showed an improvement in their average mid-year marks as compared to the baseline. All comparisons were tested for statistical significance and were proven to be statistically significant at a level of 0.05 or within a 95% confidence interval. Because it Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 23 was not possible to disaggregate the baseline data by governorate, subject, and grade, it is not possible to comment on improvement along these variables over the course of the intervention. However, the average mid-year marks by governorate, subject, and grade are reflected in the section below. The following section summarizes the findings of the Student Marks Study by variable. Detailed Findings Performance Over Time: SBR Primary schools: An overall comparison of SBR Primary schools’ students’ academic performance over the four years revealed that TILO students showed an overall improvement over the intervention period compared to the baseline measurement in 2008-2009. TILO students’ performance also generally improved more over the course of the intervention than Control students5 . See Chart 1A below for details. The average mid-year marks for TILO students at the baseline measurement (2008-2009) was 61.4%. This improved over the duration of the TILO project, with the greatest improvement of 9.8% (to 71.2%) during the final intervention year in 2010-2011. Year 2011-2012, (one year after the end of interventions, otherwise called the sustainability year), showed a slight drop of 3.8% from the previous year, down to 67.4%. However, TILO students’ performance during the sustainability year was still 6% higher than the baseline. This pattern of performance indicates of the success and potential sustainability of the TILO model. The baseline for Control school students was 64.6%, which was higher than TILO students (explained earlier in the report). The average mid-year marks for this group improved over the course of the intervention, with the greatest improvement of 6.3% (to 70.9%) during the final intervention year in 2010-2011. During the intervention years, TILO students’ maximum improvement was higher than that of Control school students (9.8% improvement versus 6.3% improvement). In the 2011-2012 sustainability year, the average mid-year marks for both TILO and Control school students dropped from the previous year. This can in part be attributed to the effects of the 2011 revolution in Egypt, which certainly impacted the education system and schools along with all other sectors. The drop for TILO schools was greater than Control schools, which is also understandable as it was the first year in which the schools did not receive dedicated support from TILO. 5 Non-TILO schools selected by Idaras to provide marks representing the norm of students’ performance in the Idara 61.4% 64.6% 69.4% 70.2% 71.2% 70.9% 67.4% 70.6% 0% 20% 40% 60% 80% 100% 2008-2009 (Baseline) 2009-2010 2010-2011 2011-2012 Chart 1A: SBR Primary Students' Performance - Comparison of Average of Overall Marks (%) - Performance Over Time TILO Control Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 24 TSS Prep schools: An overall comparison of TSS Prep schools’ students’ academic performance over the four years revealed that TILO students showed an overall improvement over the intervention period compared to the baseline measurement in 2008-2009. See Chart 1B below for details. The baseline measurement (2008-2009) for Control school students was lower than TILO students, at 63.3%. The average mid-year marks for Control school students improved over the course of the intervention. They showed the greatest improvement in the 2009-2010 year of 17% (to 80.3%) and culminated with an improvement of 14.8% (to 78.1%) during the sustainability year of 2011-2012. The improvement of Control schools students was overall higher than that of TILO students. As noted in the limitations section of this report, a possible reason for the high improvement in TSS Control schools is the fact that these schools also received interventions from other sources over the course of the TILO project, which could have affected their results. The average mid-year marks for TILO students at the baseline measurement (2008-2009) was 69.3%. This improved over the duration of the TILO project and ended with an improvement of 13.7% (to 83%) during the sustainability year in 2011-2012. The fact that TILO students continued to show improvement one year after the end of the interventions is a positive sign for the sustainability of the model. Training: SBR Primary Schools: An overall comparison of TILO Primary schools’ students’ academic performance to the performance of Control school students revealed that the average mid-year marks of students in TILO schools showed greater improvement over the course of the intervention than those in Control schools. Also, students taught by TILO Locally Trained teachers performed better than those taught by TILO Trained Teachers. The average mid-year marks of TILO students at the baseline was 3.2% lower than that of Control school students. As noted in the “Limitations of the Study” section of this report, this is likely because the Control schools chosen were specifically selected by the MOE and happened to be some of the best schools in the Idara. However, as shown in Chart 2A, a comparison of baseline data with the average mid-year marks showed that, over the course of the intervention, the performance of TILO students improved more than Control school students. The average mid-year marks of Control school students increased by 4.6% points from the baseline (from 64.6% to 69.2%). In TILO schools, the 69.3% 83.1% 79.3% 83.0% 63.3% 80.3% 78.5% 78.1% 0% 20% 40% 60% 80% 100% 2008-2009 (Baseline) 2009-2010 2010-2011 2011-2012 Chart 1B: TSS Students Performance - Comparison of Average of Overall Marks (%) - Performance Over Time TILO Control Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 25 average mid-year marks of students taught by TILO Trained teachers6 improved by 6.3% points from the baseline (from 61.4% to 67.7%), and those of students taught by TILO Locally Trained teachers7 improved by 9.3% points (from 61.4% to 70.7%). An interesting finding is that the average mid-year marks of students of TILO Locally Trained teachers were even higher than those taught by TILO Trained teachers. This was arguably a result of the high motivation of TILO Trained Teachers to pass on their knowledge to their colleagues. They knew the challenges their colleagues were facing and were familiar with the teaching environment, which allowed them to address their problems in a relevant manner. Additionally, the TILO Trained Teachers were experts in their specific subject areas and were able to provide more targeted support to TILO Locally Trained teachers than TILO Master Trainers were able to provide to them. TSS Prep Schools: An overall comparison of TILO TSS Prep schools’ (predominantly in the Greater Cairo area), students’ academic performance to the performance of Control school students revealed that the average mid-year marks of students of TILO Trained teachers showed greater improvement over the course of the intervention than those from Control schools. The average mid-year marks for TILO students at the baseline was 6% higher than that of Control school students. As shown in Chart 2B, a comparison of baseline data with the average mid-year marks over the course of the intervention showed that the performance of students of TILO Trained teachers improved more than Control school students. The average mid-year marks of Control school students increased by 11.8% points over the course of the intervention (from 63.3% to 75.1%). In TILO schools, the average mid-year marks of students taught by TILO Trained teachers improved by 13.1% points (from 69.3% to 82.4%). The average mid-year marks for students of TILO locally trained teachers improved by 5.7% points (from 69.3% to 75%). This group also showed improvement but not as much as the Control schools students or students of TILO Trained teachers. 6 TILO teachers trained by TILO Master Trainers 7 TILO teachers trained by TILO Trained Teachers 64.6% 69.2% 61.4% 67.7% 70.7% 0% 20% 40% 60% 80% 100% Control Baseline Control (average over intervention) TILO Baseline TILO Trained (average over intervention) TILO Locally Trained (average over intervention) Chart 2A: SBR Primary Students' Performance - Comparison of Average of Overall Marks (%) - Training Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 26 Governorate: As the baseline data for the Student Marks Study was not disaggregated by Governorate, it is not possible to comment on any improvements by Governorate for either TILO or Control schools over the course of the intervention. The data in Chart 3 provides a snapshot of the average mid-year marks over the course of the intervention by Governorate for both TILO and Control SBR Primary schools. No governorate comparison was conducted for TSS Prep students as measurements essentially only took place in one governorate - Greater Cairo. School Grades: As the baseline data for the Student Marks Study was not disaggregated by school grade, it is not possible to comment on any improvements by grade level for either TILO or Control schools over the course of the intervention. The data in Charts 4A and 4B respectively provide a snapshot of the average mid-year marks over the course of the intervention by school grade for both TILO and Control schools in SBR Primary and TSS Prep schools. 63.3% 75.1% 69.3% 82.4% 75.0% 0% 20% 40% 60% 80% 100% Control Baseline Control (average over intervention) TILO Baseline TILO Trained (average over intervention) TILO Locally Trained (average over intervention) Chart 2B: TSS Prep Students' Performance - Comparison of Average of Overall Marks (%) - Training 66.7% 79.7% 63.8% 68.1% 69.8% 83.8% 66.8% 71.8% 72.4% 61.8% 71.6% 73.1% 0% 20% 40% 60% 80% 100% Alex Aswan Beni-Suef Fayoum Minya Qena Chart 3: SBR Primary Students' Performance - Comparison of Average of Overall Marks (%) - Governorate TILO Control Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 27 Subject: As the baseline data for the Student Marks Study was not disaggregated by subject, it is not possible to comment on any improvements by subject for either TILO or Control schools over the course of the intervention. The data in Charts 5A and 5B respectively provide a snapshot of the average mid-year marks over the course of the intervention by subject for both TILO and Control schools in SBR Primary and TSS Prep schools. 73.1% 72.5% 67.8% 70.2% 65.2% 71.6% 0% 20% 40% 60% 80% 100% Grade 3 Grade 4 Grade 5 Chart 4A: SBR Primary Students' Performance - Comparison of Average of Overall Marks (%) - Grade TILO Control 69% 83% 81% 63% 80% 78% 0% 20% 40% 60% 80% 100% Grade 7 Grade 8 Grade 9 Chart 4B: TSS Prep Students' Performance - Comparison of Average of Overall Marks (%) - Grade TILO Control Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 28 Conclusion Overall, TILO students showed improvements over the course of the intervention as compared to the baseline, and the results for both SBR Primary and TSS Prep students in the 2011-2012 sustainability year (one year after the end of the interventions) show improvement as compared to the baseline. Additionally, both students of TILO trained teachers and TILO Locally Trained teachers showed an improvement as compared to the baseline. In SBR Primary schools, TILO students showed a greater level of improvement than Control schools over the course of the intervention. As noted previously, all comparisons reported are statistically significant at a level of 0.05 or within a 95% confidence interval. Based on the findings of both the SBR Primary schools in six governorates, and the TSS Prep schools in Cairo, it can be concluded that TILO has successfully achieved the intended Result #1, “Improved Student Learning.” 67.3% 73.7% 68.5% 68.1% 71.5% 64.1% 74.5% 65.2% 71.0% 72.6% 0% 20% 40% 60% 80% 100% Arabic English Math Science Social Studies Chart 5A: SBR Primary Students' Performance - Comparison of Average of Overall Marks (%) - Subject TILO Control 84% 79% 74% 79% 78% 82% 79% 71% 76% 68% 0% 20% 40% 60% 80% 100% Arabic English Math Science Social Studies Chart 5B: TSS Prep Students' Performance - Comparison of Average of Overall Marks (%) - Subject TILO Control Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 29 2.1.2 Impact Intended Result 2: Sustainable Decentralized Management of Use of Technology for Teaching/Learning at the School and Idara Levels This impact level result was assessed by examining five indicators:  Data driven decisions at the Idara and school levels  Effective use of technology for teaching and learning in schools  Sustainable use of resources of education technology in schools: Functioning maintenance system (measurements conducted for SBR Primary and TSS Prep)  Sustainable use of resources of education technology in schools: Plan and budget for IT resources periodic updating  Schools introducing technology-based innovations As mentioned in the Methodology section, some changes were made in the structure of measurements and the tools used over the course of the project. Certain tools were withdrawn after the Year 3 measurement (M3 in April 2011) and the SCOPE and T4++ tools were used for the M4 (April 2012) measurements. As a result, for certain sub-indicators the information at M4 is presented, but since the baseline data were collected using a different tool, it is not possible to show improvement over time. Table 4 shows the data sources/tools that were used in order to acquire data to assess the five indicators pertaining to Impact Intended Result 2. Summary of Findings  Data driven decisions at the Idara and school levels (2.1.2.1): This indicator looked at the percentage of schools that reported “effectively using technology in the decision making process at school.” For SBR Primary schools, this percentage improved by 73% between baseline and M3, and by 61% for TSS Prep schools. Measurements were conducted for SBR Primary and TSS Prep from baseline through the M3 measurement (April 2011). No results are presented for SBR Prep schools as they began interventions in 2012.  Effective use of technology for teaching and learning in schools (2.1.2.2): Measurements were conducted for SBR Primary schools from baseline through the M4 sustainability measurement (April 2012) and for SBR Prep schools from baseline through mid-line (April 2012). No results are presented for TSS Prep schools as these schools were not measured after M3. Four items were examined: o TILO schools’ performance in integrating technology in teaching and learning: This item measured teachers’ progress in using technology for effective teaching and students’ behavior in engaging technology in their learning.  SBR Primary schools: Teachers from Cohort 1 schools improved by 29% on the SCOPE scale between baseline and M4, and students improved by 17%. Teachers from Cohort 2 schools improved by 57% between baseline and M4, and students improved by 45%. Comparisons are statistically significant at a 95% confidence interval.  SBR Prep schools: Teachers in SBR Prep schools improved by 27% on the SCOPE scale between the baseline and mid-line measurements and students improved by 16%. Comparisons are statistically significant at a 95% confidence interval. o Teachers’ confidence and comfort using technology in education:  SBR Primary schools: Teachers who reported “full” confidence in using technology decreased by 11% between baseline and M4. However, those who Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 30 reported being confident “to some extent” rose by 32% and those who reported “not having confidence” fell by 21%. These teachers had very limited technology in their classrooms at the start of TILO interventions, which explains the decrease in “full” confidence responses as they were handling much more complicated technology by M4. However, the overall confidence level increased over the interventions, as noted by the increase in confidence “to some extent” and by the fact that those “not having confidence” fell to 2% by M4.  SBR Prep schools: Teachers who reported “full” confidence rose by 13% between baseline and mid-line. Those who reported “to some extent” remained at 64% over the course of the study, and those who reported “not having confidence” fell by 10%. o Teachers' pedagogical competence comparison. This item analyzed teachers’ reporting on student-centered learning, classroom management, teaching towards critical thinking and problem solving skills, change management and team leading skills.  SBR Primary schools: There was an improvement for all items analyzed between baseline and M4 measurements, with improvements ranging between 8% and 16% and the percentage of teachers reporting positively on these items at M4 ranging from 92%-99%.  SBR Prep schools: There was an improvement for all items analyzed between baseline and mid-line measurements, with improvements ranging between 3% and 10% and the percentage of teachers reporting positively on these items at mid-line ranging from 91%-98%. o Teachers' perception of effect of technology on teaching and learning: All the teachers interviewed in TILO SBR Primary schools and 95% of the teachers in SBR Prep schools confirmed that technology had a positive effect on the teaching – learning process in their classes. Teachers reported that education was more interesting as they adopted active learning, creative thinking, and problem solving strategies to work with the technology. Teachers also reported that technology changed their role to facilitators rather than lecturers, and that students were more interested and attentive in school.  Sustainable use of resources of education technology in schools (2.1.2.3): a. Functioning maintenance system: o Effective management of technology: Measurements were conducted in SBR Primary and TSS Prep schools between baseline and M3. The percentage of schools that reported that they were “effectively managing technology” rose from 7% at the baseline to 19% at the M3 measurement. o Satisfaction with availability of resources and technical support: Measurements were conducted for SBR Primary schools at M4 and for SBR Prep schools between baseline and mid-line in April 2012.  SBR Primary schools: Teachers reported a high level of satisfaction with both technology resources and technical support, with 94% of teachers reporting satisfaction with resources and 89% reporting satisfaction with technical support. As a different tool was used at the baseline, it is not possible to show improvement over time.  SBR Prep schools: Teachers’ satisfaction with both technology resources and technical support increased from baseline to mid-line, with a 13% improvement for resources and a 26% improvement for technical support. Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 31  Sustainable use of resources of education technology in schools (2.1.2.3): b. Plan and budget for IT resources periodic updating: Measurements were conducted for SBR Primary and TSS Prep schools between baseline and M3. Measurements at M3 revealed the lack of a clear system for managing technology resources at the Muderiya and Idara level. This is an area that requires additional attention, in order to allow for more decentralized decision making about financial matters.  Schools introducing technology-based innovations (2.1.2.4): TILO provided hardware and software packages as well as 30 Interactive White Boards in selected schools. Through its PPPs, the project provided TILO schools with 1,507 Intel Classmates, as well as software applications and resources such as DCGEP videos, the IBM Reading Companion, Kidsmart and a software package from Orchtech. Tools and Instruments Used After the Year 2 measurement (M2 in April 2010), T7 (Idara and Muderiya Interview Protocol) was terminated due to the unavailability of high level management personnel at the Muderiya and Idara levels after the revolution. These personnel were the data sources for responding to TILO interviews (T7). In addition, T6 (TILO Technology Management Assessment Protocol) was withdrawn after the Year 3 measurement (M3 in April 2011) from the data collection tools used for monitoring the extension activities. As a result, SCOPE and the modified teacher survey T4++, were used in Year 4 (M4 in April 2012) to substitute for the tools which had been discontinued. Table 4 below shows the changes in data sources/tools that were made in order to acquire data to assess the five indicators relating to Impact Intended Result 2. Table 4: Impact Level Intended Result 2, Indicators, Data Sources, and Tools Indicators Sources of Data Tools 2.1.2.1 Data driven decisions at the Idara and school levels Project reports, school records, MOE reports, policies T6, Qs: 11-13 and T7, Q2. T7 was stopped after M2 and only T6 was used at M3. This indicator was not assessed at M4. 2.1.2.2 Effective use of technology for teaching and learning in schools Project reports, school records, MOE reports, policies T6 Qs: 1-4 and T7, Q1, 5. T7 was stopped after M2. T6 was stopped after M3. T4++ and SCOPE were used at M4. 2.1.2.3 Sustainable use of resources of education technology in schools: a. functioning maintenance system 2.1.2.3 Sustainable use of resources of education technology in schools: b. plan and budget for IT resources periodic updating Muderiya and Idara staff, school administration and teachers T6, Qs 5-10 and T7, Q 3 and 4. T7 was stopped after M2. T6 was stopped after M3. T4++ and SCOPE were used at M4. 2.1.2.4 Schools introducing technology￾based innovations TILO Database Document Review. Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 32 Detailed Findings 2.1.2.1 Data Driven Decisions at the Idara and School Levels (Impact PI 2.1) Idara and Muderiya levels: This indicator was originally planned to be examined at the Muderiya and Idara levels using the T7 tool. No conclusions could be drawn for this indicator at the Idara / Muderiya level as this tool had to be cancelled after the M2 measurement. Due to the restructuring of the MOE leaders at the Governorates during and after the January 2011 revolution, the needed data sources (MOE management at the Muderiya and Idara level) were not available for use in measuring this indicator. School level: Tables 5 and 6 below show improvement at the school level in using technology in the decision making process at SBR Primary and TSS Prep schools during the course of the TILO intervention. School Principals responded to questions on the tool and scores on a 5-point scale were assigned to these responses. Tables 5 and 6 looked at the percentage of schools that responded at each point of the scale at Baseline, M2 and M3 to provide information on any changes as a result of the TILO intervention. The tool differentiates between two levels of performance based on the scores: “Effectively Using Technology in the Decision Making Process at School” (Scores 3-5) and “NOT Effectively Using Technology in the Decision Making Process at School” (Scores 1-2). Table 5 shows the data for SBR Primary schools and Table 6 shows the data for TSS Prep schools. Table 5: Cumulative Evaluation Scores of SBR Primary Schools’ Use of Technology for Decision Making, T6, Qs 11-13 Scores 1-5 % # of Schools / Score BL M2 M3 5 3% 16% 16% 4 0% 34% 45% 3 3% 18% 18% 2 23% 21% 3% 1 72% 11% 18% Cumulative 3, 4, and 5 scores (Effectively Using Technology) 6% 50% 79% Table 6: Cumulative Evaluation Scores of TSS Prep Schools’ Use of Technology for Decision Making, T6, Qs 11-13 Scores 1-5 % # of Schools / Score BL M2 M3 5 0% 28% 28% 4 11% 39% 33% 3 17% 0% 28% 2 44% 0% 6% 1 28% 33% 6% Cumulative 3, 4, and 5 scores (Effectively Using Technology) 28% 67% 89% Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 33 Tables 5 and 6 show that the use of technology for decision making at schools progressed considerably over the years of the TILO intervention in both SBR Primary and TSS Prep schools. In SBR Primary schools, the percentage of schools that reported “effectively using technology” (Cumulative scores 3, 4, and 5) was 6% at the baseline but rose to 79% by M3 (a 73% increase). In TSS Prep schools, 28% of schools reported “effectively using technology” at the baseline, but this figure rose to 89% by M3, (a 61% increase). For indicator 2.1.2.1, the analysis of the data from the M3 measurement indicates that data driven decision making at both SBR Primary and TSS Prep schools progressed with TILO interventions, and that TILO has successfully made a positive change regarding this indicator at the school level. 2.1.2.2 Effective Use of Technology for Teaching and Learning in Schools (Impact PI 2.2) This indicator was examined for SBR Primary and SBR Prep schools by looking at four items:  TILO schools’ performance in integrating technology in teaching and learning  Teachers’ confidence and comfort using technology in education  Teachers' pedagogical competence comparison  Teachers' perception of effect of technology on teaching and learning This indicator was originally planned to be examined at the Muderiya levels using T7. However, T7 was discontinued after M2 (April 2010), as noted in the section above. Therefore, this indicator was not assessed at the Muderiya level for the end of project report. This indicator was originally planned to be examined at the Idara and school levels using T6. As T6 was discontinued after M3, substitute data from the T4++ and SCOPE tools were utilized to provide an assessment of this indicator at M4 for end of project reporting. SCOPE provided data on teachers’ progress in using technology for effective teaching (item 17) and on students’ learning behavior in engaging technology in their learning (item 23). Results are presented below for SBR Primary schools between baseline and M4, and for SBR Prep schools between baseline and the mid-line measurement in April 2012. No results are presented for TSS Prep schools as these schools were not measured after M3. TILO Schools’ Performance in Integrating Technology in Teaching and Learning The two sets of charts below (Charts 6A-C and Charts 7A-C) exhibit the improvement of SBR Primary schools’ and SBR Prep schools’ teachers and students in terms of their effective integration of technology in their teaching and learning in class. This was measured by the change in SCOPE scores on teachers’ item 17 and students’ item 23 on the SCOPE 5-point scale between the baseline measurements and the final measurement in April 2012. SBR Primary Schools: Chart 6A showed that the average score of SBR Primary school teachers from Cohort 1 schools rose by 29% along the SCOPE 5-point scale between the 2008-2012 measurements, going from 1.03 to 2.47, while the average score of students from the same cohort rose by 17% over the same period, moving from 1.01 to 1.88 on the same scale. Chart 6B showed that the average score of SBR Primary school teachers from Cohort 2 schools rose by 57% along the SCOPE 5-point scale between the 2009-2012 measurements, going from 1.03 to 3.88, while students from the same cohort rose by 45% over the same period, moving from 1.00 to 3.24 on the same scale. These comparisons are statistically significant at a 95% confidence interval. The performance of teachers and students from Cohort 2 showed greater improvement than those from Cohort 1, as shown in Chart 6C. This was consistent with the TILO team’s observation of Cohort 1 and 2’s performance overall. It is understandable that Cohort 2 performed better as the TILO team began implementation with Cohort 2 after they had gained one year’s experience with Cohort 1. They learned from their initial Cohort 1 implementation run what practices worked well or did not work and made adjustments from the start to Cohort 2. Also, the provision of technology and digital resources were more efficient and timely as any glitches in the system had been ironed out during the first year’s implementation with Cohort 1. This made training and all other intervention activities for Cohort 2 smoother and more effective. Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 34 Charts 6A-C: TILO SBR Schools’ Performance in Integrating Technology in Teaching and Learning as Measured by the SCOPE 5-point scale SBR Prep schools: Chart 7A-C represent the improvement of teachers and students from TILO SBR Prep schools in integrating technology in their teaching and learning processes. The measurements compare the baseline in October 2011 and at the mid-line measurement in April 2012. These charts show that the average score of teachers rose by 27% along the SCOPE 5-point scale over the one year of intervention, going from 1.1 to 2.5, while the average score of students rose by 16% over the same period, going from 1.1 to 1.9 on the same scale. Comparisons are statistically significant at a 95% confidence interval. 1.03 1.00 3.88 3.24 0.00 1.00 2.00 3.00 4.00 5.00 Teacher's Item 17 Students' Item23 Chart 6B: Cohort 2 Use of Technology for Teaching & Learning, SCOPE Teacher's Item 17 & Students' Item 23 2009-2012 BL2 M4 29% 57% 17% 45% 0% 20% 40% 60% 80% 100% Cohort 1 Change Cohort 2 Change Chart 6C: Performance Change in Using Technology Between BL and M4 for TILO Cohorts, SCOPE Teachers' Item 17 & Students' Item 23 Teacher's Item 17 Students' Item23 1.03 1.01 2.47 1.88 0.00 1.00 2.00 3.00 4.00 5.00 Teacher's Item 17 Students' Item23 Chart 6A: Cohort 1 Use of Technology for Teaching & Learning, SCOPE Teacher's Item 17 & Students' Item 23 2008-2012 BL1 M4 Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 35 Although the average teachers’ score did not reach point “3” on the SCOPE scale (which indicates the turning point for a shift to using reform-based practices), their performance was close to this point in a very short period of time (less than a year). It is expected that their performance continued to improve as teachers became more familiar with the practice. Charts 7A-C: TILO SBR Prep Schools’ Performance in Integrating Technology in Teaching and Learning as Measured by the SCOPE 5-point Scale 1.1 2.5 0.00 1.00 2.00 3.00 4.00 5.00 Baseline Mid Chart 7A: TILO SBR Prep Schools' Performance in Integrating Technology in Teaching, SCOPE Teacher's Item 17 Item 17 1.1 1.9 -1.00 1.00 3.00 5.00 Baseline Mid Chart 7B: TILO SBR Prep Schools' Performance in Integrating Technology in Learning, SCOPE Student's Item 23 Item 23 27% 16% -10% 10% 30% 50% Item 17 Item 23 Chart 7C: TILO SBR Prep Schools' Percentage Change in Teacher & Students' SCOPE Scores, Items 17 & 23, Baseline - Mid Percentage Improvement Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 36 Data acquired through T4++ as extracted from M4 Teacher's Survey reports (see graphs 9A-B) below), provided information that was also used to assess this indicator “Effective use of technology for teaching and learning in schools.” Teachers’ Confidence and Comfort Using Technology in Education Charts 8A and 8B below exhibit the change in SBR Primary schools’ and SBR Prep schools’ teachers’ confidence in using technology in education. SBR Primary schools: Per Chart 8A, teachers who reported “full” confidence in using technology decreased from 52% at baseline to 41% at M4 (an 11% decrease). However, those who reported being confident “to some extent” rose from 25% to 57% (a 32% increase) and those who reported “not having confidence” fell from 23% to 2% (a 21% decrease). These teachers had very little technology in their classrooms at the start of the TILO interventions, which explains the decrease of “full” confidence responses from the baseline to M4, when they were handling much more complicated technology. However, the overall confidence level increased over the interventions, as noted by the increase in confidence “to some extent” and by the fact that those “not having confidence” fell to 2% by M4. Teachers also reported that they could efficiently use and integrate technology in various forms, including equipment, software titles, commercial programs, pictures and films form the internet, specialized educational and curriculum subject matter related applications, internet based applications, and TILO-provided software titles like Crocodile and Geogebra. They reported feeling confident in changing their teaching methodology to be more modern and using new strategies in teaching, such as preparing lessons electronically. They used technology to stay up-to-date with their subject matter and felt more productive and more interested in their lessons when using technology. SBR Prep schools: Per Chart 8B, a comparison of baseline and mid-line data acquired through item 28 showed that the percentage of teachers who reported being “fully comfortable” using technology increased from 18% to 31% (a 13% increase), while those who reported “not comfortable” decreased from 15% to 5% (a 10% decrease), and those reporting “to some extent” remained at 64% on both measurements. 52% 25% 23% 41% 57% 2% 0% 20% 40% 60% 80% 100% Fully To Some Extent Not Chart 8A: SBR Primary School Teachers' Confidence, Comfort with Using Technology in Education BL M4 Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 37 The majority of teachers who reported “fully comfortable” indicated that they use computers very comfortably. Some teachers used PowerPoint to present lessons that they had prepared electronically, and many teachers reported using TILO educational software such as Crocodile and Geogebra. Teachers who reported “to some extent” expressed their concern about the time consumed in preparing and delivering lessons using technology and their inability to handle some of the software titles introduced by TILO e.g. Crocodile. Most of the teachers who reported “not comfortable” attributed this to their not receiving enough training on technology in general or on certain software packages that were introduced by TILO. Overall, the results in SBR Prep schools were very positive. Teachers' Pedagogical Competence Comparison Charts 9A and 9B exhibit the change in SBR Primary schools’ and SBR Prep schools’ teachers’ pedagogical competence based on examining the following five items:  Item 16: Student centered learning  Item 17: Classroom management  Item 18: Teaching towards critical thinking and problem solving skills  Item 19: Change management  Item 20: Team leading skills SBR Primary schools: Per Chart 9A, a comparison of baseline and M4 data acquired through items 16-20 showed an improvement on all items. The improvements in teachers’ pedagogical competence by item were as follows: Item 16: 16%, Item 17: 8%; Item 18: 14%; Item 19: 14%; and Item 20: 8%. These results are an encouraging sign for sustainability, especially in light of the fact that the M4 measurement was taken one year after the end of interventions in SBR Primary schools. 18% 64% 15% 31% 64% 5% 0% 20% 40% 60% 80% 100% Fully To Some Extent Not Chart 8B: SBR Prep School Teachers' Confidence, Comfort Using Technology in Education Pre Mid Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 38 SBR Prep schools: Data in Chart 9B indicate an improvement in teachers’ pedagogical competence as manifested by the number of teachers reporting positively on Items 16–20. The percentage rate of cumulative agreement showed an increase between baseline and mid-line along all five items as follows: Item 16: 10%, Item 17: 3%; Item 18: 6%; Item 19: 2%; and Item 20: 5%. The overall responses to this set of questions showed a high number of teachers who viewed themselves as having mastered the pedagogical skills as per the TILO model. Teachers' perception of effect of technology on teaching and learning All teachers in SBR Primary schools (148) and 95% of teachers in SBR Prep schools (190 of 201) who responded to this item confirmed that technology had a positive effect on the teaching -learning process in their classes. Teachers’ qualitative responses provided evidence of the positive change they observed in their classes, as well as the impact on the teacher and the students. Teachers reported that technology made education a more interesting process which adopted active learning, creative thinking, and problem solving strategies. This made classes more enjoyable for students, helped develop closer relationships between teachers and students, and moved the learning approach away 83% 91% 84% 82% 84% 99% 99% 98% 96% 92% 0% 20% 40% 60% 80% 100% Item 16 Item 17 Item 18 Item 19 Item 20 Chart 9A: SBR Primary Teachers' Pedagogical Competence, Cumulative Agreeing Comparison, Baseline - M4, Items 16 - 20 Baseline M4 88% 95% 90% 89% 88% 98% 98% 96% 91% 93% 0% 20% 40% 60% 80% 100% Item 16 Item 17 Item 18 Item 19 Item 20 Chart 9B: SBR Prep School Teachers' Pedagogical Competence, Cumulative Agreeing Comparison, Baseline - Mid, Items 16 - 20 Baseline Mid Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 39 from traditional memorization. Technology was reported to have changed the role of teachers to facilitators rather than lecturers, and to have given them more space to use creativity and innovation in teaching. On the part of students, teachers reported technology to have made students more interested in school, as well as more open minded, spirited, competitive, attentive and observant. From this series of analysis, it can be concluded that the majority of teachers in SBR Primary and SBR Prep schools expressed confidence in their mastery of technology for enhancing teaching and learning. Additionally, teachers’ confidence in using technology increased over the course of the interventions. 2.1.2.3 Sustainable Use of Resources of Education Technology in Schools (PI 2.3): 2.1.2.3.1 Functioning Maintenance System (PI 2.3 a) This indicator was examined by looking at two items:  Effective management of technology:  Satisfaction with availability of resources and technical support:  This indicator was examined through tool T6 until M3 and through T4++ at M4. Effective Management of Technology Table 6 below shows that, for SBR Primary and TSS Prep schools, limited progress took place along this indicator between baseline and M3. The percentage of schools’ principals reporting that they were effectively managing technology rose at M2 to 24% but dropped to 19% at M3. This pattern indicates that at M3 in April 2011, there was no real structured system for establishing sustainable use of technology resources at the schools assessed in the seven governorates. It should be noted that the M3 measurement was taken soon after the January 2011 revolution in Egypt, which may have contributed to the drop. Table 6: Cumulative Evaluation Scores of Schools’ Effectively Managing Technology, T6, Qs 5 and 6 – SBR Primary and TSS Prep Scores 1-5 % # of Schools / Score BL M2 M3 5 0% 4% 0% 4 0% 7% 3% 3 7% 13% 16% 2 12% 13% 34% 1 81% 63% 47% Cumulative 5, 4 and 3 scores 7% 24% 19% Satisfaction with Availability of Resources and Technical Support Charts 10 and 11 provide information on teachers’ responses about their satisfaction with the availability of resources and the technical support they received for their technology equipment. SBR Primary schools: Chart 10 indicates that, at M4, 94% of the participating teachers were satisfied with the availability of electronic resources for teaching at their schools and 89% expressed satisfaction with the availability of technical support for technology use. The tool used at the baseline was different from the T4++ tool used at M4, and therefore it is not possible to show improvement over time. However, it is a positive indication for sustainability that such a high percentage of teachers Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 40 reported satisfaction with both resources and technical support at M4, one year after the end of TILO interventions in the schools. SBR Prep schools: The findings in Chart 11were extracted from the M4 T4++ tool. A comparison of the baseline to mid-line results shows a clear improvement over time in the percentage of teachers reporting satisfaction with resources and technical support. These percentages rose from 61%-74% for Item 14, Resources, (a 13% increase) and from 49%-75% for Item 15, Technical support, (a 26% increase). The qualitative data from the T4++ report showed that teachers reported high ratios of providing their own resources and in getting their students to participate in enriching their resource library. All these results suggest that TILO’s work at these schools has positively affected the level of services provided by schools to their teachers in terms of e-resources for the teaching and learning process and in terms of providing adequate technical support for the use of technology in this process. 94% 89% 6% 11% 0% 20% 40% 60% 80% 100% Resources Technical Support Chart 10: SBR Primary, Satisfaction With Availability of Resources & Technical Support, Cumulative Agreeing - Disagreeing, M4, Items 14 & 15 Agreeing Disagreeing 61% 49% 74% 75% 0% 20% 40% 60% 80% 100% Item 14 Item 15 Chart 11: SBR Prep, Satisfaction With Availability of Resources & Technical Support, Cumulative Agreeing Comparison Baseline-M4, Items 14 & 15 Agreeing Baseline Agreeing Mid Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 41 2.1.2.3.2 Plan and Budget for IT Resources Periodic Updating (PI 2.3 b) This indicator was examined at the school level using T6 (Q5) between baseline and M3 for SBR Primary and TSS Prep schools. The responses of Principals at M3 indicated some uncertainty over roles and responsibilities regarding financial matters, including those related to the maintenance of technology resources. School principals essentially depended upon resources provided centrally by the TDC office of the MOE. Through the M3 measurement, the data for this indicator revealed a lack of and a need for a clear, sustainable decentralized system for managing technology resources at the schools. It is recommended that the MoE, Muderiya and Idara management make the necessary changes to allow for more decentralized decision making about financial matters, especially in regards to technology resources. 2.1.2.4 Schools introducing technology-based innovations (Impact PI 2.4) TILO succeeded in introducing technology innovations into schools through its various activities. A number of innovations point towards TILO’s satisfying this indicator, including the introduction of 30 Interactive White Boards in selected schools as a pilot initiative that can be developed and expanded further. Many innovations were introduced into schools through the second component of the TILO intervention: Public Private Partnerships. Through its PPPs, TILO succeeded in bringing to its schools new innovations in education technology. Such innovations included, but were not limited to, providing TILO schools with 1,507 Intel Classmates. In addition to hardware, software applications and resources such as the DCGEP videos, IBM Reading Companion, Kidsmart and the Orchtech software package also contributed greatly towards making learning at TILO schools more fun and attractive to students, easier to facilitate, and more effective, as reported by teachers in their testimonies. Some of these tools (such as the DCGEP videos, Orchtech package) were used by teachers to better explain science and math topics to students. Other tools (such as the Intel Classmates, IBM Reading Companion and Kidsmart) were used by the students to practice skills they learned in the classroom. 2.2 Outcome Level Results The TILO PMP identified five intended results at the outcome level that mark the progress and impact of the intervention at the medium term. These five results were assessed by looking at 11 sub-results that are described under each Outcome Intended Result in the sections below. The five intended results are:  Increased student engagement with IT;  Changed classroom practice (Teachers);  Increased local support for IT in schools;  Enhanced school management of IT; and  Enhanced MOE capacity for management of education technology resources. Outcome Intended Result 1: Increased Student Engagement with IT This outcome level intended result was assessed by examining two sub-results: 1: Students engaged in active learning powered by technology 2: Students develop positive attitudes towards technology These two sub-results were assessed by examining the three indicators and using the tools and questions shown in Table 7 below. See PMP, Appendix 1, for details. Certain changes were made in the structure of measurements and the tools used over the course of the project. The T5 tool used for Sub-Result 1 was discontinued after the Year 2 measurement (M2 in April 2010) and the SCOPE tool Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 42 was substituted through M4. Therefore, results for this Sub-Result are not provided for TSS Prep schools as they were not included in the M4 measurement. Similarly, the T3 tool used for Sub-Result 2 was discontinued after the Year 3 measurement (M3 in April 2011), and therefore results are not provided for SBR Prep schools (which were only measured in 2012-2013). Summary of Findings Sub-Result 1: Students engaged in active learning powered by technology  Level of technology integration in student learning activities (2.2.1.1): o SBR Primary schools: The extent to which students integrated technology into their learning increased by 17% in Cohort 1 schools and 45% in Cohort 2 schools between the baselines and M4 measurements. o SBR Prep schools: The extent to which students integrated technology into their learning increased by 16% from the baseline to the mid-line measurements. Sub-Result 2: Students develop positive attitudes towards technology  No. of students with personal email addresses (2.2.2.1): In SBR Primary and TSS Prep schools, the percentage of students who used the internet increased by 33% (from 51% to 84%) from the baseline to the M3 measurement. Additionally, the percentage of students who owned email accounts increased by 15% (from 27% to 42%) from the baseline to M3 measurement.  Students’ perceptions towards use of technology in learning (2.2.2.2): In SBR Primary and TSS Prep schools, almost all students commented positively about the hardware and digital resources provided by TILO. There was also an increase in the percentage of students who used technology equipment between the baseline and M3 measurements for all the types of technology assessed (computers, digital cameras, video/television, and cassette players). Tools and Instruments Used Table 7 shows the changes in data sources/tools that were made in order to acquire data to assess the three indicators pertaining to the two sub-results for Outcome Intended Result 1. Table 7: Outcome Level Intended Result 1, Indicators, Data Sources, and Tools Sub-Results Indicators Sources of Data Tools 2.2.1. Students engaged in active learning powered by technology 2.2.1.1 Level of technology integration in student learning activities Video tapes of TILO sample classes T5 T5 discontinued after M2. Substitute evidence obtained from SCOPE at M4. 2.2.2. Students develop positive attitudes towards technology 2.2.2.1 No. of students with personal email addresses Students T3, Qs (10 – 13) 2.2.2.2 Students’ perceptions towards use of technology in learning Students T3 Qs (1 – 5) Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 43 Detailed Findings 2.2.1 Students Engaged in Active Learning Powered by Technology 2.2.1.1 Level of Technology Integration in Student Learning Activities (Outcomes PI 1.1) After the T5 tool was discontinued at M2, this indicator was assessed through the SCOPE tool at the M4 measurement in April 2012. In particular, Item 23 (“utilize technology tools in ways that help them to manipulate knowledge, explore, and practice”) was used to assess change in students’ performance in terms of efficiently integrating technology into their learning activities. The results from this Item showed that students’ learning behavior in terms of integrating technology into their learning in TILO classrooms improved in SBR Primary schools by 17% between the baseline and M4 measurements in Cohort 1 governorates and by 45% in Cohort 2 governorates. This is consistent with other results that also showed a greater improvement in Cohort 2 schools than in Cohort 1 schools. TILO SBR Prep school students also showed improvement over their one-year intervention period, with a 16% increase between baseline and mid-line measurements. These findings provided evidence that TILO has successfully satisfied this indicator “Level of technology integration in student learning activities” which contributes towards attaining the Outcome Level Intended Result 1, “Increased Student Engagement with IT”. 2.2.2 Students Develop Positive Attitudes Towards Technology 2.2.2.1 Number of Students with Personal Email Addresses (Outcomes PI2.1) Questions from the T3 tool were used to examine this indicator until M3 in April 2011 for SBR Primary and TSS Prep schools. Table 8 shows responses to questions 9 and 10 from T3 data. While the title of the indicator refers specifically to students with personal email addresses, this indicator was also designed to capture students’ use of the internet in general as well. Question 9 examined the percentage of students who used the internet in TILO SBR Primary and TSS schools. This figure increased by 33% between baseline and M3, going from 51% to 84%. Question 10 examined the percentage of students who specifically owned email accounts. This figure increased by 15% between baseline and M3, going from 27% to 42%. This finding indicated two changes in students’ patterns of learning behavior regarding using technology. Firstly, their use of the internet increased over the years of TILO implementation, primarily as a result of their needing to master the internet to satisfy the learning requirements at their schools. Secondly, students did increasingly use emails for communication, although this change was not as pronounced as their use of internet. This could be due to a number of factors, such as their preference for using other social media for communication such as Facebook and Twitter. Overall, the findings for this indicator suggest that TILO has made progress toward satisfying this outcome indicator PI2.1. 2.2.2.2 Students’ Perceptions of the Use of Technology in Learning (Outcomes PI2.2) Questions 1-4 from the T3 tool were used to examine this indicator between baseline and M3 for SBR Primary and TSS Prep schools. Table 8: T3 Qs 9 and 10 responses “YES” Q # Measurement BL M3 Change Q9 51% 84% 33% Q10 27% 42% 15% Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 44 Question 1 asked students what they thought about using technology in the classroom. Information extracted from T3 reports and data on this item confirmed that students established positive perceptions and attitudes towards technology in learning based on their experiences with TILO. Almost all students consistently commented positively about the hardware and digital resources provided by TILO and about the way their teachers were utilizing technology differently to make them learn better and feel freer to work independently and in teams. They commented that technology helped them be part of the Egyptian revolution and feel the value of their citizenship by participating in communication about the revolution and giving their opinions freely. Question 2 asked about students’ use of different technologies to assist their learning. This item also showed progress in terms of students’ use of technology for learning. Table 9 shows the results of information extracted from T3 data and reports. These results demonstrated an increase in the percentage of students who used technology equipment between baseline and M3 measurements in SBR Primary and TSS schools. Students reported an increase in the use of all the types of technology, with the greatest increase relating to digital cameras. TILO had specifically introduced the use of digital cameras in schools and this improvement is an indication of TILO’s effect in the schools. Questions 3 and 4 asked students if they owned technology equipment as personal property and for what purposes. The results for these items indicated that students owned technology equipment for a variety of purposes, with a particular emphasis on using them for educational purposes. A number of students were explicit about using cameras for projects and using computers for accessing TILO digital resources as well as for communicating with family and friends abroad. Most of the reasons reported for not owning technology equipment were related to economic or cultural reasons, where parents agreed with the need for the technology but had deferred acquiring the equipment to a later date. Conclusion: Overall, the findings indicate that TILO has changed students’ attitudes towards technology and made them more positive about their need for and the benefit of using technology for learning and other purposes. All these findings indicate that TILO has made progress towards attaining Outcome Level Intended Result 1, “Increased Student Engagement with IT”. Outcome Intended Result 2: Changed Classroom Practice (Teachers) This outcome level intended result was assessed by examining four sub-results: 3: Teachers integrating technology into student-centered methods 4: Teachers acquire positive attitudes to integrating technology into instruction 5: School leadership supporting teachers’ use of technology 6: Digital content used in the classroom These four sub-results were assessed by examining the six indicators and using the tools and questions shown in Table 10 below. See PMP, Appendix 1, for details. Certain changes were made in the structure of measurements and the tools used over the course of the project. Certain tools were discontinued after M2 and M3 and the SCOPE and T4++ tools were used for the M4 (April 2012) measurements. As a result, for certain indicators, the information at M4 is presented, but since the baseline data were collected using a different tool, it is not possible to show improvement over time. Table 9: %/# Students Responding “YES” to T3 Q2 Name of Technology BL M3 Change Computer 70% 77% 7% Digital Camera 17% 64% 47% Video/television 38% 46% 8% Cassette Player 57% 94% 37% Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 45 For other indicators, as results were compared between baselines and the M4 measurements, results for TSS Prep schools are not provided as they were not included in the M4 measurement. Summary of Findings Sub-Result 3: Teachers integrating technology into student-centered methods  Percent increase in effective classroom teaching practices as measured by SCOPE (2.2.3.1) o SBR Primary schools: Teachers from Cohort 1 schools improved by 27% on the SCOPE scale between baseline and M4, and students improved by 16%. Teachers from Cohort 2 schools improved by 45% between baseline and M4, and students improved by 34%. Comparisons are statistically significant at a 95% confidence interval. o SBR Prep schools: Teachers in SBR Prep schools improved by 32% on the SCOPE scale between the baseline and mid-line measurement and students improved by 22%. Comparisons are statistically significant at a 95% confidence interval.  Level of IT integration in teaching materials (2.2.3.2): This indicator was measured based on teachers’ effective use of technology for teaching and learning in schools. o SBR Primary schools: Teachers from Cohort 1 schools improved by 29% on the SCOPE scale between baseline and M4, and teachers from Cohort 2 schools improved by 57%. Comparisons are statistically significant at a 95% confidence interval. o SBR Prep schools: Teachers in SBR Prep schools improved by 27% on the SCOPE scale between the baseline and mid-line measurement. Comparisons are statistically significant at a 95% confidence interval. Sub-Result 4: Teachers acquire positive attitudes to integrating technology into instruction  Level of confidence of teachers in using IT in teaching (2.2.4.1) o SBR Primary schools: Teachers’ confidence level rose between baseline and M4 for all types of equipment, as follows: computers: 14%; cameras: 29%; overhead projectors: 27%; interactive white boards: 5%, and videos: 30%. o Prep schools: Teacher’s confidence level rose for all types of equipment between baseline and the mid-line measurements except for interactive white boards, which dropped slightly by 2%. However, teachers’ confidence in the other types of equipment rose as follows: computers: 14%; cameras: 8%; overhead projectors: 9%; and videos: 12%.  No. of teachers with personal email addresses (2.2.4.2): For SBR Primary schools, only 29% of teachers owned email accounts. For SBR Prep schools, the percentage of teachers owning email accounts dropped 3% between baseline and mid-line. Sub-Result 5: School leadership supporting teachers’ use of technology  Level of satisfaction of teachers of administrators' technology support (2.2.5.1): o SBR Primary schools: At the M4 measurement, 69% of teachers reported satisfaction with the support they received from their school management, and 52% reported satisfaction with support from Idara supervisors. o SBR Prep schools: Teachers’ satisfaction with support from school management rose by 14% and satisfaction with support from Idara supervisors rose by 25%. Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 46 Sub-Result 6: Digital content used in the classroom  Range of digital material / resources used by teachers and students (2.2.6.1): Results for this indicator are provided for SBR Primary and SBR Prep schools together. Information yielded at the M4 measurement indicated that almost all teachers understood the resources and technical they received for using technology in their classes. Apart from using what was provided, teachers also reported contributing to the e-resources in their classes by downloading pictures, e-learning materials, and video tapes. Tools and Instruments Used Table 10 shows the changes in data sources/tools that were made in order to acquire data to assess the six indicators pertaining to the four sub-results for Outcome Intended Result 2. Table 10: Outcome Level Intended Result 2, Indicators, Data Sources, and Tools Sub-Results Indicators Sources of Data Tools and Questions 2.2.3 Teachers integrating technology into student￾centered methods 2.2.3.1 Percent increase in effective classroom teaching practices as measured by SCOPE SCOPE results Administration of SCOPE Using M4 SCOPE 2.2.3.2 Level of IT integration in teaching materials Teacher portfolios Teacher Products Rubric, T1 T1 discontinued after M2. Substitute data source from SCOPE results on Item 17. 2.2.4 Teachers acquire positive attitudes to integrating technology into instruction 2.2.4.1 Level of confidence of teachers in using IT in teaching Self￾assessment data Questionnaire and Interviews, T2 and T4 T2 discontinued after M3. T2 and T4 folded into T4++ at M4 2.2.4.2 No. of teachers with personal email addresses Self￾assessment data Questionnaire, T4 T4 folded into T4++ at M4 2.2.5. School leadership supporting teachers’ use of technology 2.2.5.1 Level of satisfaction of teachers of administrators' technology support Satisfaction data from teachers Questionnaire and Interviews, T2 and T4 T2 discontinued after M3. T2 and T4 folded into T4++ at M4 2.2.6. Digital content used in the classroom 2.2.6.1 Range of digital material / resources used by teachers and students Teachers and students Questionnaire and Interview, T3 and T4 T3 stopped at M3 T4 folded into T4++ at M4 Detailed Findings 2.2.3 Teachers Integrating Technology into Student-Centered Methods 2.2.3.1 Percent Increase in Effective Classroom Teaching Practices as Measured by SCOPE (Outcome PI 3.1) Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 47 This indicator was assessed by monitoring SCOPE scores for TILO teachers. SCOPE results demonstrated a trend for consistent improvement between baseline and M4 for Primary SBR schools and between the baseline and mid-line measurements for the SBR Prep schools, as demonstrated by Charts 12 – 14 below. SBR Primary schools: Chart 12 showed that participants from SBR Primary schools from Cohort 1 governorates improved on the teachers’ items (items 1-17) by 27% on the SCOPE 5-point scale, going from 1.31 to 2.86. On the students’ items (items 18-23), participants improved by 16%, going from 1.03 to 1.83. Chart 13 showed that participants from SBR primary schools from Cohort 2 governorates improved on the teachers’ items (items 1-17) by 45% on the SCOPE 5-point scale, going from 1.40 to 3.63. On the students’ items (items 18-23), participants improved by 34%, going from 1.08 to 2.78. Comparisons are statistically significant at a 95% confidence interval. SBR Prep schools: Chart 14 showed that participants from SBR Prep schools also improved between the baseline and mid-line measurements. Teachers improved by 32% on items 1-17 the SCOPE 5- point scale, going from 1.40 to 3.0. Students improved by 22% on items 18-23, going from 1.10 to 2.20. Comparisons are statistically significant at a 95% confidence interval. 1.31 1.57 2.02 2.71 2.68 1.03 1.11 1.33 1.77 1.83 0.00 1.00 2.00 3.00 4.00 5.00 BL M1 M2 M3 M4 Chart 12 SBR Primary Cohort 1: Average Perfromance on Teachers' Items 1-17 & Students' Items 18-23, Baseline - M4 Avrg 1-17 Avrg 18-23 Linear (Avrg 1-17) Linear (Avrg 18-23) 1.40 2.72 3.15 3.63 1.08 1.81 2.35 2.78 0.00 1.00 2.00 3.00 4.00 5.00 BL M2 M3 M4 Chart 13 SBR Primary Cohort 2: Average Performance on Teachers' Items 1-17 & Students' Items 18-23, Baseline - M4 Avrg 1-17 Avrg 18-23 Linear (Avrg 1-17) Linear (Avrg 18-23) Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 48 TILO SBR Primary and SBR Prep school teachers’ and students’ performance on the SCOPE measurements, and the improvement showed between the baseline and the final run in April 2012, provided strong evidence that TILO has successfully satisfied the indicator and significantly contributed towards attaining the Outcome Level Intended Result 2, “Changed Classroom Practice.” 2.2.3.2 Level of IT Integration in Teaching Materials (Outcome PI 3.2) This indicator was meant to be examined by evaluating teachers’ portfolio using the TILO tool T1, Teacher Product Rubric. However, as T1 was discontinued after the first year (as explained in the methodology section), SCOPE results for item 17 at M4 were used as a substitute source of information to examine this indicator. The data on Charts 6A-C in section 2.1.2.2 (Effective Use of Technology for Teaching and Learning in Schools) revealed that teachers’ performance on item 17 improved over the course of the project. For SBR Primary schools, Cohort 1 teachers’ performance improved between baseline and M4 by 29% on the SCOPE 5-point scale, and Cohort 2 teachers improved by 57%. The data on Charts 7A-B shows that in SBR Prep schools, teachers’ performance on item 17 improved by 27% in just one year. These increases in the average scores for teachers demonstrate that TILO has successfully satisfied this performance indicator. 2.2.4 Teachers Acquire Positive Attitudes Towards Integrating Technology Into Instruction 2.2.4.1 Level of Confidence of Teachers in Using IT in Teaching (Outcome PI 4.1) This indicator was originally planned to be examined using TILO tools T2 and T4. T2 and T4 were discontinued after M3 and the T4++ tool was used at M4. For this report, this indicator is examined based on responses of teachers to a set of five items (1-5) on the T4++ tool. Responses “agreeing” to any of these items indicated confidence in using technology in teaching using one of five types of technology equipment: computers, digital cameras, overhead projectors (OHPs), interactive white boards (IWB) and video extracts on tapes or DVDs, which were used in TILO schools for enhancing teaching and learning in class. SBR Primary schools: Chart 15 below for SBR Primary schools’ teachers’ responses shows that teachers’ confidence level rose between baseline and M4 for all the types of equipment. Teachers’ confidence for using the various types of equipment improved as follows: computers: 14%; cameras: 29%; overhead projectors: 27%; interactive white boards: 5%, and videos: 30%. 1.4 1.1 3.0 2.2 0.00 2.00 4.00 Average 1-17 Average 18-23 Chart 14 SBR Prep Schools: Average Performance on Teachers' Items 1- 17 and Students' Items 18-23, Baseline - Mid Baseline Mid Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 49 SBR Prep schools: Chart 16 below for SBR Prep schools’ teachers’ responses between the baseline and mid-line measurements shows a slight increase in confidence for using all types of equipment except for IWBs, which dropped slightly, by 2%. However, teachers’ confidence in the other types of equipment rose as follows: computers: 14%; cameras: 8%; overhead projectors: 9%; and videos: 12%. The findings from Charts 15 and 16, Charts 8A and B in section 2.1.2.2, and the qualitative information gathered in the T4++ report at M4 all indicate that teachers’ confidence in using technology for teaching was maintained at high levels one year after concluding TILO’s activities at SBR Primary schools. This is a clear indicator of sustainability of TILO classroom practices adopted 84% 48% 58% 35% 58% 98% 77% 85% 40% 88% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Computer Camera OHP IWB Video Chart 15: Confidence in Using Technology in Classroom, T4 and T4++ Items 1-5, Cumulative "Agreeing" between BL & M4 Agreeing @ BL Agreeing @ M4 74% 19% 55% 28% 44% 88% 27% 64% 26% 56% 0% 20% 40% 60% 80% 100% Computer Camera OHP IWB Video Chart 16: Confidence in Using Technology in Classroom, T4 and T4++ Items 1-5, Cumulative "Agreeing" between Baseline and Mid Agreeing Baseline Agreeing Mid Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 50 at these schools. The findings for SBR Prep schools showed that teachers, who started with a relatively high level of confidence in using technology, gained more confidence over the one year of TILO intervention. A possible reason for the lower responses for IWBs at both SBR Primary and Prep schools is because TILO only provided 30 IWBs to SBR primary schools and none to Prep schools. The MOE provided some additional IWBs to other schools but TILO could not control the level of training provided by the MOE to schools on their use. In summary, based on the findings presented above, it can be stated that TILO has successfully satisfied this outcomes level indicator towards achieving the Outcome Level Intended Result 2, “Changed Classroom Practice”. 2.2.4.2 Number of Teachers with Personal Email Addresses (Outcome PI 4.2) This indicator was assessed using the T4 tool until M3, after which the T4++ tool was used at M4. Chart 17 provides data extracted from the T4++ report which addresses teachers’ attitudes towards using technology in their teaching in terms of owning a personal email address. The data showed that in SBR Primary schools, owning an email account was not particularly popular among the majority of teachers, with only 29% of teachers owning email accounts. Chart 18 shows that for Prep schools, the percentage of teachers was higher than in SBR Primary schools. However, this percentage decreased by 3% from the baseline to mid-line measurement, going from 46% to 43%. This lack of interest in owning email accounts does not conform to the positive attitudes towards technology expressed by teachers in all other aspects assessed within this study. It is possible that 29% 71% Chart 17: SBR Primary: Attitude - Teachers Own Email Accounts, T4++, Item 21, M4 YES NO 46% 54% 43% 57% 0% 20% 40% 60% 80% 100% Yes No Chart 18: SBR Prep: Attitude - Teachers Own Email Accounts, T4++ Item 21, Comparison Baseline - Mid Baseline Mid Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 51 teachers found other social media tools such as Facebook, Skype, and Twitter preferable, which could have contributed to their not having a great need for email. 2.2.5 School leadership supporting teachers’ use of technology 2.2.5.1 Level of Satisfaction of Teachers of Administrators' Technology Support (Outcome PI 5.1) This indicator was originally planned to be examined using TILO tools T2 and T4. T2 and T4 were discontinued after M3 and T4++ was used at M4. This indictor was examined through T4++, looking at items addressing teachers’ satisfaction with management support for integrating technology in teaching. SBR Primary schools: Chart 19 shows that the majority of teachers in SBR Primary schools were satisfied with the level of support they were receiving from their school management and from Idara Supervisors for using technology to teach. Their satisfaction for support received from school management (69% yes) was higher than from Idara supervisors (52% yes). This was natural, given that, within the TILO support scheme, the school management was fully engaged in all details of the intervention activities and were greatly involved in assisting their teachers through the activities. SBR Prep schools: As shown by Charts 20 and 21, there was an increase in teachers’ satisfaction with support received from both their school management and from Idara Supervisors between the baseline and mid-line measurements. Teachers’ satisfaction with support from school management rose by 14% (from 27% - 41%) and satisfaction with support from Idara supervisors rose by 25% (from 20% - 45%). 69% 28% 3% 52% 38% 10% 0% 50% 100% Yes To Some Extent No Chart 19: SBR Primary, T4++ Items 34 & 35 - Satisfaction, Support from School Management & Idara Supervisors School Management Idara Supervisors Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 52 As seen by Charts 19-21, TILO teachers in both SBR Primary and Prep schools were overall highly satisfied with the support they were receiving from both their school management and from Idara Supervisors. In addition, testimonies of teachers on T4++ items also provided evidence of such improvement. Teachers noted that, over the course of the interventions, both the schools’ management and Idaras showed a greater understanding of the TILO approach and provided both moral and material support to teachers and students in implementing the TILO model. Many teachers attributed this improvement to a great extent to the fact that TILO included the management and the Idara Supervisors in their trainings. Based on these results, it can be claimed that TILO has successfully satisfied this indicator towards achieving the intended Outcome Level Intended Result 2, “Changed Classroom Practice”. 2.2.6 Digital Content Used in the Classroom 2.2.6.1 Range of Digital Material/Resources Used by Teachers and Students (Outcome PI 6.1) This indicator was planned to be examined using TILO tools T3 and T4. These tools were discontinued after M3 and T4++ was used at M4. This indictor was examined through T4++, looking at the items addressing teachers’ satisfaction with the resources they received for integrating technology in teaching. Results for this indicator are provided for SBR Primary and SBR Prep schools together. The information yielded through T4++ indicated that almost all teachers were clear about the resources and technical support they received for their use of technology in their classes and were satisfied with it. Teachers reported on a variety of digital resources provided by TILO, including the 27% 46% 27% 41% 51% 8% 0% 50% 100% Yes To Some Extent No Chart 20: SBR Prep, T4++ Item 34 - Teachers' Satisfaction, Support from School Management, Baseline - Mid School Management Baseline School Management Mid 20% 44% 36% 45% 38% 17% 0% 50% 100% Yes To Some Extent No Chart 21: SBR Prep, T4++ Item 35 - Teachers' Satisfaction, Support from Idara Supervisors, Baseline - Mid Idara Supervisors Baseline Idara Supervisors Mid Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 53 Discovery package and KidSmart programs. Some of these included: Crocodile for Physics and Chemistry, GeoGebra, Tux Math, Tux Paint, Google Earth, KidSmart, Word Sight Boddy, Movie Maker, Science lab Program, Yanka, a variety of video tapes and CDs, Audesy, Admin 101 for lesson preparation, Lingos, and Net Sport. Teachers also reported that they contributed to the provision of e-resources for their classes by downloading pictures, e-learning materials for different subjects, video tapes relating to religion, history and other subjects. They downloaded testing materials for all subjects, videos, and research work materials from the internet. They also downloaded and used commercial software titles such as hissab.com for Math and Office applications. The MOE was reported to have provided CDs or downloadable subject matter content, and e-resource for all subjects from the MOE portal and eLearning sites. Overall, TILO has successfully satisfied this indicator. Conclusion: Based on the findings for indicators 2.2.3.1 through 2.2.6.1, TILO has succeeded in achieving the Outcome Level Intended Result 2, “Changed Classroom Practice”. All indicators which contributed to attaining this result were met. Outcome Intended Result 3: Increased Local Support for IT in Schools This outcome level intended result was assessed by examining three sub-results: 7: Parents support use of computers in teaching. 8: Opportunities for parents and community members to use technology 9: Public-Private-Partnerships increase resources and support of technology use in schools beyond TILO’s provision These sub-results were assessed by examining the four indicators and using the tools and questions shown in Table 11 below. Summary of Findings Sub-Result 7: Parents support use of computers in teaching  Level of generation of funds/resources for IT by BOTs in TILO schools (2.2.7.1): The majority of schools assessed could not acquire funds directly from the Boards of Trustees (BOTs) as the system did not allow for this. Some contributions were made by BoTs to support IT in schools. However, there was no evidence of schools having financial plans to support the maintenance of technology in their schools. Sub-Result 8: Opportunities for parents and community members to use technology  Degree of utilization of school IT resources by community (2.2.8.1): Less than 20% of TILO schools were able to successfully make their IT resources available for the community, as the concept of sharing school resources in this manner was new to both schools and the communities. Sub-Result 9: Public-Private-Partnerships increase resources and support of technology use in schools beyond TILO’s provision  Monetary ($) value of PPP interventions (2.2.9.1)  Range of products and services provided by PPPs (2.2.9.2) Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 54 Over the lifetime of the project, TILO acquired 13 PPP partnerships with international and Egyptian organizations working in Egypt. The monetary value of these activities by the end of the project in September 2013 was $9,472,985. TILO PPP organizations included International corporates and organizations such as Intel, Microsoft, Oracle, IBM, the Discovery Channel, and Coca Cola Africa, Orchtech, Houghton Mifflin Harcourt, HP, RWE, HSBC, Exon Mobil, and national organizations like Yousef El-Genade, and Winning Egypt. The range of products/services provided included training, software titles, hardware equipment, educational video clips, land, and professional development scholarships. Tools and Instruments Used Table 11 shows the data sources/tools that were used in order to acquire data to assess the four indicators pertaining to the three sub-results for Outcome Level Intended Result 3. Table 11: Outcome Level Intended Result 3, Indicators, Sources of Data and Tools Sub-Results Indicators Sources of Data Tools 2.2.7 Parents support use of computers in teaching. 2.2.7.1 Level of generation of funds/resources for IT by BOTs in TILO schools School records/visits Document review, interviews, T6 (Qs 14 and 16) Measured until M3 2.2.8 Opportunities for parents and community members to use technology 2.2.8.1 Degree of utilization of school IT resources by community School records/visits Document review - interviews, T6 (Q15) Measured until M3 2.2.9 Public-Private￾Partnerships increase resources and support of technology use in schools beyond TILO’s provision 2.2.9.1 Monetary ($) value of PPP interventions Project reports Document review 2.2.9.2 Range of products and services provided by PPPs Project reports Document review Detailed Findings 2.2.7 Parents Support Use of Computers in Teaching 2.2.7.1 Level of Generation of Funds/Resources for IT by BOTs in TILO Schools (Outcome PI 7.1) This indicator was assessed through information acquired through T6 data and reports at M3. This information revealed that the majority of schools could not become involved in acquiring direct funds from/through the school Boards of Trustees (BoTs). The system did not allow school managements such freedom. Nevertheless, through efforts at various schools, some contributions were made by BoTs in the form of technology equipment (e.g. copy machines, computers, printers, paper, ink, etc.) or technology room construction materials (e.g. security iron bars on windows, computer lab and technology rooms). A few schools’ BoTs raised limited funds for technology use that ranged between L.E. 400 to L.E. 6,000. There was no evidence of schools having their own financial plans for supporting the use of technology in their schools in terms of maintenance and spare parts. The main source of provision of funds for this need was the TDC office at the Muderiya level. Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 55 Indeed, working with BoTs was not initially part of TILO’s scope until the inclusion of the DCGEP partnership with a focus on community outreach. In the TILO extension, community outreach was included as a component for SBR Primary schools. Despite the lack of improvement in this particular indicator, the TILO project (especially through its PPP with DCGEP) was able to successfully establish a base for community outreach and set extraordinary examples of success in building trust within communities and garnering their support for school projects. This is further explained in the upcoming sections. This indicator was not fully satisfied in TILO schools in terms of contributing towards achieving Outcome Level Intended Result 3, “Increased Local Support”. 2.2.8 Opportunities for Parents and Community Members to Use Technology 2.2.8.1 Degree of Utilization of School IT Resources by Community (Outcome PI 8.1) This indicator was assessed through information acquired from T6 data and reports at M3. The information revealed that a limited number of schools (less than 20%) were able to successfully make their IT resources available for the community. Indeed, the concept of sharing school resources with the larger community seemed to be new to both the communities and the schools. This was largely due to the fact that teachers and school managers in the Egyptian educational system are not familiar or comfortable with the concept of sharing resources with the community. It is recommended that schools be encouraged to utilize their newly introduced TILO resources for the benefit of the community within the MoE scheme of the “Productive School” initiative. Training in this regard is recommended for future projects. This indicator was not fully satisfied in TILO schools in terms of contributing towards achieving Outcome Level Intended Result 3, “Increased Local Support”. 2.2.9 Public-Private-Partnerships increase resources and support of technology use in schools beyond TILO’s provision 2.2.9.1 Monetary ($) value of PPP interventions (Outcome PI 9.1) and 2.2.9.2 Range of products and services provided by PPPs (Outcome PI 9.2) Analysis of data for assessing these two indicators compared the performance of the project at the first year’s measurement M1 in 2009 and at the end of project as reported through the TILO reporting system (e.g. quarterly and annual reports). Over the lifetime of the project, TILO acquired 13 PPP partnerships with international and Egyptian organizations working in Egypt. Many of these contributed to TILO PPP activities more than once. The monetary value of these activities rose from $1,603,930 reported at M1 in 2009 to $9,472,985 at the end of the project in September 2013. As noted, the PPPs enabled the leveraging of additional resources that expanded and supported project activities. The value of resources leveraged amounted to almost 30% of the project’s ceiling. TILO PPP organizations included international corporates and organizations such as Intel, Microsoft, Oracle, IBM, the Discovery Channel, and Coca Cola Africa, Orchtech, Houghton Mifflin Harcourt, HP, RWE, HSBC, ExXon Mobil, and national organizations like Yousef El-Genade and Winning Egypt. The range of products/services provided by these organizations to TILO schools included training, software titles, hardware equipment, educational video clips, pieces of land, and professional development scholarships. Table 12 below provides more detail about the TILO PPP partnerships and the monetary value for each of them. Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 56 Table 12: PPP Interventions’ Monetary Value and Range of Products Provided Organization Basis of Partnership Monetary Value Microsoft (1) Microsoft provided TILO with Basic IT Skills training for 8 teachers from each TILO SBR school in Alexandria and Beni Suef governorates, as part of the Microsoft Digital Literacy program. $11,000 (for both Beni Suef and Alexandria training) Microsoft (2) The 2008 Microsoft-MOE School Agreement provided TILO with free software for 4,470 computers $233,558 Oracle Oracle sponsored the training of one teacher per TILO Smart School (TSS) on the Introduction to Computer Science program. not valued Oracle (2) Oracle granted access to TILO schools in Cairo, Alexandria and Beni Suef to participate in the Oracle Education Foundation Project Learning Institute (PLI) training program. not valued Intel (1) Intel provided Basic IT Skills training for 8 teachers from each TILO Smart School (TSS) for the Intel "Getting Started" course. not valued Intel (2) Intel donated 1,000 Classmate laptops, memory upgrades, a percentage of the charging racks, and training per school. $524,500 Intel (3) Intel, in cooperation with the MOE, donated an additional quantity of 415 Classmate PCs to TILO schools. This quantity provided the CMPC model to an additional 16 SBR Prep expansion schools. $249,000 Intel (4) Intel donated 15 new laptops as incentive prizes for best performing MOE master teachers conducting training in schools. $20,000 IBM (1) IBM donated the Reading Companion program in TILO Smart Schools (TSS) plus $1,000 worth of hardware (headsets for TILO student computers in TSS schools). $86,000 (all licenses in all TSS schools) IBM (2) IBM contributed (through the MOE) 6 Kid-Smart Units (machines) to TILO Helwan schools $15,600 IBM (3) IBM contributed an additional 78 Kid-Smart hardware units to TILO expansion schools. $202,800 IBM (4) IBM provided a $10,000 grant to TILO to provide expansion schools with IT suitcases to support the technology model. $10,000 IBM (5) IBM donated an additional 180 headsets for TILO schools using the Reading Companion Program. $3,600 IBM (6) IBM contributed an additional $5,000 grant for the development of curriculum related e-books for the Reading Companion Book Builder program. $5,000 Houghton Mifflin Harcourt (HMH) Houghton Mifflin Harcourt Publishing donated Kid-Smart software licenses and training manuals to every TILO primary school. This content is in Arabic. $180,000 (licenses and manuals for all SBR primary schools) Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 57 The PPP activities noted above were contributed to TILO schools based on a structured plan made created by the TILO management team. TILO has fully satisfied the two indicators 2.2.9.1 and 2.2.9.2 identified for monitoring PPP activities of the project and contributed towards achieving the Outcome Level Intended Result 3, “Increased Local Support.” Outcome Intended Result 4: Enhanced School Management of IT This outcome level intended result was assessed by examining one sub-result: 10: School administration managing education technology resources effectively Orchtech Orchtech provided TILO with a much reduced price (80% lower) for Crocodile Chemistry and Physics for all TILO TSS schools and Crocodile Math for all TILO schools. $304,000.00 DCGEP/ Coca Cola Africa Foundation (1) With funds from the Coca Cola Africa Foundation, the Discovery Channel Global Education Partnership contributed funds to TILO to customize 150 educational video segments and provide training to 60 primary schools in Beni Suef and Alexandria. $625,372 DCGEP/ Coca Cola Africa Foundation (2) The DCGEP partnership granted additional funds to expand the program into 40 additional schools: Minya 19, Alexandria 7, Fayoum 6, and Beni Suef 8. $246,308.96 DCGEP/ Discovery Communications Inc Discovery Communications (DCI), the parent company for the Discovery channels, contributed the original versions of 150 video segments and raw footage, valued at $5.5 million for the videos and $1 million for the footage. This was a one-time contribution and the videos were used as the base to customize Egyptian versions. DCI also contributed staff time and local production, but these contributions are shared with other DCGEP countries and a percentage of the contribution was not assigned. $6,500,000 Hewlett Packard (HP) HP sponsored ICDL training for TILO school teachers. $74,000 RWE RWE supported part of the TILO technology and training model for 4 SBR primary schools in Helwan governorate. $42,746 HSBC Bank HSBC fully funded the equipping of one TILO SBR Prep expansion school in Cairo governorate. $29,500 Exxon Mobil Exxon Mobil donated used IT equipment in very good condition to 20 TILO expansion schools in Cairo and Alexandria Governorates. Each school received 10 PCs and 4 laptops to implement the TILO model. $40,000 Yousef El Genade Yousef El Genade, a wealthy landowner in Beni Suef, donated land next to the El Shahed Hesham El Gamal school so the school could expand their grounds. $60,000 Winning Egypt Winning Egypt, a Training and Consultancy firm in Minya, provided grants to 20 teachers in the best performing schools to attend professional development courses. $10,000 TOTAL $9,472,985 Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 58 Summary of Findings Sub-Result 10: School administration managing education technology resources effectively  Percent increase in school technology support as measured by T6 (2.2.10.1): The percentage of SBR Primary schools whose Principals reported a satisfactory level of support for technology at their schools rose by 67% between the baseline and the M3 measurement in April 2011. Tools and Instruments Used Table 13 shows the data sources/tools that were used in order to acquire data to assess the indicator pertaining to the sub-result for Outcome Level Intended Result 4. Table 13: Outcome Level Intended Result 4 Indicators, Sources of Data and Tools Detailed Findings 2.2.10 School Administration Managing Educational Technology Resources Effectively 2.2.10.1 Percent Increase in School Technology Support as Measured by T6 (Outcome PI 10.1) This indicator was assessed through the T6 tool. Table 14 below looked at the percentage of schools whose principals responded at each point of the scale to Questions 7-10 at the baseline and the M3 measurement in April 2011. Responses were graded on the tool’s 5-point scale, with a score of “five” indicating the highest level of support and “one” indicating the lowest. The tool differentiates between two levels of performance based on the scores: “Effectively Managing Technology Resources” (Scores 3-5) and “NOT Effectively Managing Technology Resources” (Scores 1-2). Table 14: Cumulative Evaluation Scores of the Schools’ Management of Technology Resources, T6, Qs 7-10 Scores 1-5 % # of Schools / Score BL M3 5 0% 19% 4 0% 17% 3 13% 44% 2 4% 15% 1 83% 5% Cumulative percentage of scores 3, 4, and 5 13% 80% The information in Table 14 showed that the effective management of technology at schools progressed considerably over the years of the TILO intervention. The percentage of schools that reported “effectively managing technology resources” (cumulative scores 3, 4, and 5) rose by 67%, from 13% at Sub-Results Indicators Sources of Data Tools 2.2.10 School administration managing education technology resources effectively 2.2.10.1 Percent increase in school technology support as measured by T6. Schools/Principals/ Managers of Technology Administration of T6 (Qs7 – 10) Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 59 baseline to 80% at M3. This indicator was reasonably satisfied in TILO schools and contributed towards the achievement of Outcome Level Intended Result 4, “Level of School Technology Support”. Outcome Intended Result 5: Enhanced MOE Capacity for Management of Education Technology Resources This outcome level intended result was assessed by examining one sub-result: 11: Increased capacity in managing education technology resources at MOE Muderiya, Idara, and school levels Summary of Findings Sub-Result 11: Increased capacity in managing education technology resources at MOE Muderiya, Idara, and school levels  Key new technology management practices adopted and identifiable in the workplace (2.2.11.1): TILO introduced and provided training on a number of tools at the Muderiya, Idara and school levels to help TILO Idaras to manage and sustain interventions in TILO schools and institutionalize the TILO model in additional schools. These tools included the “TILO Training and Follow up Support Plan” and the “School Technology Advanced Management Plans” (STAMP). Tools and Instruments Used Table 15 shows the data sources/tools that were used in order to acquire data to assess the indicator pertaining to the sub-result for Outcome Level Intended Result 5. Table 15: Outcome Level Intended Result 5 Indicators, Sources of Data, and Tools Sub-Results Indicators Sources of Data Tools 2.2.11 Increased capacity in managing education technology resources at MOE Muderiya, Idara, and school levels 2.2.11.1 Key new technology management practices adopted and identifiable in the workplace MoE Managers of Muderiya and Idara T7 Qs (6-9) Detailed Findings 2.2.11 Increased Capacity in Managing Education Technology Resources at MOE Muderiya, Idara, and School Levels 2.2.11.1 Key new technology management practices adopted and identifiable in the workplace (PI 11.1) This indicator was planned to be assessed by using tool T7, which was discontinued after the M2 measurement. Therefore, information regarding this indicator was obtained from TILO’s formal reporting documents. TILO training providers provided testimonial reports on the “TILO Training and Follow up Support Plan”, a scheme introduced by TILO at the Muderiya, Idara and school levels with Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 60 the aim of moving TILO schools towards sustainability and helping TILO Idaras to institutionalize the TILO model in additional schools. See Appendix 2, TILO Training and Follow up Support Plan for more information. TILO coordinated with MOE officials at Muderiya, Idara and school levels to develop a scheme for the sustainability of technology support at TILO schools after project activities were completed. These efforts culminated in the development of the “School Technology Advanced Management Plans”, STAMP. The “TILO Training and Follow up Plan” was also developed, and 1,000 supervisors received capacity building training in following up and managing TILO technology at schools. Through this process, follow-up and support committees were formed at Muderiya levels and were tasked with responsibilities for sustaining the TILO model at their schools. Based on formal TILO reports and data received from training providers, it is clear that the activities were successful in introducing new technology management practices at the Muderiya, Idara and school levels. It can be said that TILO has satisfied this indicator and contributed towards the achievement of Outcome Level Intended Result 5, “Enhanced MOE Capacity for Management of Education Technology Resources”. 2.3 Output Level Results The TILO PMP has identified four intended results at the outputs level. These four results were assessed by looking at 11 sub-results that are described under each Output Intended Result in the sections below. The four intended results are:  Training (teachers);  IT infrastructure and digital content (school);  Community Awareness and Public-Private Partnerships; and  MOE Capacity Building Program Summary of Findings Overall, the TILO project satisfied all the output level results noted below. The findings for the output level results are detailed in the sections below. Detailed Findings Outputs Intended Result 1: Training (Teachers) The achievement of this Output Level Intended Result 1 was assessed by examining four sub-results and seven indicators using the tools and questions as shown in Table 16 below. Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 61 Table 16: Output Level Intended Result 1 Indicators, Sources of Data, and Tools Sub-Results Indicators Sources of Data Tools and Questions 2.3.1 Teacher/supervisors trained in effective teaching methods 2.3.1.1 No. teachers / supervisors trained in effective teaching methods TILO database Project reports, Doc Rev 2.3.1.2 No. teachers trained as Master Teachers TILO database Project reports, Doc Rev 2.3.2 Teachers and supervisors trained in how to integrate technology 2.3.2.1 No. teachers / supervisors trained in technology integration TILO database Project reports, Doc Rev 2.3.2.2 No. and type (level 1 and level 2) of training programs TILO database Project reports Doc Rev 2.3.3 Teachers and supervisors mentored and supported in integrating technology 2.3.3.1 Measures of support provided for each group within the TILO Training Support Program Supporting documents on follow up of each group Document review, Doc Rev 2.3.4 School administrators and BOTs trained in effective management of technology use in schools 2.3.4.1 No. of administrators and BOTs trained in effective management of technology use in schools TILO database Project reports, Doc Rev 2.3.4.2 No. and type of training programs TILO database Project reports, Doc Rev Output level indicators were examined through document reviews of TILO’s periodic reports (e.g. Quarterly Reports and department reports). Data for all seven indicators is tabulated in Table 17 below. Table 17: Findings and Results for Training Indicators Results 2.3.1.1 No. teachers / supervisors trained in effective teaching methods Number of Teachers/ Supervisors = 22,390 2.3.1.2 No. teachers trained as Master Teachers Number of Teachers = 1,908 2.3.2.1 No. teachers / supervisors trained in technology integration Number of Teachers/ Supervisors = 8,506 2.3.2.2 No. and type (level 1 and level 2) of training events (in technology integration) Number of events = 353 training programs categorized into 11 Types of Program including: Basic IT Skills, IT Integration, IT Integration for Administrators, Maintenance Trouble Shooting, Digital resources, Discovery, Classmate, IT Integration for Teachers, IT Skills for School Administrators, Moodle Training, and Intel Training. 2.3.3.1 Measures of support provided for each group within the TILO Training Support Program TILO Training and Follow up Support Plan 2.3.4.1 No. of administrators and BOTs trained in effective management of technology use in schools Number of Administrators = 3,988 (inclusive of both MOE and School level administrators) Number of BOTs = 0 Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 62 2.3.4.2 No. and type of training programs Number of training programs = 921 programs categorized in 22 Types of training events included: Basic IT; Effective Teaching Methods; Leading Change; Roles, Responsibilities; Classroom Observation; IT Integration; IT Integration for Administrators; Motivating and Rewarding Change; School Technology Advanced Management Plan – STAMP; Training of Trainers; Maintenance troubleshooting; Digital Resources; Discovery; Capacity Building; Classmate; IBM Reading; IT Integration for Teachers; IT Skills for School Administrators; Moodle Training; Refreshment Training; PAT; and TILO Community. Numerous training programs were provided to thousands of intended training recipients who included teachers, administrators/managers at the school, Idara and Muderiya levels. The only category that did not receive the intended trainings was the BoT members. This was because BOT training was not originally part of the TILO mandate. BOTs were added as part of the PPP with DCGEP, and although they were not provided with formal training, BOT members were included in all awareness-raising sessions and community outreach related activities. The above tables indicate that TILO has succeeded in satisfactorily fulfilling all seven indicators identified to assess the Output Level Intended Result 1, Training. Output Intended Result 2: IT Infrastructure and Digital Content (School) The achievement of this Outputs Level Intended Result 2 was assessed by examining three sub-results and five indicators using the tools and questions as shown in Table 18 below. Table 18: Output Level Intended Result 2 Indicators, Sources of Data, and Tools Sub-Results Indicators Sources of Data Tools 2.3.5 Schools equipped with IT 2.3.5.1 No. of equipment delivered by TILO TILO database Field monitoring reports 2.3.6 Schools connected to the Internet 2.3.6.1 No. of schools with Internet access by type of connection TILO database Field monitoring reports 2.3.6.2 Bandwidth of internet connection TILO database Field monitoring reports 2.3.6.3 Number of computers online TILO database Field monitoring reports 2.3.7 Digital content developed/selected 2.3.7.1 No. and type of digital resources developed / provided by TILO per school Project reports Document review Output level indicators for Outputs Intended Result 2: IT Infrastructure and Digital Content, were examined through document reviews of TILO periodic reports, particularly the quarterly and annual reports. Data for all five indicators is tabulated below in Table 19. Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 63 Table 19: Findings, Infrastructure and Digital Content Indicators Results 2.3.5.1 No. of equipment delivered by TILO 3,628 Desktops, 1,195 laptops, 982 TILO Technology Suitcases, 1,527 Intel Classmates, and 30 Interactive White Boards. 2.3.6.1 No. of schools with Internet access by type of connection 208 schools connected with ADSL, and 31 schools connected with 3G. 2.3.6.2 Bandwidth of internet connection All computers provided to schools were networked and internet connected. 2.3.6.3 Number of computers online All computers provided to schools were networked and internet connected. 2.3.7.1 No. and type of digital resources developed / provided by TILO per school TILO provided more than 60 digital resources in the schools. These included general tools for all subjects and specific subject tools (Math, Science, English, and Social Studies). TILO has successfully fulfilled the five indicators identified for assessing Output Level Intended Result 2, IT Infrastructure and Digital Content (School). Output Intended Result 3: Community Awareness and Public-Private Partnerships The achievement of this Output Level Intended Result 3 was assessed by examining two sub-results and three indicators using the tools and questions as shown in Table 20 below. Table 20: Output Level Intended Result 3 Indicators, Sources of Data, and Tools Sub-Results Indicators Sources of Data Tools and Questions 2.3.8 Parents and communities participate in awareness-raising activities 2.3.8.1 No. of awareness raising activities conducted Project reports / documents Document review 2.3.8.2 No. of participants in awareness raising activities Project reports / documents Document review 2.3.9 Public-Private Partnership agreements supporting IT use in schools 2.3.9.1 No. of PPP agreements in place Project reports / documents Document review Output level indicators were examined through document reviews of TILO periodic reports, particularly the quarterly and annual reports. Data for all three indicators is tabulated below in Table 21. Table 21: Findings, Community Awareness and Public-Private Partnerships Indicators Results 2.3.8.1 No. of awareness-raising activities conducted 153 2.3.8.2 No. of participants in awareness raising activities 4,691 2.3.9.1 No. of PPP agreements in place 25 PPPs, through partnerships with 13 groups Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 64 Overall, TILO’s achievements regarding these three indicators has satisfied the Output Level Intended Result 3, Community Awareness and Public-Private Partnerships. Output Intended Result 4: MOE Capacity Building Program The achievement of this Output Level Intended Result 4 was assessed by examining two sub-results and four indicators using the tools and questions as shown in Table 22 below. Table 22: Output Level Intended Result 4 Indicators, Sources of Data, and Tools Sub-Results Indicators Sources of Data Tools and Questions 2.3.10 A strategically targeted capacity building program for management of technology in place in the MOE 2.3.10.1 Detailed work plan for capacity building program with key management areas for strengthening identified Project reports Document review 2.3.10.2 Organizational assessment of MOE system of management of technology completed Project reports / documents This study was cancelled due to the instability in Egypt that prevented collection of data from Idara and Muderiya sources (see Section 1.2 regarding T7) 2.3.11 MOE Muderiya and Idara administration trained to manage technology effectively, and plan and allocate education resources for technology 2.3.11.1 No. of MOE administrators trained Project reports Document review 2.3.11.2 No. and types of training programs Project reports Document review Output level indicators were examined through document reviews of TILO periodic reports, particularly the quarterly and annual reports. Data for all four indicators is tabulated below in Table 23. Table 23: Findings for MOE Capacity Building Program Indicators Results 2.3.10.1 Detailed work plan for capacity building program with key management areas for strengthening identified Plan was produced within TILO’s overall implementation plan. 2.3.10.2 Organizational assessment of MOE system of management of technology completed No results – Study cancelled. 2.3.11.1 No. of MOE administrators trained 2,319 administrators were trained 12.3.11.2 No. and types of training programs 921 programmes of 22 types were offered through TILO trainings. MoE individuals whose work involved TILO were included in TILO trainings as needed and as relevant. Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 65 Data from the above table showed that TILO has successfully fulfilled three of the four indicators (2.3.10.1, 2.3.11.1, 2.3.11.2). The fourth indicator (2.3.10.2) was cancelled from the scope of the TILO PMP due to circumstances in Egypt after the revolution. Therefore, it can be stated that TILO has successfully fulfilled the indicators identified for assessing Output Level Intended Result 4, MOE Capacity Building Program. Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 66 CONCLUSIONS and RECOMMENDATIONS The M&E component of the TILO project examined 40 performance indicators to assess 11 intended results and 24 sub-results at three levels (Impact, Outcomes, and Outputs). Based on the findings and results, certain main conclusions can be drawn about the performance of the TILO project. These conclusions and as well as some recommendations for future programming are discussed in the sections below. 3.1 Impact Intended Result 1: “Improved Student Learning” This intended result looked at improved student learning as a result of TILO’s interventions over time. The two variables that were examined in detail were “performance over time” and “training.” Conclusions  Performance Over Time: The results of this study are a positive indication that the overall model and approach of the TILO interventions was sound and has potential for sustainability. The results of the “performance over time” study revealed that the academic performance of TILO students in both SBR Primary schools and TSS Prep schools showed significant improvement over the course of the intervention. Even the results of the M4 measurement conducted at the sustainability year (one year after the end of interventions in the schools) showed improvement compared to the baseline.  Training: The results of this study suggest that TILO’s cascade model for training (training some teachers directly and having them train others in their school) worked well. The results for “training” for both SBR Primary schools and TSS Prep schools showed that both students trained by “TILO Trained Teachers” and by “TILO Locally Trained teachers” improved significantly in their average mid-year marks compared to the baseline. An interesting finding in SBR Primary schools was that the average mid-year marks of students of TILO Locally Trained teachers were even higher than those taught by TILO Trained teachers. This was arguably a result of the high motivation of TILO Trained Teachers to pass on their knowledge to their colleagues. They knew the challenges their colleagues were facing and were intimately familiar with the teaching environment, which allowed them to better address any problems that arose. Additionally, the TILO Trained Teachers were experts in their specific subject areas and could provide more targeted support to their colleagues than TILO Master Trainers were able to provide to them. This finding provides additional support for TILO’s cascade model in SBR Primary schools. This model is more cost-effective and also more sustainable in the long-run than training all teachers through project-funded Master Trainers. Recommendations  Replicate the Student Marks Study in 2014 and 2015 in TILO schools which have not received further interventions in order to see if the impact of the TILO activities have been sustained. 3.2 Impact Intended Result 2: “Sustainable Decentralized Management of Use of Technology for Teaching/Learning at the School and Idara Levels” Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 67 This intended result examined the use of technology for teaching and learning as well as decision￾making, and the sustainable use of resources for maintaining and supporting education technology in schools. The main conclusions noted below relate to both SBR Primary and SBR Prep schools. Conclusions  Use of technology at school and Idara levels: o The M&E results indicate that the TILO technology model was successful in helping Idaras and schools become more efficient through the use of technology. o Teachers’ level of engagement and interest in using technology in their classrooms is a positive indication of their sustaining these practices after the life of the project. Schools reported a more effective use of technology over the course of the TILO interventions, and said that the technology was well-integrated into the classrooms. Teachers reported both using the resources that were provided and taking the initiative to provide e-resources through their own research. o The findings relating to teachers’ satisfaction with the support provided by school management, Idaras and Muderiyas underscores the importance of engaging higher level stakeholders throughout project interventions. Schools reported that they were managing their technology well and receiving sufficient support from their school management, Idaras and Muderiyas to do so. One of the main reasons cited for this was the inclusion of school managers, Idara and Muderiya supervisors in the TILO trainings. TILO’s decision to include these personnel helped them to understand what the TILO schools were doing, so that they could provide effective support.  Perceptions of technology: o The TILO model of technology integration was successful in showing how technology can be a good aid to the teaching-learning process. Teachers’ perceptions of technology improved significantly over the course of TILO interventions. Teachers reported greater levels of confidence and competence in using technology. They also stated that using technology had changed their role to facilitators rather than lecturers. o The M&E findings suggest that the technology provided through the TILO approach was successful in engaging students’ attention and desire to remain in school. The findings showed that students had positive perceptions towards the use of technology in learning. They commented that their teachers were using technology to help them learn and feel freer to work better both independently and in teams. Teachers, on their part, reported that students were more engaged, attentive and collaborative.  Integration technology into the teaching-learning process: o While technology was useful to teachers in their teaching, the results indicate that the teachers were able to use the technology in appropriate ways because they had first been trained on how to teach well. In the TILO model, technology was not considered an add￾on but was consciously integrated into the curriculum at different grade levels. It was also included only after teachers had been trained on basic pedagogical techniques. Teachers reported very positively about the trainings they received on topics such as student￾centered teaching, classroom management, etc. They reported that they adopted active learning, creative thinking, and problem solving strategies in the classroom when working with technology. This finding suggests the soundness of the TILO model in terms of using technology as a tool to enhance learning, rather than an end in itself.  Sustainable management of resources: o While TILO was successful in increasing the effective use of technology at the school level, control of financial resources was not decentralized and financial allocation of Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 68 technology resources was conducted primarily at the central level. This will make it difficult for Idaras and schools to plan financially for the proper maintenance and support of the technology in their jurisdictions.  Phasing of interventions in cohorts: o TILO conducted its activities in two phases with two cohorts of schools. Cohort 1 received interventions in the first year, during which TILO learned from successes and challenges and made adjustments to the technology and training models when working with Cohort 2. The findings show clearly that the adjustments made a difference not just to the smooth implementation of interventions in Cohort 2 but to its results. These results corroborate the concept of working in phases, or starting with a pilot, making necessary adjustments, and then scaling out to the remaining target areas.  Phasing of interventions in cohorts: o The M&E results showed that Cohort 2 schools performed better than Cohort 1 and showed greater improvement in all categories evaluated through the M&E process. This corroborates the concept of working in phases – starting with a pilot, making necessary adjustments, and then scaling out to the remaining target areas. TILO conducted its activities in two phases with two cohorts of schools. Cohort 1 received interventions in the first year, during which TILO learned from successes and challenges and made adjustments to the technology and training models when working with Cohort 2. The findings show that these adjustments made a difference not just to the smooth implementation of interventions in Cohort 2 but also to the results of these interventions. Recommendations:  Include other relevant stakeholders (such as Idara and Muderiya education officers as well as school principals) in interventions involving training of teachers. This makes a big difference in terms of securing their buy-in and their engagement during the intervention, as well as their interest in and ability to continue to support the initiatives after the end of the project.  Consider ways to decentralize decision-making about the allocation of financial and technology resources to ensure the sustainability of TILO interventions.  Use a phased approach to implementing interventions so that adjustments can be made to address any problems identified in a pilot before the model is rolled out widely. Overall conclusion The findings of the M&E measurements which were conducted over a four-year period during the TILO project indicate that TILO has successfully achieved almost all its intended results at the outputs, outcomes and impact levels. The project has shown that technology must not be seen as an add-on or a goal in itself. It should not supersede pedagogy in classes or be introduced into schools as a separate product. Rather, it should be seen as a tool to be integrated into all aspects of the system to enhance the teaching and learning process. Similarly, in order to make sure that an approach is accepted and understood by all stakeholders, and institutionalized within the system, it is essential to involve not just teachers but management staff at all levels (school, Idara, Muderiya and the central MOE). If the buy￾in of all stakeholders is not ensured early in the process, the interventions will not survive the duration of the program. TILO has produced a model that shows positive signs of being both sustainable and replicable within the Egyptian education system. The fact that, by the end of the project in September 2013, the MOE had begun to expand the TILO model by itself in 288 non-TILO schools (40 in Fayoum, 213 in Minya, 14 in Beni Suef and 21 in Alexandria) is testament to its success. Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 69 APPENDICES Appendix 1: TILO Performance Monitoring Plan (PMP) Project Components 1- Improve the Quality of Teaching, Learning and IT Management in TILO Schools 2- Public-Private Partnerships 3- Capacity (MOE) for Management of Technology Project Component RESULTS PERFORMANCE INDICATORS SOURCES OF DATA METHOD OF DATA COLLECTION PERSON RESPONSIBLE FREQUENCY IMPACT All 1. Improved student learning 1.1 Increase on CAPS scores at participating schools CAPS results Administration of CAPS NCEEE in coordination with M&E TILO Baseline and annual Since CAPS was not available, this result was assessed through a study designed to measure the increase in mid￾year exam scores of TILO students in 3 grades for 5 subjects over four school years for both SBR Primary and TSS Prep schools. All 2. Sustainable decentralized management of use of technology for teaching / learning at the school and Idara levels 2.1 Data driven decisions at the Idara and school levels Project reports, school records, MOE reports, policies, Organizational Impact Assessment Study (document review, focus groups and Capacity building consultant with M&E team Baseline, mid￾project, end-of￾2.2 Effective use of project technology for teaching and learning in schools Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 70 Project Component RESULTS PERFORMANCE INDICATORS SOURCES OF DATA METHOD OF DATA COLLECTION PERSON RESPONSIBLE FREQUENCY 2.3 Sustainable use of resources of education technology in schools: a. functioning maintenance system, b. plan and budget for IT resources periodic updating Muderiya and Idara staff, school administration and teachers interviews with MOE staff) 2.4 Schools introducing technology-based innovations TILO Database Field monitoring reports recording technology innovations introduced at TILO schools TILO Software Application Manager OUTCOMES: INTERMEDIATE RESULTS Increased Student Engagement with IT 1 1. Students engaged in active learning powered by technology 1.1 Level of technology integration in student learning activities Video tapes of TILO sample classes Video Classroom Observation form M&E team Baseline and annual 1 2. Students develop positive attitudes towards technology 2.1 No. of students with personal email addresses Students Interviews M&E team, Governorate team, Baseline and 2.2 Students’ perceptions towards annual use of technology in learning Changed Classroom Practice 1 3. Teachers integrating technology into student￾centered methods 3.1 Percent increase in effective classroom teaching practices as measured by SCOPE+ SCOPE+ results Administration of SCOPE M&E team, Governorate team Baseline and annual Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 71 Project Component RESULTS PERFORMANCE INDICATORS SOURCES OF DATA METHOD OF DATA COLLECTION PERSON RESPONSIBLE FREQUENCY 3.2 Level of IT integration in teaching materials Teacher portfolios Teacher Products Rubric M&E team, Governorate team Baseline and annual 1 4. Teachers acquire positive attitudes to integrating technology into instruction 4.1 Level of confidence of teachers in using IT in teaching Self assessment data Questionnaire and Interviews M&E team, Governorate team Baseline and annual 4.2 No. of teachers with personal email addresses 1 5. School leadership supporting teachers’ use of technology 5.1 Level of satisfaction of teachers of administrators' technology support Satisfaction data from teachers Questionnaire M&E team, Governorate team Baseline and annual 1 6. Digital content used in the classroom 6.1 Range of digital material / resources used by teachers and students Teachers and students Questionnaire and Interview M&E team, Governorate team Baseline and annual Increased Local Support 2 7. Parents support use of computers in teaching 7.1 Level of generation of funds/resources for IT by BOTs in TILO schools School records/visits Document review, interviews M&E team, Governorate team Baseline and annual 2 8. Opportunities for parents and community members to use technology 8.1 Degree of utilization of school IT resources by community School records/visits - Document review - interviews M&E team, Governorate team Baseline and annual 2 9. Public-Private-Partnerships increase resources and support 9.1 Monetary ($) value of PPP interventions Project reports Document review M&E team, TILO team Baseline and annual Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 72 Project Component RESULTS PERFORMANCE INDICATORS SOURCES OF DATA METHOD OF DATA COLLECTION PERSON RESPONSIBLE FREQUENCY of technology use in schools beyond TILO’s provision 9.2 Range of products and services provided by PPPs Enhanced School Management of IT 1 10. School administration managing educational technology resources effectively 10.1 Percent increase in school technology support as measured by T6 Schools Principals / Managers of Technology Administration of T6 M&E team, Governorate team Baseline and annual Enhanced MOE Capacity for Management of Education Technology Resources 3 11. Increased capacity in managing education technology resources at MOE Mudireya, Idara and school levels 11.1 Key new technology management practices adopted and identifiable in the workplace MOE and school administrators Interviews with MOE; Observations of office operations M&E team, Governorate team Baseline and annual OUTPUTS: SHORT TERM RESULTS Training 1 1. Teacher / supervisors trained in effective teaching methods 1.1 No. teachers / supervisors trained in effective teaching methods 1.2 No. teachers trained as Master Teachers TILO database Project reports SW Application Manager Quarterly Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 73 Project Component RESULTS PERFORMANCE INDICATORS SOURCES OF DATA METHOD OF DATA COLLECTION PERSON RESPONSIBLE FREQUENCY 2. Teachers and supervisors trained in how to integrate technology 2.1 No. teachers / supervisors trained in technology integration 2.2 No. and type (level1 and level2) of training programs 3. Teachers and supervisors mentored and supported in integrating technology 3.1 Measures of support provided for each group within the TILO Training Support Program Supporting documents on follow up of each group Document review M&E team 1 4. School administrators and BOTs trained in effective management of technology use in schools 4.1 No. of administrators and BOTs trained in effective management of technology use in schools TILO database Project reports SW Application Manager 4.2 No. and type of training programs IT Infrastructure and Digital Content 1 5. Schools equipped with IT 5.1 No. of equipment delivered by TILO TILO database Field monitoring reports SW Application Manager Quarterly 1 6. Schools connected to the Internet 6.1 No. of schools with Internet access by type of connection TILO database Field monitoring reports SW Application Manager Quarterly 6.2 Bandwidth of internet connection TILO database Field monitoring reports SW Application Manager Quarterly 6.3 Number of computers online TILO database Field monitoring reports SW Application Manager Quarterly 1 7. Digital content developed/selected 7.1 No. and type of digital resources developed / provided by TILO per school Project reports Document review M&E team, TILO team Quarterly Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 74 Project Component RESULTS PERFORMANCE INDICATORS SOURCES OF DATA METHOD OF DATA COLLECTION PERSON RESPONSIBLE FREQUENCY Community Awareness and PPPs 22 8. Parents and communities participate in awareness raising activities 8.1 No. of awareness raising activities conducted Project reports / documents Document review M&E team Quarterly 8.2 No. of participants in awareness raising activities 9. Public-Private Partnership agreements supporting IT use in schools No. of PPP agreements in place Project reports / documents Document review M&E team Quarterly MOE Capacity Building Program 3 10. A strategically targeted capacity building program for management of technology in place in the MOE 10.1 Detailed work plan for capacity building program with key management areas for strengthening identified Project reports Document review M&E team Cancelled 10.2 Organizational assessment of MOE system of management of technology completed Cancelled 3 11. MOE Mudireya and Idara administration trained to manage technology effectively, and plan and allocate education resources for technology 11.1 No. of MOE administrators trained Project reports Document review M&E team Quarterly 11.2 No. and types of training programs Project reports Document review M&E team Quarterly Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 75 Appendix 2: TILO Follow Up and Support Plan Evidence for Indicator 9.1 (Outcome Level)  The TILO training team designed and developed the TILO Training Follow Up and Support Plan. The goal of the plan was to: o move TILO schools towards self-sustainability. o help TILO Idaras to institutionalize the TILO model in their schools where there is sufficient interest to expand the TILO model to additional expansion schools.  The TILO Follow Up and Support Plan identified and addressed the needs of each TILO school, enabling them to move towards self-sustainability and provided the necessary support at the Muderiya and Idara levels to institutionalize the TILO model.  In coordination with the Undersecretaries and the Director Generals at the Muderiya and Idara levels, TILO teams held a series of meetings with the Heads of the TDC, Technical Follow up and Support Units, Quality Accreditation Units and Training Units in addition to TILO school Principals, the General Inspectors and the MOE Supervisors to coordinate the implementation of the Follow up and Support Plan.  The TILO team communicated the Follow up and Support Plan to the Muderiyas and Idaras. TILO worked with them to help schools overcome the different challenges they were encountering.  After identifying the schools level of priority, the follow up and support activities aimed to support Priority One (weak) and Two (moderate) schools in moving towards Priority Three (Excellent).  TILO team focused their school visits on supporting the school leadership to apply and sustain TILO activities by implementing their School Technology Advanced Management Plans (STAMP). In addition, they supported classroom teachers to integrate the Effective Teaching Methods and the Integration of the TILO Digital Resources into their teaching practice.  Based on the follow up and support visits and the schools’ performance, a follow up and support Intervention plan for each school was created stating what should be done after each training component, what was achieved and what wasn’t and at what phase the school was performing. The plan included recommendation for the school, Idara and Muderiya.  Each month, TILO teams in all governorates including the Team Leaders met to communicate any issue and exchange opinions on how to deal with these issues. The intervention plans were to be discussed and updated according to the new level of performance of each school and the outcomes of the Follow Up and Support Plan in order to prepare a new and updated intervention plan for the next month for each school.  TILO team built the capacity of more than 1,000 supervisors in all the governments to help in conducting follow up and support to TILO schools in their Idaras and Muderiyas in partnership with TILO team and at the same time plan for sustainability.  TILO formed a team of Multi-Taskers to do the follow up and support in coordination with the Master Trainers and to support them while giving the training when needed.  The MOE formed Muderiya-level Follow up and Support Committees to provide follow up and support to TILO schools. Currently, the Idaras have their own Follow up and Support Plans and have begun to provide support to TILO schools independently.  Generally, TILO team carried out of approximately 1,890 visits to the SBR and TSS schools (phase 1, 2, 3, and 4) and about 1,134 visits to the 127 Extension Prep Schools (phase 1 and 2).  At the end of the Follow up and support plan activities, School Status Reports Revision workshops were held in all governorates. The participants from each Idara and Muderiya with Egypt Technology for Improved Learning Outcomes Project - Final M&E Report 76 their different departments, involved in the follow up and support on TILO schools, revised the content of the reports, agreed upon their content and added their inputs. After that the reports were sent back to the office in Cairo for final editing and formatting.  After formatting, revising and editing the School Status Reports at the office in Cairo, a final School Status Report was prepared for each school to be sent to the Muderiyas and then formally delivered to each school through their Idaras.