RADIO INSTRUCTION TO STRENGTHEN EDUCATION (RISE) AND ZANZIBAR TEACHER UPGRADING BY RADIO (ZTUR) Post-Project Evaluation in Zanzibar Final Report January, 2015 EXECUTIVE SUMMARY BACKGROUND TO THE RISE AND ZTUR PROJECTS The Radio Instruction to Strengthen Education (RISE) Projects and Zanzibar Teacher Upgrading through Radio (ZTUR) Projects were established in Zanzibar in 2006. The programs ran consecutively through 2011 with the goals of: 1. Building the capacity of Zanzibar’s Ministry of Education and Vocational Training (MoEVT) staff in the area of early childhood development (ECD) and education – which this report defines as services for children through the age of 8, and within Zanzibar’s education system covers two years of preschool education and Standards 1 and 2; 2. Improving access to quality math, literacy and life skills instruction and materials for underserved young children (preschool to grade two); and 3. Developing a quality distance and open learning program through which in-service early childhood teachers can upgrade their skills and work towards preschool accreditation. RISE was implemented jointly by Education Development Center, Inc. (EDC) and Zanzibar’s MoEVT, and sponsored by the United States Agency for International Development (USAID). RISE activities were implemented in Zanzibar’s two lowest-performing districts, and eventually expanded into a Zanzibar-wide teacher training and certification program through ZTUR. The key activities under the RISE project were:  Developing and piloting three series of Tucheze Tujifunze1 (Tu Tu) Interactive Audio Instruction (IAI)2 programs for preschool, Standard 1 and Standard 2 students, including those in formal and non-formal settings, with accompanying instructional and learning materials;  Establishing over 180 Tucheze Tujifunze early childhood learning centers (TuTu Centers) in areas with poor access to ECD and education;  Producing interactive videos for guiding teacher professional development in teaching English as a second language;  Training a cadre of preschool and early grade educators (formal and non-formal) to teach literacy, math, and life skills; and 1Tucheze Tujifunze, translates as “Let’s Play, Let’s Learn” and is a name used locally to refer to the IAI programs. 2 IAI is an interactive teaching and learning pedagogy that promotes quality learning in diverse environments, including those with shortages of qualified teachers, school infrastructure, and learning materials. Based on child￾friendly instruction, which is inclusive, playful, active, and engaging, each 30-minute broadcast uses the formal Zanzibar curriculum as the foundational content (Kiswahili, math, English and basic life skills), but integrates games, songs, and activities through a common story-line. The IAI programs encourage problem solving and self-directed exploration. ii  Building the capacity of a MoEVT team to lead distance and open learning for all of Zanzibar. At the close of the project, RISE had provided ECD opportunities to over 35,000 Zanzibari children3 and trained over 809 formal teachers and non-formal facilitators in IAI pedagogy, classroom management and other key areas. The project also distributed over 28,000 teaching and learning materials developed by the team. The subsequent ZTUR Project was designed to further RISE’s ECD work by creating a comprehensive preschool teacher certification program to be implemented by MoEVT. The intention of the certification program (the Early Childhood Advancement Certificate Program, or ECACP) is to provide in-service teachers with critical knowledge, skills, and practice through a distance learning platform administered through their district in-service Teacher Centers. The ECACP program consists of audio, video and print materials that guide teachers through ECD instructional theory and practice, and support them as they actively apply their knowledge in their classrooms. STUDY OBJECTIVES In 2014, EDC commissioned a post-project study to answer the following questions: 1. To what extent have the ZTUR and RISE Project activities been sustained or expanded under the leadership of MoEVT after the projects were transferred in March 2011? 2. Do learners that participated in IAI (Tucheze Tujifunze) programming perform higher at the end of their primary cycle than those that did not participate? The study uses a mixed-method design (process and impact evaluations) to explore these questions. The process evaluation looks at what has been sustained, how it has been sustained, and what challenges and success MoEVT has experienced along the way. Interviews and surveys were conducted between March and June 2014 with 94 individuals, including MoEVT officials from the central and district offices, Teacher Center staff, head teachers of RISE intervention schools, teachers that were trained in Tucheze Tujifunze programming, and learners that listened to the programs. Data were analyzed qualitatively to elicit common themes in responses related to the research questions. The impact evaluation looks at student performance at Standard 7 (equivalent to grade 7 in the United States) Kiswahili, English, and Math. The test was developed by the evaluation team based on an existing government exam. The same sample of learners who took part in an earlier evaluation in 2008 (when they were in Standard 1) was used for the subsequent 2014 data 3Note that over 35,000 unique learners were reached. However, as some learners received multiple years of intervention (preschool, Standard 1 and Standard 2) the total aggregated learners by year is 44,843. iii collection. Of the total 1,543 learners from the 2008 sample, 904 (or 58.6%) were tested in 2014; including 552 from the RISE group (those that had listened to the Tucheze Tujifunze programming) and 352 from the comparison group (students in a district with similar demographics that did not have direct exposure to the Tucheze Tujifunze programming). Not all students tested in 2014 were found to be in Standard 7; assessment data for those who were in lower grades were not included in the analysis of the assessment results since the test was developed to test Standard 7 proficiency. FINDINGS INSTITUTIONAL TRANSFER AND SUSTAINABILITY OF RISE AND ZTUR PROJECTS The main success of the RISE and ZTUR Projects consisted of successfully establishing a distance and opening learning division (the eLearning Division), within the MoEVT. The eLearning Division consists of the staff who were trained and supported by RISE and ZTUR; this continuity has allowed the division to further some of the work started under the projects. By 2014 there were 14 technical staff working in the eLearning Division, and their activities (which continue to build on the foundational activities established under the RISE and ZTUR Projects) were in the process of being further developed with support of the Global Partnership for Education (GPE). By 2016, 120 new centers (60 in Pemba’s Mkoani, 60 in Unguja’s North B) will be opened, materials and training will be provided to 240 mentors, and 200 new preschool programs will be recorded. As of 2014, 179 of the original 180 Tucheze Tujifunze ECD centers are still in existence, and some efforts have been made to help build more permanent spaces for the centers that were in non￾permanent structures. Payments for TuTu Center mentors have been fully integrated into the Ministry’s annual budget, and the augmentation of mentor remuneration in 2013 is evidence of the division’s commitment to providing continued support to its mentors. Broadcast of Tu Tu lessons has continued, with the exception of a 4-month interruption in 2014, when the Voice of Tanzania, Radio Zanzibar underwent restructuring and semi-privatization. All preschool programs are now being aired by ZBC. Head teachers (principals), formal school teachers and mentors (non-formal teachers in the Tucheze Tujifunze centers) all expressed the continuing importance of the Tucheze Tujifunze programming. Mentors have continued to facilitate classes, despite limited access to the programs and materials and the resource-lean circumstances. The RISE continuous training model has not been sustained, largely because of a lack of allocated funds for training. However, two trainings took place in early 2014 to build the capacity of new mentors for the 51 newly established centers. Funds were provided by GPE. iv The ECACP certification program, which was designed and developed under ZTUR, was rolled out in seven Teacher Centers early in 2014.4 Currently, support is being provided by the GPE to implement ECACP across Zanzibar. At the time of this study, there were 352 teachers from all ten districts going through step 1 and 26 trainers had been trained by the e-Learning Division and Aga Khan Foundation. All 26 trainers had received ECD certificates from Aga Khan Foundation. Over 700 teacher participants are expected to benefit from the ECACP, starting in 2014. The major challenges to continued quality implementation lie in maintaining both access to and quality of materials and programming. Maintenance and replacement of radios and broadcasting are major barriers to listenership. A lack of printing and distribution of new materials and limitations on the resources committed to regular trainings and monitoring also affect the quality and fidelity of implementation. LEARNER PERFORMANCE The 2008 evaluation showed that the RISE IAI programming and approach had increased learning gains in both formal and non-formal settings in Zanzibar relative to a comparison group (students in formal schools that did not receive IAI programming). RISE participants demonstrated substantial test score gains across all three tested disciplines (Kiswahili, math and English). Although learners in formal treatment classrooms made the most significant learning gains, non-formal treatment learners were not far behind them. Whereas both boys and girls benefited from the RISE intervention, girl learners showed greater overall growth. The 2008 evaluation demonstrated that RISE’s unique combination of IAI radio programming and early childhood education not only engaged learners, but also promoted the quality of their learning. The 2014 study tracked approximately 59% of the students from the original sample. Slightly more students from the RISE formal school model group (63.4%) were located and included in this study than from the RISE non-formal school model group (55.9%). Fewer students from the RISE non-formal and combination non-formal and formal intervention models were located than from the RISE formal schools. Consequently, the study team made a decision to combine students from the RISE non-formal and combination models into a “mixed” group for analysis. The 2014 assessment found that RISE formal school students performed significantly better on standardized grade-level assessments in Kiswahili, English and math, compared to their counterparts from the comparison group and from the RISE mixed group. Six years after the intervention, RISE formal school students demonstrated a better mastery of grade-level concepts. However, the overall performance level in English and math was found to be very low among all study groups. 4Centers include: Kiembe, Samaki, Bububu, Mkwajuni, Dunga, Kitungoni, Mizingani-Mkoani. v RISE appears to have contributed to sustained advantages for the formal school students who participated in the program. Unfortunately, not enough students from the non-formal or combined groups (the mixed group) could be traced to enable the drawing of strong conclusions about the effect of the program on non-formal populations. A significant portion of RISE mixed group students either could not be found for inclusion in the study, or had repeated grades and thus could not be included in the analysis. Since those students were particularly disadvantaged, due to very low parental literacy or due to poverty, they were an important focus of the RISE intervention. Determination of the long-term benefit of the Tuchzeze Tujifunze programming on their academic performance would be helpful in informing policy decisions about how to target continued interventions for similar populations through the eLearning Division of the MoEVT. Additional follow up or tracer studies that focus particularly on this non-formal group of students would be beneficial. CONCLUSION Despite logistical and coordination challenges, which are common when donor-funded projects are transferred to an official counterpart institution, the MoEVT’s goal of using RISE and ZTUR as the foundation for long-term improvements in ECD in Zanzibar has largely been achieved. The eLearning Division has been strategic about moving its work forward with the support and guidance of the MoEVT leadership. eLearning staff have made conscious and noble efforts to sustain and scale-up RISE and ZTUR activities and to ultimately ensure that their policy goals related to early childhood development are advanced. The insistence of the MoEVT leadership that the division remains cohesive, as it was created under the RISE and ZTUR Projects, has ensured the sustainability of the team and their work, and encouraged the confidence of new partners who can substantially contribute to continued development of this sector. In addition, the Tucheze Tujifunze model of intervention has demonstrated sustained positive effects on students who participated in the programs, providing them with an advantage in academic performance 6 years after their initial performance was assessed. This is particularly true for students in the RISE formal school group. However, the assessed achievement of both RISE and comparison participants was still relatively low, when compared to national expectations for grade-level performance in Standard 7. Clearly, as identified in the Zanzibar Education Policy and Zanzibar Education Development Programme, work remains to be done to increase the percentage of Zanzibari students who meet performance standards across pre￾primary and primary education. Additional tracer research on the impact of the Tucheze Tujifunze programming model on the most disadvantaged students (those in the RISE non￾formal group) would also help ensure the greatest possible impact on that group as programming continues to evolve. Given the Zanzibari context, IAI programming remains a viable and useful intervention format for increasing access to high-quality early education, particularly in the most challenged districts. Careful attention will need to be paid to the mechanisms necessary for continuing to ensure the vi quality of Tucheze Tujifunze programming (specifically: management of hardware resources; thoughtful integration of the programs into the increasingly dense school timetable; the provision of continuous support and professional development for teachers and mentors; and sufficient and effective monitoring and evaluation of programs as they are rolled out). With additional investment in these quality assurance processes, and the continued commitment of the MoEVT to the work of the eLearning Division, the improvements begun under RISE and ZTUR can continue to be built upon and sustained. TABLE OF CONTENTS EXECUTIVE SUMMARY ................................................................................................................................................................. i BACKGROUND TO THE RISE AND ZTUR PROJECTS .................................................................................................. i STUDY OBJECTIVES ................................................................................................................................................................ ii FINDINGS ...................................................................................................................................................................................iii INSTITUTIONAL TRANSFER AND SUSTAINABILITY OF RISE AND ZTUR PROJECTS .......................iii LEARNER PERFORMANCE ........................................................................................................................................... iv CONCLUSION .............................................................................................................................................................................. v ACRONYMS .................................................................................................................................................................................... viii ACKNOWLEDGEMENTS ............................................................................................................................................................ ix INTRODUCTION ............................................................................................................................................................................. 1 EDUCATION IN ZANZIBAR .................................................................................................................................................. 1 THE RISE AND ZTUR PROJECTS ....................................................................................................................................... 3 METHODOLOGY ............................................................................................................................................................................. 6 2008 EVALUATION ................................................................................................................................................................. 7 2014 POST-PROJECT EVALUATION ................................................................................................................................ 8 SAMPLE .................................................................................................................................................................................. 8 ........................................................................................................................................................................................................10 DATA COLLECTION (PROCESS AND TOOLS) ............................................................................................................10 DATA ANALYSIS .....................................................................................................................................................................11 LIMITATIONS ..........................................................................................................................................................................11 FINDINGS ........................................................................................................................................................................................13 FINDINGS: INSTITUTIONAL TRANSFER AND SUSTAINABILITY OF RISE AND ZTUR PROJECTS .....14 RELEVANCE TO THE ZANZIBAR EDUCATION POLICY AND ZANZIBAR EDUCATION DEVELOPMENT PROGRAMME (ZEDP) ..................................................................................................................14 SUSTAINABILITY OF RISE AND ZTUR ACTIVITIES AFTER TRANSFER TO MOEVT ..........................15 CONCLUSION: INSTITUTIONALIZATION AND SUSTAINABILITY ..............................................................32 FINDINGs: LEARNER PERFORMANCE .........................................................................................................................34 STUDENT DEMOGRAPHICS ........................................................................................................................................34 ASSESSMENT FINDINGS ...............................................................................................................................................39 DISCUSSION .......................................................................................................................................................................48 CONCLUSION: LEARNER PERFORMANCE .................................................................................................................49 CONCLUSION .................................................................................................................................................................................50 REFERENCES .................................................................................................................................................................................51 APPENDIX: LEARNING AND TEACHING MATERIALS ................................................................................................. 52 ACRONYMS ECACP Early Childhood Advancement Certification Program ECD Early Childhood Development EDC Education Development Center, Inc. GER Gross Enrollment Ratio GPE Global Partnership for Education IAI Interactive Audio Instruction IEU Inclusive Education Unit MoEVT Ministry of Education and Vocational Training NER Net Enrollment Ratio RISE Radio Instruction to Strengthen Education SDLM Self-directed Learning Modules SMC School Management Committee TuTu Tucheze Tujifunze (IAI series) USAID United States Agency for International Development ZBC Zanzibar Broadcasting Corporation ZEDP Zanzibar Education Development Plan ZTUR Zanzibar Teacher Upgrading by Radio ix ACKNOWLEDGEMENTS EDC thanks the Ministry of Education, Training and Vocational Training (MoEVT), primarily the Deputy Principal Secretary, Mr. Abdulla Mzee; Director of ICT in Education, Mr. Omar Said Ali; and MoEVT Pemba office staff for their generous support of this evaluation. The Ministry’s eLearning Division representatives, Abrahman Faki Othman and Sabrina Masoud Suleiman, were instrumental in coordinating the testing of learners and donated countless hours and resources to ensuring that quality data were collected. The study’s two lead researchers and coordinators, Asha Mohammed Ahmed and Fatma Kassim Mohammed, managed all the interviews, surveys and testing, and ensured data was processed efficiently. We thank the eLearning Division staff and field data collectors for their tireless efforts to delivery timely, accurate and high quality data. 1 INTRODUCTION EDUCATION IN ZANZIBAR Zanzibar is an archipelago in the Indian Ocean situated off the east coast of Tanzania, the two largest islands of which are Pemba (population 406,848) and Unguja (population 896,721).5 Zanzibar is approximately 99% Muslim and is economically reliant on tourism and other services as well as agriculture, including fishing, seaweed farming, and production of cloves and other spices.6 Approximately 49% of the population is between the ages of 0 and 17.7 Although Zanzibar officially became part of the United Republic of Tanzania in 1964, it has retained autonomy over its internal affairs, including the provision and financing of education. The government of Zanzibar is committed to and has made progress toward achieving universal primary education. In 2006, the Ministry of Education and Vocational Training (MoEVT) established the Zanzibar Education Policy (“Education Policy”) and set a number of goals, including improving the quality of education and instruction, increasing the number of trained preschool teachers, reducing the Standard 1 entry age, and addressing an acute shortage of classrooms and serious overcrowding in a number of districts. From the policy, the Zanzibar Education Development Plan (2008-2015), or ZEDP, was developed to set a framework for action and to develop achievable targets. In 2006, when the Education Policy was approved, primary school starting at Standard 18 was free and compulsory in Zanzibar, and the gross enrolment ratio (GER) 9 was over 100%.10 However the net enrolment ratio (NER) was 77.3% for boys and 78.7% for girls, 11 which increased to an overall 81.5% by 2010.12 Regionally, there are some differences in enrolment across islands and urban and rural areas; according to 2010 MoEVT data, Pemba island had a 5 National Bureau of Statistics, United Republic of Tanzania. (2013). Population and Housing Census 2012. pp. vii 6 ILO. (2008). Good Practices on Social Protection and Coping Strategies used by Low Income Women Workers in the Informal Economy in Zanzibar to Mitigate Against Social and Economic Risks, 6. Accessed on November 2, 2014. http://www.ilo.org/wcmsp5/groups/public/---dgreports/--- gender/documents/publication/wcms_098172.pdf. 7 National Bureau of Statistics, United Republic of Tanzania. (2013). Population and Housing Census 2012. pp. vii 8 Standard 1 is equivalent to US Grade One. 9 GER measures the total number of children enrolled, regardless of age, as a percentage of the population of children of that school age. The NER measures children of the appropriate age group enrolled in school as a percentage of the population of children at that school age. 10 Note that specific data related to preschool education is not disaggregated from past national data. 11Ministry of Education and Vocational Training, Revolutionary Government of Zanzibar. (2006). Education Policy. p.11 12Tanzania Millennium Development Goals. (September 2011). Country Report on the Millennium Development Goals. Dar es Salaam, p. 16 2 NER of 80.9%, while Unguja island’s NER was 93.2%. These data indicate that students in Pemba are enrolling in school later than learners in Unguja, especially those in the rural areas.13 Zanzibar’s NER is lower than GER largely due to the significant number of over-aged students matriculated.14 In order to address the over-age challenge, the new education policy officially lowered the Standard 1 entry from eight to six years of age and set the target of raising preschool (pre-primary) gross enrolment from 15.9% in 2005 to 35% by 2010.15 The ZEDP further set the target of “increased enrolment in public preschools to reach 30% by 2016 and expansion of enrolment to private preschools to reach 40% by 2016.”16 Gender balance in access and retention is another focus of the ZEDP. During the first period of the policy, between 2006 and 2009, gender parity in primary education remained stable, with 49.9% males and 50.1% females enrolled in schooling.17 Although gender parity in gross enrollment had been achieved at the primary level by 2006, the Education Policy recognized the importance of ensuring the retention and progression of girls throughout secondary and tertiary levels, as potential gender inequalities caused by early marriages and pregnancies, household responsibilities, cultural values, and lack of resources for supporting female achievement and retention have led to gender disparities in enrolment in higher grades.18 Another major concern addressed in the Education Policy and subsequent ZEDP is the availability of primary and preschool education facilities. Efforts are being made by MoEVT and partners to increase the number of facilities (both public and private) to address overcrowding. Between 2006 and 2009, the number of primary schools increased by 25% from 185 to 232.19 By 2010 there were a total of 425 preschools in Zanzibar, and the number of preschools is steadily increasing, with a particularly rapid increase in the number of private pre-schools. Despite 13Ministry of Education and Vocational Training, Revolutionary Government of Zanzibar. (2013). Zanzibar EMIS office under the Department of Policy, Planning and Research (DPPR). 14USAID/Tanzania (2009). USAID/Tanzania Education Strategy for Improving the Quality of Education FY 2009-2012, p. 2 15Ministry of Education and Vocational Training, Revolutionary Government of Zanzibar. (2006). Education Policy, p.11. Data on the gross enrollment is from the MKUZA;/ZSGRP (Zanzibar Strategy for Growth and Reduction of Poverty 2007-2010), as referenced in the ZEDP report: Ministry of Education and Vocational Training, Revolutionary Government of Zanzibar. (2007). Zanzibar Education Development Program 2008-2016, p.11. 16 Ministry of Education and Vocational Training, Revolutionary Government of Zanzibar. (2007). Zanzibar Education Development Program 2008-2016, p.59. 17 Ibid, p. 52 18 Ministry of Education and Vocational Training, Revolutionary Government of Zanzibar. (2006). Education Policy. p.59 19Office of Chief Government Statistician, Zanzibar. (May 2010). Socio-Economic Survey, p. 49 “Preschool education was not compulsory in the past, so the Ministry neither had pre-schools nor qualified teachers at this level. RISE helped the Ministry to increase the pre-school coverage in remote areas in the same way as in town.” Director, Department of ICT in Education 3 increases in education facilities, however, student to teacher ratios have only slightly changed, with an average of 29 students for every 1 teacher in 2006 to 28 students to 1 teacher in 2009.20 The teacher-student ratio is greater in primary schools than in secondary schools, and can reach as high as one teacher to 100 students in the early grades. Despite the progress that has been made, a lack of sufficient facilities in some areas and the need for trained teachers to keep up with the expansion are still critical issues. Finally, the Education Policy and ZEDP are intended to address the overall quality of education. Formal test results are low for Zanzibar overall, with 2000 SACMEQ results showing that less than 3% of Standard VI (Grade 6) performed at adequate and proficient levels during a reading and mathematics assessment.21 While more students are matriculating at younger ages, their performance on high-stakes assessments and transition to secondary school is still a critical issue. THE RISE AND ZTUR PROJECTS The Radio Instruction to Strengthen Education (RISE) and Zanzibar Teacher Upgrading by Radio (ZTUR) projects, hereafter referred to as RISE and ZTUR, were established to support the Ministry of Education and Vocational Training (MoEVT) in its efforts to put early childhood development and education strategies from the 2006 Education Policy into practice. Both projects were funded by USAID, and were implemented by Education Development Center (EDC) in partnership with MoEVT. The main objectives of RISE were to: 1) build the capacity of Zanzibar’s MoEVT staff in the area of early childhood development (ECD) and education; and 2) improve access to quality math, literacy and life skills instruction and materials for underserved children (preschool to Standard 2). The primary objective of ZTUR was to develop a quality distance and open learning program through which in-service teachers can upgrade their skills and work towards preschool accreditation. RISE worked primarily in Zanzibar’s two lowest-performing districts (Micheweni District in Pemba and North A District in Unguja) between 2006 and 2010, and eventually expanded into the Zanzibar-wide teacher training and certification program implemented by ZTUR between 2010 and 2011. 20 Ibid, 59 21 The Southern and Eastern Africa Consortium for Monitoring Educational Quality (SACMEQ) includes 17 countries in East, Central and Southern Africa. The SACMEQ provides the only consistent comparative information on regional education quality. 4 The key activities under RISE were:  Developing and piloting three series of Interactive Audio Instruction (IAI)22 programs for preschool, Standard 1 and Standard 2 learners, including those in formal and non￾formal settings, with accompanying instructional and learning materials. The Tucheze Tujifunze23 IAI programs were carefully developed by Zanzibari educators and integrated math, life skills, and English and Kiswahili literacy lessons into comprehensive 30-minute programs. The IAI programs complemented the standard curriculum, engaged listeners through locally based stories and activities, and supported teachers with minimal ECD instructional experience. The programs were broadcast on the government-sponsored radio station as well as through portable media player technology.  Establishing over 180 Tucheze Tujifunze (TuTu) early childhood learning centers (TuTu Centers) in areas with poor access to early childhood development (ECD) and education. TuTu Centers offered non-formal preschool and Standard 1 education to the most vulnerable children in remote communities in two of Zanzibar’s ten districts. RISE staff trained local community members as mentors to guide learners through the TuTu broadcasts and post-broadcast activities in non-formal settings identified by the community. This intervention was known as the RISE Non￾formal School Model. In some pilot circumstances students received both non-formal TuTu programming after the formal school day in their formal RISE schools (Combination Model). However, this model was discontinued in 2009, based on results from the 2008 Impact Study.  Training formal Standard 1 and Standard 2 teachers in IAI pedagogy and how to utilize Tucheze Tujifunze lessons and materials in their classrooms. This intervention was known as the RISE Formal School Model.  Producing interactive videos for guiding teacher professional development in teaching English as a second language.  Building the capacity of a cadre of MoEVT educators to write and develop IAI programs and accompanying instructional and learning materials; produce audio and video 22 IAI is an interactive teaching and learning methodology that promotes quality learning in diverse environments, including those with shortages of qualified teachers, school infrastructure, and learning materials. Based on child￾friendly instruction, which is inclusive, playful, active, and engaging, each 30-minute broadcast uses the formal Zanzibari curriculum as the foundational content and includes Kiswahili, math, English and basic life skills, integrating games, songs, and activities through a common story-line. The IAI programs encourage problem solving and self￾directed exploration. 23Tucheze Tujifunze, translates as “Let’s Play, Let’s Learn” and TuTu is the name that is used locally to refer to the IAI programs. 5 programs; and conduct monitoring and evaluation activities to measure fidelity of implementation and outcomes.  Training district MoEVT officials in IAI implementation, in addition to monitoring and evaluation. By the end of RISE, ECD opportunities had been provided to over 35,000 Zanzibari children,24 and the team had trained 809 formal teachers and non-formal facilitators in IAI pedagogy, classroom management and other key areas. The project also procured over 28,000 teaching and learning materials, ranging from portable media players to slates, manipulatives, and print materials. All print materials were developed by the cadre of MoEVT educators who were seconded as RISE staff and now compose the eLearning Division. The subsequent ZTUR Project was designed to further the ECD work started under RISE by creating a comprehensive preschool teacher certification program (through distance learning) to be implemented by MoEVT. The intention of the certification program (the Early Childhood Advancement Certification Program, or ECACP) is to provide in-service teachers with critical knowledge, skills, and practice through a distance learning platform administered through their local Teacher Centers, which are the in-service training centers located in each district. The ECACP program consists of audio, video and print materials that guide teachers through ECD instructional theory and practice, and support them as they actively apply their knowledge in their classrooms. The specific contents of the ECACP program are: a) a series of IAI programs, including activities in reading, math and science that embody best-practices, directed at teachers for use in their classrooms; b) a series of video-based self-directed learning modules (SDLM) that cover six key content areas25 and integrate theory as well as practice; c) print materials that accompany the teacher audio and video programs; and d) workbooks that guide teachers’ independent studies, cluster meetings and workshops. ZTUR also conducted a Zanzibar-wide needs assessment with stakeholders across Zanzibar identifying the gaps and opportunities for implementing successful preschool programming.26 24 Over 35,000 unique learners were reached. However, as some learners received multiple years of intervention (preschool, Standard 1 and Standard 2) the total of aggregated learners is 44,843. 25 These six key content areas are: child development; interpersonal communication and classroom management; play and early childhood curriculum; building and using teaching and learning materials in language, mathematics and science; assessment and reporting; and learners with special needs. 6 When the RISE and ZTUR Projects were handed over to MoEVT in March of 2011, the intention was to leave behind a sustainable ECD program. The leave-behinds included: a trained team of early grade and distance-learning experts, a community ECD model that could be sustained and replicated across the islands, and the foundation for an ECD teacher certification program that could be expanded upon, and implemented, by the MoEVT. This evaluation addresses the extent to which the goal of sustainability was met, and the extent to which lasting effects of RISE and ZTUR programming remain in the system. 26 Downie, Bruce and Peter Mwaura. (January, 2010) Building Capacity for Pre-School Teaching: A Status Assessment and Recommendations for Teacher Training Programming. Reported was commissioned by the ZTUR project, with direct support from USAID. 7 METHODOLOGY This study was designed as a follow on to and expansion of the RISE Project’s 2008 evaluation, which explored the effects of the intervention on participating students. The 2014 study tracks students from the 2008 sample to explore whether they have continued to enjoy a performance advantage over time, and adds an institutional evaluation of the extent to which the RISE and ZTUR programs have been sustained, integrated, and expanded under MoEVT. 2008 EVALUATION RISE’s 2008 evaluation was a quasi-experimental study measuring the learning gains of Standard 1 learners who participated in the RISE activities in formal classrooms or non-formal Tucheze Tujifunze (TuTu) centers. It tested the hypothesis that the sample of RISE students would perform better on basic competency tests than a group of comparison students, as a result of the RISE intervention. The comparison group had similar characteristics, and were attending formal schools at a Standard 1 level, but they had not been exposed to RISE’s IAI programs. The study also analyzed the difference in RISE students’ performance according to the setting in which they received the intervention. In 2008, three different models were used to deliver Standard 1 Tucheze Tujifunze programming to students in Zanzibar: 1. Non-Formal model: In remote communities, due to a lack of government schools, learners received preschool and Standard 1 Tucheze Tujifunze lessons in non-formal centers, which ranged in structure from non-permanent (i.e. under a tree) to permanent (i.e. local religious school classroom); 2. Formal model: Learners received Standard 1 Tucheze Tujifunze lessons at the formal government schools that they were already attending; 3. Combination model: Learners in this group received two years of exposure to Tucheze Tujifunze lessons; first at non-formal TuTu centers as preschool learners in 2007, and later in 2008 as formal Standard 1 learners who continued to attend Tucheze Tujifunze lessons at non-formal centers after their formal school day. The evaluation results showed that RISE’s learner beneficiaries outperformed comparison group learners by 7.5 points out of 75 total points (or 10%) overall on the Standard One-level assessment.27 Analysis of pre- and post-tests showed that the greatest subject gains were found in Kiswahili. Progress was also evident in English and math results, where scores had risen by about 2 and 2.5 points from pre-test to post-test, respectively, for RISE learners, over and above gains made by the comparison group learners. As for delivery models, learners in the formal 27 These findings were confirmed by a positive program effect size of 0.55, which falls in the medium range. In other words, the mean of the treatment (IRI) group was at the 76th percentile of the comparison group. 8 schools (Formal Model) had the greatest gains, followed by the RISE non-formal learners (Non￾formal Model); both groups had significant gains relative to the comparison group. 2014 POST-PROJECT EVALUATION This post-project evaluation expands on the 2008 study. The 2014 study was designed to assess the degree to which project transfer to the MoEVT was successful, and to analyze any observable long-term advantages in learning outcomes among learners that participated in RISE activities when they were in Standard 1. This evaluation was conducted three years after the RISE and ZTUR Projects were handed over to MoEVT Zanzibar and addressed the following questions: 1. To what extent have the ZTUR and RISE Project activities been sustained or expanded under the leadership of MoEVT after the projects closed (March 2011)? 2. Do learners that participated in IAI (Tucheze Tujifunze) programming perform higher at the end of their primary cycle (Standard 7) than those that did not participate? The study used a mixed-method design, namely a process (qualitative) and impact evaluation (quantitative). The process evaluation looked at what has been sustained, how it has been sustained, and what challenges and successes MoEVT has experienced in attempting to further programming since 2011. Interviews and surveys were conducted between March and June 2014 with 94 individuals, including MoEVT officials from the central and district offices, Teacher Center staff, head teachers of RISE intervention schools, teachers that were trained in Tucheze Tujifunze programming (formal and non-formal), and learners that listened to the programs. The quantitative analysis looked at Standard 7 (equivalent to grade 7 in the United States) learner performance in literacy (Kiswahili and English) and math. The test was developed by the evaluation team based on an existing government exam. It was piloted in April 2014, and between May and June of 2014 the testing was conducted. The same sample of Standard 1 learners who took part in the 2008 evaluation was used for the 2014 data collection, at which point the learners were at the end of their primary cycle (ranging from Standard 4 to 7). Of the total 1,543 learners from the 2008 sample, 904 (or 58.6%) were tested in 2014; including 552 from the RISE group (Formal, Nonformal and Combination Models) and 352 from the comparison group (learners in a district with similar demographics that did not have direct exposure to the Tucheze Tujifunze programming). SAMPLE Respondents for the structured key informant interviews (designed to address evaluation question 1 on the institutionalization of the program within the MoEVT) are listed in the table below by type (including MoEVT division). Respondents were selected to represent the range of actors and stakeholders involved in RISE and ZTUR programming. MoEVT management staff and 9 all Teacher Center staff in the two RISE target districts were interviewed, as were head teachers (principals) from all treatment schools in the sample, and a few of the comparison group schools. Teachers at the test schools and non-formal Mentors who were still teaching Standard 1 were also interviewed from a random sample. Students were selected randomly from among the sampled schools on the two islands. TABLE 1. SAMPLE OF BENEFICIARIES WHO PARTICIPATED IN STRUCTURED INTERVIEWS (N=94) Division/ Type of respondent Protocols (Tools) Location Sample Central, District, MoEVT staff (management and ECD Division)  Structured Interviews  Unguja (Central)  Pemba  6 officials/staff  6 officials/staff eLearning Department staff (former RISE/ZTUR staff)  Focus Group Discussion  Questionnaire  Central Vuga Road Office  2 staff (director & head of M&E) Teacher Center staff (tasked with implementing the certification program)  Structured Interviews  Teacher Center staff in Unguja  Teacher Center staff in Pemba  2 in Unguja;  3 in Pemba Teachers and head teachers from Tucheze Tujifunze schools, comparison schools and non￾formal TuTu centers  Questionnaires  Micheweni Pemba  North A Unguja (all from form RISE supported schools)  24 head teachers (12 Unguja; 12 Pemba)  17 teacher (8 Unguja; 9 Pemba) TuTu Center mentors (facilitators)  Structured Interviews  Micheweni Pemba  North A Unguja  10 mentors (8 Pemba, 2 Unguja) Former RISE learners (participated in TuTu schools/centers in 2008)  Structured Interviews  Micheweni Pemba  North A Unguja  24 total learners (10 Unguja; 14 Pemba) To answer the second evaluation question (on learning outcomes), learners who were assessed against Standard 1 basic competencies in 2008 were tested again in 2014 using a standard equivalency exam, which was based on an official Standard 7 National exam paper. The 2014 sample was nearly evenly divided by island, with 47% of assessed learners from Pemba and 52% from Unguja. Approximately 59% of students from the original sample were found and included in this evaluation. Slightly more students from the comparison group (63.4%) were found and included in this study than the intervention group (55.9%). Within the intervention group, fewer students from the RISE non-formal and combination intervention models could be located than from the formal government school model. Consequently, the study team made a decision to combine students from RISE in non-formal and combination models into a “mixed” group for the 2014 analysis. Table 2 shows the description of the 2008 sample and the 2014 sample. 10 TABLE 2. SAMPLE BY STUDY GROUP AND ISLAND ISLAND Year Comparison group RISE formal RISE non￾formal RISE combination TOTAL Pemba 2008 261 252 141 110 764 2014 170 124 132 (“mixed”) 426 Unguja 2008 294 274 128 83 779 2014 182 167 129 (“mixed”) 478 Total 2008 555 526 269 193 1543 2014 352 291 261 (“mixed”) 904 DATA COLLECTION (PROCESS AND TOOLS) Qualitative data were collected by an independent Zanzibari researcher who had no prior knowledge of or relationship to the RISE and ZTUR projects. Structured interview forms, a focus group protocol, and an informant questionnaire were developed specifically for these activities. The administration of the learner tests in 2014 was managed by the same team of MoEVT officials and eLearning staff as in the 2008 evaluation. Whereas the Standard 1 test was administered in a combination of oral and written formats in 2008 to accommodate the age￾level of the learners, only a written test, using the format of the national Standard 7 exam, was administered in 2014. The 2014 testing was overseen by seven MoEVT staff from a mixture of six departments, all of which collaborate with the eLearning Division. These departments included:  Department of ICT in Education  Department of Pre-School and Primary Education  Zanzibar Institute of Education (curriculum and materials content development)  Department of Teacher Training  Central Library  District Education Offices 11 The 2014 assessors attended a two-day training, facilitated by the former ZTUR Director/Chief of Party and the current eLearning Division Monitoring and Evaluation Officer, who coordinated the 2008 evaluation activities. DATA ANALYSIS Data from in-depth interviews were analyzed using principles of content analysis28 and grounded theory.29 During the first stage of data analysis, all data were coded according to pre￾determined questions. The analysis began with some initial codes (based on the well-tested semi-structured protocol employed for the study), which were revised throughout the analysis, and each transcription was marked with the appropriate codes. However, whenever relevant information was found that was not congruent with any of the existing codes, a new code was added. The central higher-order codes remained the same throughout the analysis process, and only lower-order codes went through fine-tuning. Only students who were tested in both 2008 and 2014 were included in the learner assessment data analysis. All data analyses were disaggregated by sex and study group (RISE formal, RISE mixed (non-formal), and comparison group). Assessment data were analyzed utilizing standard statistical methods. Univariate and multivariate statistical analyses were used for different analytical purposes. Central tendency analyses (mean, median) were used for continuous demographic variables. Comparison of means statistical tests (independent samples t-test) were conducted on the results between different study groups, as well as between assessment results of males and females. Bivariate statistical analyses (e.g., correlations) were conducted to examine the relationship between different variables. Multivariate analyses (regression) were used to determine relationships between variables. The null hypothesis is that there is no significance in gain score between the baseline and the subsequent measurements. The probability that the null hypothesis is true (the p-value) was determined on the basis of the t score. Finally, the p-value was compared to the predetermined .05 significance level. LIMITATIONS Quasi-experimental design: In 2008, it was not possible to randomize learners into participants and non-participants as the target districts were identified prior to the implementation of RISE activities and the intervention included all schools in the two target districts. Therefore, the strength of attribution of observed outcomes to the programming is somewhat reduced due to the selection bias. The comparison group was comprised of the students from formal schools, which is only directly comparable to the “RISE formal schools” intervention group. 28 A content analysis approach assumes a coding frame based on a set of predefined categories for which evidence is sought in the data. 29 A grounded theory approach assumes that the explanatory framework is developed through the process of analysis rather than based on a predetermined set of categories. 12 Data points: Data were collected in 2008 and again in 2014, with no formal contact by evaluators with this group of learners occurring in the interim. More data would have helped address issues of maturation and identify contextual factors that are likely to have influenced learner performance. Social factors and context: Research indicates that social factors (i.e. parent/household demographics, family’s educational values, etc.) are important in understanding and interpreting the findings of studies such as these. However, it was not possible in either of the data collections to collect household level data, given the limited resources and time constraints. Attrition: Tracking the sample group across time posed a number of challenges. Nearly 59% of learners were recovered in the 2014 sample six years later, but it was not possible to determine what happened to the over 40% of learners that were not traced. The evaluation team was able to identify a few major factors that may have accounted for attrition: 1) movement to other schools, districts or regions; 2) drop-out from school; 3) absent on the day of the testing and 4) accidents/disasters, including a ferry accident in September 2011 that claimed 1000 total lives, including two mentors and at least five children from the 2008 sample. 13 FINDINGS The study’s findings are divided into two sections: 1) Institutional Transfer and Sustainability, and 2) Learner Performance. In the first section, findings are reported based on relevance to the Zanzibar Education Policy and Development Programme and sustainability after the transfer to MoEVT. In the second, the sustained RISE effect on student achievement is examined. 14 FINDINGS: INSTITUTIONAL TRANSFER AND SUSTAINABILITY OF RISE AND ZTUR PROJECTS RELEVANCE TO THE ZANZIBAR EDUCATION POLICY AND ZANZIBAR EDUCATION DEVELOPMENT PROGRAMME (ZEDP) The RISE and ZTUR projects were designed to be directly aligned with the following areas in the Education Policy: 30 TABLE 3: ALIGNMENT OF RISE/ZTUR ACTIVITIES TO POLICY AREAS Education Policy Area RISE or ZTUR Activities (Post-project)  Merging the existing strengths reflected in the government infrastructure and teacher qualification with the strong good practices in community-based schools, especially in modes of delivery and teacher-student interaction  Introduction of the Interactive Audio Instruction methodology  Integrating child-centered classrooms and effective teacher-learner interaction with the Zanzibar learner and teacher curricula through RISE teacher/mentor training and the ZTUR ECACP  Establishment of satellite preschools  Establishment of 180 community (satellite) TuTu Centers (with preschool and Standard 1 classes)  Establishment of a community preschool model  Improving ECD training  Development of the ECACP ECD training for in￾service teachers  Ensuring that the community fully participates in the management of the school with regard to both enhancement of curriculum content and management of resources  Prior to starting up community preschools (TuTu Centers), holding a community mobilization meeting to arrive at a joint management plan for establishing and supporting the center  Training of school management committees (SMCs) to ensure they are supporting TuTu Centers in addition to the formal schools; ensuring presence at facilitator and teacher trainings  Creating programs to acquaint children with ICT as early as possible  Creating audio programs to orient educators and learners to radio and portable media player technology While RISE and ZTUR were not involved in construction of formal preschool classrooms and playgrounds, school feeding, or dental health programming (the main activities outlined in ZEDP), RISE did support ZEDP’s efforts to “conduct awareness campaigns and provide incentives 30 Ministry of Education and Vocational Training, Revolutionary Government of Zanzibar. (2006). Education Policy. 15 for establishing preschools”31 through its community mobilization meetings and by working with communities to identify volunteers and space for establishing non￾formal preschools. In addition, during the development of the ECACP training program, ZTUR also addressed “criteria for preschools” in the ECD Working Group. This Working Group was established by ZTUR staff in partnership with the Aga Khan Foundation’s staff from the Zanzibar Madrassa Resource Programme. Other members of the working group were the Ministry of Health and Social Welfare; Ministry of Youth, Employment, Women and Children Development; Save the Children; the Zanzibar Association for People with Disability; and Voluntary Services Overseas. SUSTAINABILITY OF RISE AND ZTUR ACTIVITIES AFTER TRANSFER TO MOEVT The greatest accomplishment in sustaining the efforts of the RISE and ZTUR Projects was establishing the eLearning Division within the MoEVT. The projects’ space, team and equipment became the eLearning Division under the Department of Information and Communication Technology (ICT) in Education.32 The Department of ICT in Education is one of eight MoEVT departments and was created in 2010/2011 to integrate ICT into basic education in Zanzibar. According to the MoEVT, the establishment of this department was a key step in the implementation of the 2006 Education Policy goal of employing ICT across the education system. The eLearning Division’s mandate is to:  train the staff of all divisions and sections in ICT, including schools, teaching colleges, and libraries;  integrate ICT into teaching and instruction;  conduct seminars and training (long and short-term) on how to use ICT in Ministry work (i.e. communications and management); and  ensure that MoEVT divisions are working together in the areas of ICT33 31 Ministry of Education and Vocational Training, Revolutionary Government of Zanzibar. (2007). Zanzibar Education Development Program 2008-2016, p. 60. 32Initially called the Training Division, eLearning became the official name in 2012. 33 Summarized from website. MoEVT, Zanzibar. (2014). Accessed on November 2, 2014. http://www.moez.go.tz/?cq=dept&dept=10 “I think the fact that MoEVT continue to support and manage this [RISE] project is an indication that there was a very good relationship between the Ministry and RISE. The Ministry has allowed its teachers to continue using tools, reading materials and other skills acquired from RISE in their day to day teaching and learning.” eLearning District Coordinator, Micheweni, Pemba “The most important and effective aspect of this [RISE and ZTUR] project was its participatory approach, by which community members were directly engaged in searching for areas to be used as Learning Centers as well as identifying mentors that come from the communities themselves.” District Education Officer (DEO), North A, Unguja 16 The table and text that follow provide additional detail on the extent to which the key project components of RISE and ZTUR have been institutionalized and sustained 3 years post-project, based on interviews with key staff in the eLearning Division and the Director of the Department of ICT in Education at the MoEVT. TABLE 4: KEY PROJECT COMPONENTS AND CURRENT POST-PROJECT STATUS Key project components March 2010/11 (end of RISE and ZTUR) April 2014 I. Policy and Ministry Level 1. Establish an eLearning division under MoEVT, with a trained team - RISE/ZTUR team of 11 MoEVT seconded technical staff were trained and became the eLearning Division (after initially being called the ICT Division) - eLearning Division currently has 14 staff (12 worked with RISE/ZTUR in some capacity). 2. Create synergy with other divisions in MoEVT - eLearning was collaborating with 4 divisions: 1) Teacher Training, 2) Preschool, 3) Primary, and 4) Department of Curriculum (now Zanzibar Institute of Education) - eLearning is collaborating with 4 original divisions plus 3 new divisions: 5) Technical Division under Department of ICT in Education, 6) Inspectorate, and 7) Inclusive Education Unit 3. Develop the Early Childhood Advancement Certificate Program (ECACP) - first module developed, materials distributed to Teacher Centers, and Teacher Center staff trained - MoEVT had not yet figured out the logistics of program implementation - certification training started in 2014 with trainers of ECD teachers. 350 formal teachers in 10 districts are about to start 2 years (includes field practice) of ECACP; 30 Trainers have been trained. 4. Establish Tucheze Tujifunze centers & schools - 180 non-formal TuTu centers established - Centers were supposed to cap at 25-30 students - 264 formal Standard 1 & 2 classrooms using TuTu materials - 179 out of 180 original non-formal TuTu Centers still open - 51 new centers being opened in 2 new districts (with GPE support) - Issues of up to 80 learners/new center - no TuTu materials are currently being used in formal classrooms 5. Set up a mentor payment system - mentors paid equivalent of $20 a month - payments transferred to MoEVT budget - mentor payment increased by 33% - payments still in MoEVT budget II. Training 6a. Train TuTu Center facilitators (non￾formal mentors) in TuTu programming - RISE (Tucheze Tujifunze) training program included initial training and 2 refreshers each year - 430 mentors trained - two trainings for former mentors have taken place since project closed - two trainings for new mentors in 51 new (25 in Pemba) centers started in March 2014 (using GPE funds) 17 6b. Train formal school teachers in TuTu programming - 273 teachers trained - no additional training 6c. Train school management committees (SMCs) and officials - 138 SMC members trained (in original TuTu Centers) - 182 head teachers and MoEVT officials trained - no training for committees has taken place and no follow-up support in pedagogy and instruction III. Development and Delivery of Materials 7. Audio and video programs (for learners and educators) produced - 3 Tucheze Tujifunze series developed (preschool to Standard 2) - first of six ECACP training videos produced; 3 others designed (4 videos produced) - produced 2 IAI teacher series - produced English Language Instruction for Standard 1 teachers - play-to-learn was catching on - preschool series #2 in process - two preschool teacher training modules produced (using UNICEF funds) - fourth module is being developed and modules 5/6 to be ready by 2016 - Tucheze Tujifunze (play-to-learn) pedagogy has become well￾accepted by education community and parents 9. Broadcasting - 3 series broadcast 3 days/week under state-run station; no direct charge - programs broadcast for no direct charge until December 2013 - in Feb 2014 station was semi￾privatized and stopped broadcasting; programs were off air from 11 February - 28 May 2014. - Airing programs resumed from 29 May 2014. All programs (including preschool and primary) are now being aired by ZBC. 10. Print and technology materials developed and distributed - 18,892 print materials for classroom packages distributed - 5,637 print materials for ECACP packages distributed - 1,752 wind-up, solar radios and portable audio players (with integrated Mp3s) for schools and TuTu Centers distributed - 14 technology and materials kits for Teacher Centers - no materials printed or distributed to old sites - 51 copies of Mentor’s Guides and Learner Materials to be distributed to new centers (through GPE) IV. Monitoring and Evaluation 18 11. Monitoring - 6 coordinators overseeing two districts; paid by the project but employed by MoEVT. - 1 M&E coordinator in central project offices - each center and school was to be monitored each quarter per a set schedule - 6 coordinators overseeing four districts (4 in Unguja, 2 in Pemba); coordinators are employed and paid by MoEVT. - 2 M&E coordinators in central project offices - not enough resources to visit centers and schools regularly, so actual monitoring is limited. 13. Evaluation - impact evaluation of grade 1 student learning conducted in 2008 - no further evaluations conducted POLICY AND MINISTRY LEVEL EFFECTS 1. Establish an eLearning Division under MoEVT, with a trained team The RISE and ZTUR teams officially became the eLearning Division in 2010/11. Currently there are 14 technical team members and one administration staff person in the Division. The Head of the eLearning Division was previously the RISE and ZTUR Lead Materials Development Specialist/Training Coordinator, and assumed leadership of the division at the close of the projects. She reports directly to the Director of the Department of ICT in Education, MoEVT Zanzibar. Of the 14 current technical staff, nine were part of the original RISE team as MoEVT-seconded trainers, scriptwriters, and studio team members. Of the remaining five, three joined the RISE/ZTUR team in varying capacities in 2008, and officially joined the eLearning Division in 2011. Before joining the division officially they were: a former RISE/ZTUR video teacher, a Ministry ICT staff member trained in RISE studio production, and a Standard 1 teacher from the RISE IAI pilot school (the formative evaluation site for the IAI programs as they were being developed). The only two members new to the team are a former primary school teacher who was trained in writing radio scripts by the eLearning Division, and a staff member from the ICT in Education Division who has been trained in studio production. Of the original RISE/ZTUR-MoEVT team, only one staff member, a studio producer, has transferred to a different MoEVT department. In the interviews conducted with the eLearning Division staff for this evaluation, all identified a need to have more professional development in distance and eLearning now that their scope of work has been clearly defined. The eLearning division is tasked with all technology and professional development for MoEVT, but has limited knowledge outside of development of print, audio and video materials. They also see the immediate need for integrating new technologies, specifically mobile phones, into their division, in order to share content and training materials with schools and teachers and to better monitor schools. As the team has no 19 prior experience with mobile phone technology for technical and M&E purposes, this is a critical area of needed support. As the eLearning Division moves forward, their planned activities continue to build on the foundational activities established under the RISE and ZTUR Projects. According to the Director of eLearning (Interview, 12 May 2014), 2014-2015 plans for the Division include:  Developing an additional year of Tucheze Tujifunze preschool programs  Continuing to produce the ECACP video models: ZTUR completed the first of six Self￾directed Learning Modules (SDLM) with four completed videos; UNICEF funded modules 2 and 3; and the 4th module is currently being developed. The last two modules are to be completed by 2016;  Monitoring and establishing Tucheze Tujifunze Centers: The Division will monitor the 179 existing TuTu centers, and establish 51 new centers in two new districts: Unguja’s North B and Pemba’s Mkoani. Establishment of new centers includes community mobilization, mentor (facilitator) trainings, sensitization of head teachers and officials, material printing and distribution, procurement of radios, and other activities. Support for this activity is coming from the Global Partnership for Education (GPE)  Developing math and science programming: In the future, the team will be developing math and science programs for primary school and eventually for secondary and above. 2. Create synergy with other divisions in MoEVT The Department of ICT in Education meets weekly with all four divisions. Cross-division meetings hosted by MoEVT are held twice a year. The eLearning Division is expected to collaborate with the following five MoEVT divisions/units: 1. Teacher Training (for training of facilitators and teachers and pedagogy); 2. Preschool (in the areas of materials and planning centers); 3. Primary (in the areas of materials and trainings); 4. Zanzibar Institute of Education (for the content of teacher and learner programs and curriculum); 5. Technical Division under the Department of ICT in education; 6. Chief Inspectorate Office (less direct collaboration); and 7. Inclusive Education Unit (for content to serve students with special needs) One change since 2011 is that Inspectors SMCs, which are equivalent to parent teacher associations, were officially made responsible for overseeing the new TuTu Centers. In accordance with this mandate, the eLearning Division has been trying to integrate inspectors into current and future trainings to ensure that they have more knowledge and ownership of the TuTu Centers. However, it was reported during this evaluation that inspectors have not yet begun monitoring TuTu Centers as part of their regular work load. In addition to new collaborations with the Inspectorate, the eLearning Division has also begun inviting the Inclusive Education Unit (IEU) to participate in current and future trainings. In 20 “I would like to call upon all teachers in preschools, or those who will teach preschools in future, to take advantage of building their capacity through in-service training ECCP [ECACP]. I personally promise that those teachers passing through this training program will be duly recognized.” Honorable Ramadhan A. Shabaan Minister, MoEVT (2011) theory, the IEU will also help with future content development. However, its participation until now has been limited because clear activities and guidelines for collaboration are not yet in place. 3. Develop the Early Childhood Advancement Certificate Program (ECACP) One of the main concerns in the ZEDP34 regarding the implementation of the pre-primary policy was a lack of trained preschool teachers. Preparing an adequate supply of trained teachers is one of the critical steps in ensuring that compulsory preschool education across Zanzibar is provided to all children. As the ZEDP plan describes: Enrollment at the preschool level of education is still low and teacher quality is inadequate. One area that needs special attention is childhood care and development, a concept that encompasses all children, from nine months to the time these children enter primary schools. Interventions in this regard would include developing a holistic approach to cover care, nutrition, health and education. (MoEVT, 2007, 12) The original intent of developing the ECACP under ZTUR was to help MoEVT address the shortage of trained preschool teachers35. The ECACP uses a distance-learning model to reach in-service primary school teachers throughout the islands with the ultimate goal of providing them with the additional skills and credentials needed to teach a preschool curriculum. The distance-learning methods consist of audio (Chezesha Ufundisha IAI programs), video (Self-Directed Learning Modules) and print materials, integrated with short-term and intensive trainings and peer support/mentoring through cluster meetings. The coursework is facilitated and overseen by MoEVT’s ten Teacher Centers, which are the in-service training centers located in each district. Although the majority of the ECACP design and materials was developed under ZTUR, the certification program was rolled out in seven Teacher Centers early in 2014. 36 Currently, support is being provided by the GPE to implement ECACP across Zanzibar. 34 The term preschool is used in this document, as there are few activities and guidance provided for children aged birth to three in the pre-primary section of the Education Policy and ZEDP. 35 Another advancement made in the last year is the decision to pay MoEVT-employed preschool teachers the same as primary school teachers. In 2010-11, at the time ZTUR was being implemented, preschool teachers were being paid less than the primary school teachers, which was a disincentive for joining the preschool teaching force. 36Centers include: Kiembe, Samaki, Bububu, Mkwajuni, Dunga, Kitungoni, Mizingani-Mkoani. 21 The ECACP is structured as follows:  Step 1: Complete the Aga Khan Foundation-Zanzibar Madrassa Resource Program preschool teacher training (2 years) OR Obtain a primary school teaching certificate  Step 2: Complete the ECACP distance training, which includes the audio trainings, video modules (SDLM), and independent and cluster group work (completion varies from 1.5 to 2 years)  Step 3: Receive Certificate  Step 4: Obtain appointment as a Preschool Teacher or Trainer of Trainers (includes individuals who are under the Teacher Training Division) ECACP consists of 607 core hours, to be completed over one or two years, depending on the pace of the teacher participant. Teacher participants include: a) Certified and in-service primary teachers who will teach preschool; and b) In-service preschool teachers who require certification. At the time of this study, there were 352 teachers from all ten districts going through step 1 and 26 trainers had been trained by the e-Learning Division and Aga Khan Foundation. All 26 trainers had received ECD certificates from Aga Khan Foundation. Over 700 teacher participants are expected to benefit from the ECACP, starting in 2014. Understandably it took a number of years for MoEVT to roll ECACP out, as they had to identify preschool advisors for the Teacher Centers, approve the curriculum and program requirements, and come to agreements across all the concerned divisions and departments. However, momentum and awareness now need to be rekindled, as nearly three years have passed between the initial Teacher Center staff’s ECACP orientation in 2011 and the roll-out of the official training (2014-2015). It was evident during the interviews with Teacher Center staff that not all districts have the same understanding and information about ECACP, and some effort needs to be made to get everyone on the same page. Additional follow up in 2015, when ECACP is fully rolled-out, would be useful for assessing whether the ZTUR-distributed materials and equipment are indeed being used for the training. 4. Establish Tucheze Tujifunze (TuTu) centers and schools Under RISE, 180 non-formal TuTu Centers were established in communities across two of Zanzibar’s ten districts: Micheweni (in Pemba) and North A (in Unguja). The centers reached over 20,000 non-formal preschool and Standard 1 learners over the course of the project. One of the first steps in establishing the centers was mobilizing the communities to make sure they were 1) ready to accept the Tucheze Tujifunze preschool and Standard 1 out-of-school model and 2) willing to provide structures for the non-formal centers, volunteers to serve on the SMCs, and mentors to lead the classes. RISE did not provide any direct financial resources for building centers, but did help provide communities with a design for simple cost-effective structures and linked some of them to local hotels and donors that could provide materials. Even so, one of the major concerns during the duration of RISE was the physical infrastructure of the preschool classes. Approximately 8% of the non-formal TuTu Centers (15 out of 180) were 22 situated in non-permanent structures in open environments, which was especially problematic on rainy days when the spaces were not usable. As of 2014, 179 of the original 180 centers are still in existence, and some efforts have been made to help build more permanent spaces for the centers that were in non-permanent structures. Post-RISE, the eLearning Division is also in the process of opening up an additional 51 centers in two new districts (Mkoani in Pemba and North B in Unguja) with the support of the GPE. GPE will fund 40% of preschool classroom construction (roofing and some cement), mentor training , and printing and distribution of learning and teaching materials. The communities are expected to contribute the remaining 60% of the costs of classroom construction (materials and labor). MoEVT will be responsible for community mobilization and monthly stipends for mentors. Although the Education Policy sets the goal of ensuring that every primary school has a preschool, this goal is still far from being achieved, and even those that do exist are often too far away for children to reach. Therefore the need for non-formal preschools remains high. Of the 24 formal school head teachers surveyed for this evaluation, only 30% reported actually having preschool classrooms at their schools. Although this sample is not representative of Zanzibar, it confirms that more funding for preschool infrastructure and programming is critical. While the Tu Tu Centers remain strong, the Tucheze Tujifunze formal schools have not been as successfully sustained. Few of the original 136 Standard 1 and 2 classrooms (in 65 schools) receiving technology, print materials, training and other support under RISE are still using the resources as intended. During RISE, Tucheze Tujifunze programming had reached over 20,000 Standard 1 and 2 learners. Interestingly, although the staff at the eLearning Division reported that no schools were currently listening to the TuTu programs, 33% of the head teachers surveyed reported that their learners were actually listening, even if not regularly. And of the schools listening to the TuTu programs, all but one rated the programs as “very important” to their efforts (the remaining school rated the program as “somewhat important”). The major reason that schools are not listening to programs is a breakdown in technology and broadcasting. Of the schools that were not listening to the programs, over 75% attributed not listening to problems with the operation of the radios. 100% of the surveyed schools had two or more radios and the majority (65%) of schools had at least 5 radios. However, only 50% of these radios were reported to be functioning. 23 FIGURE 1. PERCENTAGE OF SCHOOLS WITH TWO OR MORE RADIOS (N=24) In the last wave of distribution of by RISE in 2011, the company manufacturing the radios had gone through major business and leadership changes, which affected the release date and quality of the wind-up, solar charged, portable audio player devices (with integrated Mp3). The shipment of the 1,000 devices was delayed over seven months because of manufacturing and customs issues, and as a consequence devices were distributed the week before the projects were transferred to MoEVT. Thus, there was no time for the ZTUR project team to conduct quality assurance and testing of the devices. Although some schools have figured out ways to fix their radios on their own since 2011, the quality of the last wave of audio devices has been a major challenge to sustaining listenership. Furthermore, MoEVT has not provided on-going maintenance or IT support to address these issues. As of May 2014, MoEVT was working on procuring more radios for new centers, but had no plans for fixing the old radios or putting in place a technology maintenance plan. In addition to problems with technology, another notable issue affecting the utilization of programming in formal schools was the challenge of fitting the Tucheze Tujifunze programming into the primary school curriculum and timetable. According to one head teacher, the primary school timetable has gotten denser, but there is no strategy for re-integrating the programs into the new schedule. This head teacher also noted that the USAID-funded Tanzania 21st Century (TZ21) Project, which introduced new content into the primary school curriculum, was an additional challenge to the timetable. TZ21 curriculum changes were not coordinated with other programming, such as Tucheze Tujifunze, when they were developed.37 As one head teacher suggested, “MoEVT needs to be more top-down and give the official word to schools that they should be using the [TuTu] programs.” As the curriculum moves from five to eight subjects by 2016, and there are new projects targeting early grades with new content, it is critical that MoEVT look deeply at the timetable issue and make recommendations to schools about how to structure and integrate traditional content and ICT programming. 37 The TZ21 project provided technical assistance to the Tanzanian MoEVT in Zanzibar with comprehensive, localized, interactive digital student learning materials, called “e-Content”, which were designed and implemented in lower primary schools, along with corresponding teacher training materials to facilitate application of the e-Content through child-centered learning. 13.0% 4.3% 17.4% 30.4% 17.4% 13.0% 4.3% 2 radios 3 radios 4 radios 5 radios 6 radios 7 radios 8 radios 24 During the first year that Tucheze Tujifunze programs were on air (2007-2008), there were challenges in some communities in Unguja, and a few in Pemba, who were reluctant to accept the play-to-learn model and ECD pedagogy that encourages interactive and playful learning, rhythms, and gender-mixed classrooms. During the first year of implementation, community and religious members also expressed concern that the programs did not explicitly involve religious teachings or references, and instead involved rhythms or music which could be considered religiously unacceptable. Some critics also felt that play-based learning was not effective—they believed that preschoolers should focus on reading and writing. Religious and pedagogical objections to the programming were less of a concern in 2010/11 when the projects closed, likely due to the extensive efforts by the RISE and MoEVT team in sensitizing community members to the importance of ECD and working with religious leaders to vet the programming. However, one head teacher interviewed in 2014 noted that religious concerns of parents and the community about Tucheze Tujifunze programming have continued to be a challenge. According to this head teacher, parents feel that “the programs are missing religion. Islam is not in them. It would be good for the children who listen to the programs to have religious topics integrated, for the programs to be acceptable.” Another head teacher identified the need for on-going mobilization of the community to ensure that the importance of ECD and Tucheze Tujifunze programming is clear to caregivers. The fact that head teachers are still reporting issues with community acceptance of ECD and the Tucheze Tujifunze programming signals the need for ongoing community mobilization and sensitization when expanding the programming. During RISE, it became clear that many teachers and head teachers were not clear on the Zanzibar curriculum, which includes a coast ecology component in Standard 1 and 2; nor did they have access to any of the materials. Comments by some mentors interviewed in 2014 show that this trend has continued, as they commented that marine life content in the RISE programs was not suitable for their communities, as they were not directly living on the sea38 . 5. Set up a TuTu Center mentor payment system One particular success of the RISE/ZTUR programming in the last year of implementation was working with MoEVT to integrate payments for TuTu Center mentors into the Ministry’s annual budget. In July 2010, MoEVT assumed payment of mentors, and since that time the eLearning Division has systematically managed the monthly payments to mentors. Of the ten mentors surveyed for this evaluation, 100% reported receiving their payments on time. Between 2006 and 2011, mentors were paid the equivalent of $20 a month for a volunteer stipend, which was in-line with MoEVT’s policy for paying alternative education mentors and facilitators. In July 2013 the amount was increased from TSh 20,000 to TSh 30,000. Although the new amount is 38 It should be noted that, given the size of Zanzibar, no community can be more than 43 kilometers away from the sea. Unguja is 85 Kilometers (kms) long and 39 kms wide, while Pemba is 67 Kms long and 23 Kms wide. Zanzinet, “About Zanzibar,” http://www.zanzinet.org/zanzibar/visiwa.html 25 still equivalent in 2014 to $20 due to inflation, the augmentation of mentor remuneration is evidence of the division’s commitment to providing continued support to its mentors.39 TRAINING 6. Train Mentors, Teachers, Head Teachers, SMCs and MoEVT officials Under RISE, the team provided an initial training of three to four days for new TuTu Center mentors, followed by two follow up trainings of one to two days each. Every year there were refresher trainings for returning mentors. RISE also held annual two-day trainings for new Standard 1 and 2 teachers, and a one-day follow-up training. Ever year of the project, refresher trainings were provided. In addition to training the mentors and trainers facilitating the programs, RISE also trained head teachers, MoEVT officials (Teacher Center advisors and District leaders), and SMCs, to ensure that there was continued management and pedagogical support available to mentors and teachers. Over the life of the project, RISE trained 430 mentors, 273 Standard 1 and 2 teachers, 182 head teachers and MoEVT officials, and 138 members of SMCs. This continuous training model has not been sustained, largely because of a lack of allocated funds for training. However, according to the eLearning Division respondents, two trainings took place in early 2014 to train the new mentors for the 51 newly established centers. Funds were provided by GPE. By 2016, the GPE has the following plans: open 120 new centers (60 Mkoani, 60 North B), provide materials and training to 240 mentors, and record 200 preschool programs, with hope that MoEVT will broadcast them, and buy 120 mp3 radios for the 129 new centers. Sustaining access and quality of Tucheze Tujifunze programming is a challenge with high teacher turnover. Formal schools experience fairly high teacher turnover as teachers are consistently trying to move up to work with older learners (upper elementary or secondary) because the pay is higher for upper grade levels. Another contributing factor to teacher turnover is female teachers marrying and moving to other locations to be with their spouses. For non-formal mentors, major turnover is due to their volunteer status and minimal stipend; as they increase their skills and confidence, they find other opportunities that pay more. As a result, under RISE the team was training a new cohort of formal preschool, Standard 1 and 2 teachers annually to replace those that had moved or left teaching. Consequently, regular trainings and monitoring are critical to ensuring that the quality of implementation remains high. At present, with no 39 In 2013, MoEVT officials reported to EDC that a group of Tucheze Tujifunze mentors converged on the lawn of the central MoEVT offices in Stone Town to demand greater recognition, certification and pay. It is not clear if increases in stipends are related to this event. 26 formal training schedule for existing centers and schools, teachers and mentors have not been consistently and regularly trained or monitored since 2011, and teaching quality was perceived as low by MoEVT staff at all levels and by head teachers who participated in interviews. DEVELOPMENT AND DELIVERY OF MATERIALS 7. Audio and video program development (for learners and educators) Perhaps the most time and resource-intensive activity of RISE and ZTUR, beyond the trainings, was the development of the Tucheze Tujifunze IAI programs and accompanying print materials, and the interactive video programs: RISE’s English Language Training series and ZTUR’s SDLM videos Under RISE, three series of audio programs were produced for preschool, Standard 1 and Standard 2 learners. Tucheze Tujifunze programs were each 30 minutes in duration and covered life skills (environment and health), Kiswahili, English and math. Programs integrated the government curriculum with Zanzibar songs, stories, riddles, games and other creative and cultural based activities. The development process included:  intensive training of 11 scriptwriters, editors, and producers in IAI pedagogy and materials development;  researching the content for each script using the Zanzibar curriculum, and as needed consulting with local experts (health, environmental, child rights, inclusive education, etc.) for the life skills content;  writing of master plans, scope-and-sequences and scripts;  one-on-one peer and editorial reviews of each script;  first recording of the script with child and adult actors, and rhythmic/musical content;  formative evaluation of the programs in one of the pilot classrooms;  revision of the scripts; and  re-recording and editing of the scripts based on the changes made. Chezesha Ufundishe, an IAI teacher’s series, was developed under ZTUR, and focuses on the areas of ECD that are critical for preschool teachers to understand theoretically and practically. These 20 programs guide teachers in the classroom through activities, allowing them to practice new material directly with their students. The series is part of the ECACP curriculum, along with the SDLM modules. 27 The SDLM videos, with accompanying print materials, were created for ECACP to introduce teachers to early childhood theory and pedagogy, and to model effective teaching in the preschool classroom. The first module, Child Development, was fully filmed under ZTUR and includes four 30-minute video trainings with structured activities; each video frames a four-hour training day. These programs target teachers who are already working in preschool classrooms, allowing MoEVT to deploy teachers to preschool classrooms early in the program. Finally, four English Language Training Program videos were developed under RISE, with accompanying teachers guides. These programs guide teachers through an interactive training that tackles the first units of the Standard 1 English curriculum. The aim of the videos was to help teachers improve their comprehension of the English language as well as their ability to teach English content in their classroom through activities that are child centered and use active learning pedagogy. Production of these eight interactive video programs followed the same process as the audio production outlined above, but was more time intensive, as filming of children, teachers and classrooms requires significant planning, coaching and reshooting. Over the nearly four-and-a-half years of the RISE and ZTUR programming, the current staff on the eLearning Division team received considerable professional development and support in production of audio and video programs and accompanying print materials. Fortunately the eLearning team has retained all but one of the initial staff members, which has enabled them to continue producing and developing audio and video programs. At the time of this report, the eLearning Division was writing and producing a second-level of preschool Tucheze Tujifunze programming (the first 20 of 78 programs had master-plans developed under RISE) and was producing the fourth module and accompanying videos in the SDLM sequence. SDLM modules two and three were produced between 2011 and 21013 with the support of UNICEF. 8. Broadcasting Between 2007 and 2014 (January), the state-owned Voice of Tanzania, Radio Zanzibar was broadcasting Tucheze Tujifunze free of charge, as part of its educational and social commitment to its Zanzibar audience. In return, RISE and ZTUR Projects provided the station with broadcasting equipment, staff training, and logistical support to ensure the over 276 programs were aired three times a week for five years. Following the project closures in 2011, the eLearning Division continued to provide weekly support to the station to ensure the programs continued to air. The director of the radio, MoEVT officials, and even the President of Zanzibar advocated for the programs to remain on air free of charge during the RISE and ZTUR Projects, and the eLearning Division was able to maintain this high-level support after the projects closed. In late 2013, the radio station formally underwent restructuring and semi-privatization. The new leadership began demanding payment of more than $3,000 a year, which was beyond the 28 means of the eLearning budget. In consequence, the eLearning Division and MoEVT officials had to begin a new process of negotiations with the station. The President of Zanzibar asked the radio station for a transition period, as he claimed he had been listening to the programs. Although TuTu programs were off air from February to May 2014, broadcast resumed at the end of May. Preschool programs are now being aired by ZBC, but Standard 1 and 2 programs remain off the air. As noted earlier in this report, equipment maintenance poses challenges to continued successful use of the broadcasts. Although over 652 pre-loaded Mp3 radios were distributed to schools and TuTu Centers, it is estimated from the survey of 24 schools that at least 50% of these audio devices were not working for the full 30 minutes of the programming (an 80% estimate for centers is even higher than for schools). With help from the Teacher Center Mkwajuni in Unguja, some radios have been fixed using local solutions, for example, using outside batteries. However, very little to no follow-up on how to use the radios had been conducted by the eLearning team, so it was not clear how many centers and schools are using the pre-loaded programs (as opposed to broadcast). The TuTu Centers have been instructed in the absence of a functioning audio devices and broadcasts to simply follow the Mentor’s Guide, which limits the fidelity of the pedagogy, as well as the quality. In February 2011, USAID financed a radio engineer consultant to conduct an assessment of radio capacity issues and make recommendations to MoEVT on how they may institute a new frequency. However, after the initial distribution of the report no further funding or support was provided to MoEVT to realize the report recommendations.40 Until the broadcast issue and maintenance of audio devices, can be resolved, the long-term sustainability of the Tucheze Tujifunze audio programming is uncertain. 9. Print and technology materials developed and distributed RISE and ZTUR teams produced extensive materials for learners, teachers, Teacher Centers and trainers. These include:  Classroom packages for preschool, Standard 1 and Standard 2  ECACP Student Teacher packages  ECACP Trainer packages  Teacher Center packages A breakdown of the materials by recipients is presented below. 40 Kilimanjaro International. (February 18, 2011). Radio Capacity in Zanzibar. 29 TABLE 5. LEARNING AND TEACHING MATERIALS BREAKDOWN The total number of materials distributed by type is as follows (for a full list of the print materials and technology distributed under the RISE and ZTUR projects, see Appendix 1).  18,892 print materials for classroom packages  5,637 print materials for ECACP packages Learning and Teaching Materials Developed, Produced and Distributed Description Details Classroom Packages for Preschool, Standard 1 and Standard 2 (Tucheze Tujifunze materials) 1. Audio devices (2) with programs: 78 Preschool; 99 Standard 1; 99 Standard 2 2. TuTu Preschool Mentor’s Guide 3. TuTu St. 1 Teacher’s Guide 4. TuTu St. 2 Teacher’s Guide 5. Sounds in the Night (story book) 6. Classroom Posters 7. TuTu Learning Kits ECACP Student Teacher (Participant) Package 1. IRI Teacher’s Guide 2. Teacher’s Introduction to SDLM 3. SDLM: Cognitive Development 4. SDLM: Brain Development 5. SDLM: Language Development 6. SDLM: 6 Aspects of the Child 7. Sounds in the Night (storybook) 8. Technology User’s Guide ECACP Trainer’s Package 1. to 9. Same as above (Teacher’s) 10. IRI Trainers 11. Trainer’s Introduction to SDLM 12. Trainer’s SDLM: Cognitive 13 Trainer’s SDLM: Brain 14. Trainer’s SDLM: Language 15. Trainer’s SDLM: 6 Aspects 16. Equipment User’s Guide 17. Tucheze Tujifunze Songbook 18. Tucheze Tujifunze Songs CD 19. Verification Sheets IRI 20. Verification Sheets SDLM 21. List of ECD Resources-Zanzibar Technology and Materials Packages for Trainer Center 1. Pico Projectors (2) 2. IPod with videos uploaded 3. Solar chargers 4. Itrip (for connecting to frequency) 5. Windup speakers 6. Lifeplayer radio 7. Sounds in the Night (story book) 8. Tucheze Tujifunze Song Book 9. Tucheze Tujifunze Songs CD 10. Posters 11. ELTP (English) Trainer’s Guide 30  1,752 wind-up, solar radios and portable audio players (with integrated Mp3s) for schools and TuTu Centers  14 technology and materials kits for Teacher Centers Although all Teacher Centers surveyed noted that they still have the video kits, and were initially trained, they were largely un-used as the ECACP training is just starting to be rolled out in 2014. As the Teacher Centers have not been actively using the technology there may be some functionality issues in addition to a need to re-train staff how to use the technology. In regards to the print materials, no new materials have been printed and distributed to the existing 179 centers since 2011, only to the new 51 centers. As in any learning setting, print materials need to be regularly printed and distributed each year to replace lost and worn-out texts, which has not been done. According to the questionnaires of formal teachers and TuTu Center mentors, learner workbooks were no longer accessible to students; zero of 17 teachers and only 3 of the 10 mentors had copies. Of the Mentor/Teacher’s Guides, all of the mentors had the preschool guide and 70% still had a Standard 1 Guide, whereas over three-quarters of the teachers reported having their respective guides. All of the mentors reported that they still had the children’s stories and learner kit materials available. MONITORING & EVALUATION 10. Monitoring Prior to 2011, there were six appointed coordinators who oversaw non-formal and formal programming in Micheweni and North A districts. These coordinators were seconded from MoEVT (some part-time, others full-time) and paid by RISE. The coordinators were primarily MoEVT teachers and Teacher Center Subject Advisors. In addition to the coordinators within the district, a central RISE M&E Coordinator was employed to oversee all the district coordinators from the main office. He was seconded from the MoEVT Statistics Department and worked with the Department of Administration and Personnel. The M&E Coordinator was also responsible for coordinating evaluation activities and testing, and was trained extensively under RISE and ZTUR to do basic analysis support and train data collectors. This seven-person monitoring team was responsible for ensuring that each participating RISE center and formal school received monitoring visits each quarter. They were also responsible for ensuring fidelity of implementation in the TuTu Centers and schools, supporting mentors pedagogically and observing their classroom instruction, monitoring student attendance per their enrollment (monthly attendance rate), and 31 helping roll ECACP out. Under RISE and ZTUR, coordinators were given stipends for transport and mobile phone credit, and the lead coordinator was given access to vehicle transport. One of the major challenges since 2011 has been ensuring the on-going monitoring of TuTu Centers and schools. Although there are still six coordinators overseeing TuTu centers and schools in the two districts, all of whom are fully employed by MoEVT, they do not have sufficient resources to visit each center or school quarterly or follow-up on the phone. At the central level, there are now three M&E coordinators in the eLearning Division, which is the addition of one staff member, but they too lack access to a vehicle to monitor centers and schools, beyond providing the basic mentor payments. Without a vehicle, the team relies on public transport or availability of MoEVT vehicles; getting to Pemba is even more difficult as it involves a boat or airplane ride. Therefore M&E coordinators are able to visit Pemba only one to two times a year. As was noted under the training section, without resources and commitment provided for monitoring of centers and schools, the quality of programming will likely decline. 11. Evaluation Until this evaluation was executed, no additional evaluation activities had taken place since the project handover in 2011. However, as the M&E Coordinator was trained to lead testing and evaluation activities under RISE and still holds the same position in the eLearning Division, the division has the capacity to engage in future assessments. Additionally, as many of the trained MoEVT data collectors from the district offices and Teacher Centers are still in the same positions, implementing evaluations can be cost-effective with this support of this team. CHALLENGES MOVING FORWARD Although the institutionalization and sustainability of RISE and ZTUR has been significant, there are a number of challenges to the continued quality and lasting effect of the program activities. These challenges are detailed in the table below. TABLE 6. CHALLENGES TO SUSTAINABILITY OF INSTITUTONALIZED RISE/ZTUR ACTIVITIES Challenge Details  Audio devices: no maintenance and replacement of equipment has taken place  50% of treatment schools had functioning radios  Some schools have rigged ways to fix their radios  MoEVT is working on procuring more radios for new centers but has no plans for fixing the old radios 32  Training: no follow-up or support to old and new educators  Only 2 new trainings have been held since 2011, and teacher turnover/transfer is high  Trainings are in place for 51 new sites/districts, but no follow-up trainings planned for old sites  Lesson Timetable/Curriculum changes: Programming needs to be adapted  Formal schools are not using the programs as intended, as it has not been integrated into the new, denser timetable  eLearning is developing more programs, but there is no strategy for how to integrate these into the timetable  Materials: no new materials have been printed or distributed to replace lost or worn-out materials  No new materials distributions have been made since 2011, but they will be distributed to 51 new sites  There are no plans at present to print and distribute materials to former centers and schools  Monitoring: poor follow-up and inspection of existing schools and sites  Inspectors are supposed to visit TuTu centers and schools, but this doesn’t happen systematically  eLearning has an M&E unit, but limited resources  Monitoring has been more effective in Pemba than Unguja  Lack of permanent structures for 8 of 179 TuTu Centers still meeting under trees: lack of financial resources committed by MoEVT and communities  Lack of physical structures results in no schooling during the rainy reason  Convincing parents to send their children to non￾formal schools is more difficult when there is no permanent structure  Lack of collaboration with Inclusive Education Unit  Materials have not been adapted for learners with special needs  Teachers and mentors are not trained to support special needs children  Although officially the Inclusive Education Unit is collaborating with mLearning, no concrete actions or activities have taken place CONCLUSION: INSTITUTIONALIZATION AND SUSTAINABILITY The goal of achieving a Ministry-integrated program that could be sustained after the RISE/ZTUR projects closed has been achieved. A trained staff is in place, an extensive inventory of materials exists, there is policy support for the approach, 179 communities have been sensitized to ECD, and a play-based learning model in place in non-formal centers and schools. The ECACP program will be rolled out fully during 2015, although it is too soon to assess the efficiency of implementation and the outcomes of the trainings. 33 The major challenges to continued quality implementation lie in maintaining both access to and quality of materials and programming. Maintenance and replacement of radios and broadcasting are major barriers to listenership. A lack of printing and distribution of new materials, and limitations on the resources committed to regular trainings and monitoring, greatly affect the quality and fidelity of implementation. Data do show that the activities appear to have been more actively implemented and monitored in Pemba, which is consistent with reports during the implementation of RISE. This suggests that there is more community support within this region, even though they have fewer resources for the coordination and monitoring of activities than in Unguja (one less coordinator, and no eLearning Division in close proximity). Despite the major logistical and coordination challenges, which are commonplace when donor￾funded projects are transferred to an institution, the eLearning Division has been strategic about moving its work forward with the support and guidance of the MoEVT leadership. They have made conscious and noble efforts to sustain and scale-up RISE and ZTUR activities and to ultimately ensure that their policy goals related to early childhood development are advanced. The fact that the eLearning team is still in place in 2014 is testimony to MoEVT’s commitment, as they have been sought out for subsequent development projects on a number of occasions. The insistence of the MoEVT leadership that the division remains cohesive, as it was created under the RISE and ZTUR projects, has ensured the sustainability of the team and their work. Head teachers, teachers and mentors all expressed the continuing importance of the Tucheze Tujifunze programming. Mentors have continued to facilitate classes, despite poor access to the programs and materials, even in resource lean circumstances. When asked about the value of the programs, mentors noted that when the programs where regularly broadcast, truancy was low at their centers as the programs provided an incentive for learners to attend and to engage in math, language and life skills lessons and gave them a foundation for school they wouldn’t otherwise receive. While the formal school community did not express the same level of commitment to sustaining the Tucheze Tujifunze programs, the fact that a number of teachers were still using the programs despite a lack of official support shows that there is still interest and motivation at the school level. 34 FINDINGS: LEARNER PERFORMANCE The second evaluation question this study aimed to explore is whether learners that participated in IAI (Tucheze Tujifunze) programming performed higher at the end of their primary cycle than those that did not participate. In other words, had RISE students maintained their advantage over their comparison peers in the years since 2008? Quantitative analysis was performed to look at learner performance on a Standard 7 exam that tested performance in literacy, Kiswahili and English, and math. Given that the exposure of students in the treatment group to non-formal education, IAI and formal school, students in the 2014 sample were re-categorized from their original 2008 groupings into three groups:  RISE formal schools group: These are students that continue to be enrolled in government primary school classrooms. At the time of the 2008 study, these students were in a formal Standard 1 (RISE Formal Model) class led by a certified teacher who received additional training in IRI lessons and pedagogy. Classroom teachers used IRI lessons and activities to complement and supplement their instruction.  RISE mixed (non-formal) group: These students were part of the non-formal intervention group and all received IRI in non-formal settings (under a tree), or attended IRI pre-school. There was a small group of students (Combination Model in the 2008) that received non-formal programming in addition to receiving IAI in their Standard 1 classroom. In summary, this group combines students that comprised the RISE Non￾formal Model and Combination model that was used in 2008.  Comparison: Students that have similar characteristics to those students attending formal schools, but who have not been exposed to RISE’s IRI programs. STUDENT DEMOGRAPHICS SEX AND AGE OF LEARNERS IN 2014 The 2014 study team was able to locate more girls than boys from the original 2008 sample. Of all study students tested in 2014, 62.1% were girls. In 2008, 55% of the study students were girls. Attrition among boys was higher than among girls in all study groups, as can be seen in Figure 3. In both 2008 and 2014, RISE non-formal and combined groups had a higher proportion of girls than the RISE formal school model group. 35 FIGURE 2. SEX OF SAMPLE AND BY GROUP (N=904) FIGURE 3. AGE OF SAMPLED LEARNERS IN 2014 (N=901) Figure 4 shows the distribution of sampled learners by age. As this follow-up study was conducted six years after the initial study in 2008, students were expected to be in Standard 7, in which the standard age is between 12 and 14. As seen in the figure, ages vary, although RISE formal school students were found to be of very similar age with the comparison group students: 45% of students in both groups were 14 years old, and about a third of students in both groups were over age of 14. RISE mixed group students were, on average, a year younger than their peers from the formal school group. The median age in the RISE formal group and in 2% 6% 24% 40% 25% 3% 0% 0% 0% 10 11 12 13 14 15 16 17 18 RISE mixed group students, by age 1% 0% 3% 25% 45% 19% 5% 1% 0% 10 11 12 13 14 15 16 17 18 Comparison group students, by age Male 37.9% Female 62.1% Sample by sex, 2014 boys, 41.8% boys, 51.0% boys, 37.5% boys, 43.2% boys, 33.3% boys, 42.0% boys, 37.3% girls, 58.2% girls, 49.0% girls, 62.5% girls, 56.8% girls, 66.7% girls, 58.0% girls, 62.7% Comparison 2014 Comparison 2008 RISE formal 2014 RISE formal 2008 RISE mixed 2014 RISE non-formal 2008 RISE combination 2008 Sample by sex and study group 1% 1% 5% 17% 45% 20% 8% 3% 1% 10 11 12 13 14 15 16 17 18 RISE formal school students, by age 36 the comparison group was 14, the median age among RISE mixed group was 13 years old. Age distributions were consistent across sex and island (Pemba and Unguja). GRADE LEVEL OF STUDY PARTICIPANTS Current Grade Level. In the 2014 follow-up study, students were asked what grade they were currently in. Students were expected to be in Standard 7 if they had continued their education without interruptions or repeating grades. The figure below shows the grade level of sampled students. Despite the fact that all students were in Standard One (grade 1) in 2008, many students have experienced grade repetitions and/or interruptions, causing them to fall behind their peers. As seen in Figure 5, only 51.7% of students in the sample reported that they were in Standard 7. More than a third of students were currently in Standard 6; the remaining 13.5% reported that they were in Standard 5. When compared by intervention model, there were substantial differences in the current grade level of sampled learners. Figure 5 shows grade levels for each intervention model: RISE Formal Group, RISE Mixed (Non-formal Models) Group of intervention, and comparison group (non-RISE participants). FIGURE 4. STUDENT BREAKDOWN BY STANDARD AND GROUP Standard 5, 7.1% Standard 5, 4.8% Standard 5 4.8% Standard 5 31.8% Standard 6 34.7% Standard 6 30.4% Standard 6 26.8% Standard 6 49.4% Standard 7 51.7% Standard 7 62.5% Standard 7 68.4% Standard 7 18.4% All students (n=903) Comparison (non-IRI participants) (n=352) RISE formal schools (n=291) RISE mixed (n=261) 37 Students from the RISE mixed (non-formal and combination) group were targeted to receive IRI intervention through non￾formal education or through pre-school in 2008 because they came from disadvantaged backgrounds, including low levels of parental education, high poverty, and interruption in schooling. This trend appears to have continued, as shown in Figure 6. When surveyed in 2014, students from the combination group were almost twice as likely to report having had interruptions of over a month in their schooling as students from RISE formal school intervention group, and nearly 50% more likely than comparison group students. The majority of these students are in Standard 5 and 6, with nearly half in Standard 6 and one third in Standard 5. Only 18.4% of students in the Mixed Group are in Standard 7. Overall, there appears to be a strong relationship between the age and the grade of study participants, as shown in Figure 7. The majority of students in Standard 5 were found to be 12 or 13 year old, in Standard 6 they were 13 and 14 year old, and in Standard 7 nearly half was 14 year old, and the majority of the rest were either 13 or 15 year old. FIGURE 6. AGE OF SAMPLED LEARNERS IN 2014, BY STANDARD (N=901) Preschool Education. Students were asked whether they had attended preschool prior to Standard One, as it was expected that participation in preschool education would provide students with a head start in developing literacy and numeracy. When compared by group, more students in the RISE intervention (formal and non-formal) groups have received preschool 5% 10% 29% 30% 19% 5% 1% 0% 1% 10 11 12 13 14 15 16 17 18 Age of students, Std. 5 1% 2% 12% 37% 35% 9% 4% 1% 0% 10 11 12 13 14 15 16 17 18 Age of students, Std. 6 0% 0% 2% 19% 47% 22% 7% 2% 0% 10 11 12 13 14 15 16 17 18 Age of students, Std. 7 FIGURE 5. PERCENT OF STUDENTS REPORTING HAVING HAD INTERRUPTIONS OF OVER A MONTH IN THEIR EDUCATION (2014) 13.7% 10.8% 18.3% Comparison (non-IRI participants) RISE formal schools RISE mixed 38 education than RISE formal school model students, a statistically significant difference (p<.001). As seen in the figure below, students in the RISE Mixed (Non-formal and Combination Model) group have the largest percentage of students with preschool education (79.1%); this is likely due to the fact that these students were exposed to preschool education through the RISE Project, while the formal model started RISE programming in Standard 1. FIGURE 7. PRESCHOOL EDUCATION (N=887) READING AT HOME Parental literacy is an important predictor of child’s academic success. During 2014 data collection, students filled out a detailed questionnaire that asked them about their reading habits, as well as about their parents’ literacy level. More than half of the students in the study reported that their parents were literate. Levels of parental literacy varied between participants from formal schools and from RISE mixed intervention models. Among the latter, about 15% fewer students reported having a literate mother and a literate father, as shown in Figure 9. FIGURE 8. PARENTAL LITERACY REPORTED BY STUDY STUDENTS Students were asked whether they read anything at home or outside of school. The majority (85%) of students reported that read the Qur’an every day or every week. Similarly, the majority of students reported that they read at home; 61.4% read a book every day or every week and 27.6% read a book a few times a month. Newspapers were read less frequently, with 64% of students responding that they read the newspaper at least a few times a month. More than Yes, 19.9% Yes, 40.6% Yes, 79.1% No, 61.7% No, 43.7% No, 5.9% Don't Know, 18.4% Don't Know, 15.7% Don't Know, 15.0% Comparison group (n=352) RISE formal schools (n=291) RISE mixed (n=261) mother literate, 66.8% mother literate, 64.0% mother literate, 50.2% father literate, 70.2% father literate, 73.7% father literate, 59.3% Comparison group (n=352) RISE formal schools (n=291) RISE mixed (n=261) 39 three-quarters of students reported that they do not read on the internet. Results were consistent across group, gender and location, as shown in figure 10. FIGURE 9. FREQUENCY OF READING VARIOUS MATERIALS AT HOME ASSESSMENT FINDINGS Longitudinally tracked students were tested again in 2014, six years following the completion of the intervention in which they had participated. Students were tested in literacy (in Kiswahili and in English) and in math, to investigate whether students that participated in RISE IAI programming (either in preschool or Standard One) perform higher on literacy and math assessments than those that did not participate. The tests were developed by the evaluation team and were based on the end of the primary cycle standards in Tanzania (Standard 7). Findings are presented below. SUMMARY OF LITERACY AND MATHEMATICS PERFORMANCE This analysis compares the performance of comparison group students with RISE intervention students (Formal Model and Mixed (non-formal) Model) on grade level assessments in reading and math in 2008 and in 2014. From the 2008 assessment, results of 604 longitudinally tracked comparison group students were compared with 555 RISE Formal Group students and 524 RISE 37% 33% 37% 10% 9% 14% 3% 2% 4% 44% 47% 47% 75% 81% 76% 32% 38% 27% 28% 26% 29% 11% 13% 12% 40% 38% 26% 16% 8% 14% 31% 28% 36% 62% 65% 57% 86% 85% 84% 16% 15% 27% 9% 11% 10% comparison RISE formal RISE mixed comparison RISE formal RISE mixed comparison RISE formal RISE mixed comparison RISE formal RISE mixed comparison RISE formal RISE mixed newspapers books Qur'an magazines Internet never sometimes often 40 Mixed Group students. From the 2014 assessment, only results of students in Standard 7 were included in the analysis, since the test aimed to measure grade-specific competencies and would not be appropriate for learners in lower grades. After removing Standard 5 and 6 students from the analysis, 220 comparison group students, 198 RISE Formal Group students, and 49 RISE Mixed Group students were included in the analysis. The analysis of gain differences between RISE intervention (formal and non-formal) students and comparison group students on the Standard 1 Assessment in 2008 revealed that RISE Formal Group students gained statistically significantly more than their counterparts from the comparison group between the baseline in February/March 2008 and endline in February 2014. The only test in which RISE Mixed Group students gained less than the comparison group was English. FIGURE 10. RISE INTERVENTION AND COMPARISON GROUP GAINS ON STANDARD 1 TEST BETWEEN BASELINE (FEB 2008) AND ENDLINE (NOV 2008) The difference in gains between the baseline and the endline for RISE intervention (Formal and Mixed Groups) students and the comparison group students was statistically significant. Table 7 shows the results of statistical analysis, including the t value and the effect size. TABLE 7. EFFECT SIZE OF GAIN SCORES OF RISE STUDENTS OVER COMPARISON GROUP STUDENTS, STANDARD 1 RISE formal school students’ gains over comparison group students’ gains RISE mixed group students’ gains over comparison group students’ gains t Cohen’s d r t Cohen’s d r 5% 10% 5% 12% 7% 5% 11% 10% 8% comparison (n=604), 23% RISE formal (n=555), 46% RISE mixed (n=524), 29% comparison, 44% RISE formal, 52% RISE mixed, 35% comparison, 31% RISE formal, 41% RISE mixed, 32% 32% 9% 26% 11% 20% 24% 14% 23% Kiswahili English Math STANDARD 1 Baseline Feb 2008 41 Kiswahili 12.965 0.762 0.35 4.716 0.281 0.14 English 10.415 0.613 0.29 n/s Math 7.457 0.438 0.21 2.807 0.167 0.08 n/s = not significant These results clearly demonstrate that the RISE Project was highly effective in improving academic achievement of Formal Group students, and beneficial for RISE Mixed Group students, within the lifespan of the project. But did the program have a lasting effect on the participating students? The assessment of 2014 measured the reading and math competencies of former RISE participants, as well as their comparison group counterparts. Results of the 2014 assessment showed that RISE Formal Group students still outperformed comparison group students, in all three tested subjects, six years after the end of the intervention. These results were found to be statistically significant at the p<.001 level. No significant difference was found between RISE Mixed (Non-formal) Group performance and the comparison group, which performed at approximately the same levels in 2014. FIGURE 11. AVERAGE STANDARD 7 TEST RESULTS, 2014 The difference in scores between the RISE Formal Group students and the comparison group students was statistically significant. Table 8 shows the results of t test statistical analysis and the effect size. TABLE 8. EFFECT SIZE OF SCORES OF RISE STUDENTS OVER COMPARISON GROUP STUDENTS, STANDARD 7 RISE formal school students’ scores over comparison group students’ scores comparison, 34% comparison, 14% comparison, 3% RISE formal, 39% RISE formal, 18% RISE formal, 6% RISE mixed, 32% RISE mixed, 14% RISE mixed, 4% Kiswahili English Math 42 t Cohen’s d r Kiswahili 2.077 .204 .101 English 3.690 .362 .178 Math 2.939 .288 .143 Analysis showed a substantially larger number of students that scored zero on the Standard 7 assessment among comparison group and RISE Mixed Group than RISE Formal Group students. The difference in zero scores is statistically significant at p<.001 level for Kiswahili and English tests. On the math test, the difference is statistically significant at p<.1 level. The difference in zero scores between RISE mixed group and the comparison group is not statistically significant. A higher proportion of RISE Formal Group students was also found to score over 20% correct on all three tests, compared to the comparison group and the RISE Mixed Group. FIGURE 12. 2014 ASSESSMENT: PERCENT OF STUDENTS WITH ZERO SCORES AND GROUPED RESULTS, BY TEST zero, 8% zero, 3% zero, 6% zero, 17% zero, 6% zero, 12% zero, 65% zero, 57% zero, 73% <20 pct 26% <20 pct 24% <20 pct 29% <20 pct 61% <20 pct 64% <20 pct 69% <20 pct 32% <20 pct 36% <20 pct 22% 20-60 pct 51% 20-60 pct 57% 20-60 pct 57% 20-60 pct 22% 20-60 pct 29% 20-60 pct 18% 20-60 pct 3% 20-60 pct 5% 20-60 pct 4% 60-100 pct 15% 60-100 pct 17% 60-100 pct 8% 60-100 pct 0% 60-100 pct 1% 60-100 pct 0% 60-100 pct 2% comparison RISE formal RISE mixed comparison RISE formal RISE mixed comparison RISE formal RISE mixed Kiswahili English Math STANDARD 7 43 SUMMARY OF FINDINGS BY SEX Nearly twice as many Standard 7 girls than boys participated in 2014 assessment. The table below shows the distribution by gender and study group. TABLE 9. 2014 ASSESSMENT PARTICIPANTS BY GENDER Boys Girls Total Comparison group Count 82 138 220 % 37.3% 62.7% 100.0% RISE Formal Group Count 62 136 198 % 31.3% 68.7% 100.0% RISE Mixed Group Count 15 34 49 % 30.6% 69.4% 100.0% TOTAL Count 159 308 467 % 34.0% 66.0% 100.0% Results of data analysis showed that within the two intervention groups, as well as within the comparison group, there were some differences in performance between girls and boys. In the RISE intervention groups, boys scored higher than girls on the Standard 7 assessment (except the Kiswahili test, on which the RISE mixed group girls scored higher than boys). Among the comparison group students, girls scored higher than boys, although the difference is not statistically significant and can be due to chance variation in the sample. This is similar to results in 2008 assessment, when results were similar for both boys and girls FIGURE 13.TOTAL PERCENT CORRECT, BY SEX male 30% male 41% male 31% female 36% female 37% female 33% Comparison group RISE formal schools RISE mixed Std. 7 Kiswahili male 11% male 20% male 15% female 15% female 17% female 13% Comparison group RISE formal schools RISE mixed Std. 7 English male 3% male 8% male 8% female 4% female 5% female 2% Comparison group RISE formal schools RISE mixed Std. 7 Math 44 KISWAHILI LITERACY PERFORMANCE To assess the literacy of students in Standard 7, students were tested on Kiswahili reading competency in five areas: reading comprehension, identifying opposites, grammar (parts of speech), plural words and free writing. Overall students performed the best in writing words in plurals and reading comprehension. Tests results also showed that students performed the worst in free writing, in which they were graded on their response to the question and on sentence structure, as well as spelling, grammar and punctuation. FIGURE 14. AVERAGE PERCENT CORRECT ON KISWAHILI SUBTESTS, BY GROUP As shown in Figure 15, when analyzed by group, RISE Formal Group students performed the best of the three study groups in all areas, except free writing and antonyms. Comparison group students outperformed other students on free writing part of the assessment. The difference male 21% male 48% male 27% female 28% female 51% female 30% Comparison group RISE formal schools RISE mixed Std. 1 Kiswahili male 42% male 53% male 33% female 46% female 54% female 34% Comparison group RISE formal schools RISE mixed Std. 1 English male 32% male 41% male 26% female 32% female 42% female 29% Comparison group RISE formal schools RISE mixed Std. 1 Math 49% 38% 38% 30% 34% 38% 21% 22% 30% 63%67% 71% 8% 15% 21% 0% 10% 20% 30% 40% 50% 60% 70% 80% Reading comprehention Antonyms Comparison RISE Mixed RISE Formal Grammar Plurals Free writing 45 between groups was found to be statistically significant at p<.05 level, except for the antonyms subtest where no statistically significant differences were detected. ENGLISH LITERACY PERFORMANCE Students were tested in four areas of English language:  ability to form a question (e.g., for a sentence “I like mangos”, what would be a corresponding question?)  ability to understand a written sentence (e.g., “The sun rises from the west” – true or false?)  ability to understand a text and answer questions associated with the text  ability to write a few sentences on a given topic that are grammatically correct All tested students performed poorly on the English language assessment. The portion of the test that students did the best on was the ability to understand a written sentence and provide an answer as to whether the sentence was true or false. However, even on this portion of the test students did not perform very well: since each question required a “true” or “false” answer, there is a 50% chance that the student would get a correct answer by chance. The average correct was, however, only 41%. Students in all study groups performed poorly on the English assessment, although the RISE Formal Group students did better than their counterparts from the comparison group or from the RISE Mixed Group. Figure 16 shows the results, by subtest and study group. FIGURE 15. ENGLISH SUBTESTS AVERAGE PERCENT CORRECT, BY GROUP 1% 5% 1% 43% 39% 42% 7% 10% 14% 1% 0% 0% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% Grammar (writing questions) Vocabulary questions (true/false) Comparison RISE mixed RISE formal schools Reading comprehension questions Free writing 46 MATH PERFORMANCE The math portion of the assessment was intended to test grade-level competencies, such as basic number manipulations (include large numbers as well as decimals), foundations of algebra and geometry, and fractions. Students were presented with 10 tasks. Students did poorly on all tasks, with the best result for the simple subtraction problem. However, students from the RISE Formal Group did better than their counterparts on nine out of ten questions. The figure below shows results grouped by type of math problem. FIGURE 16. MATH SUBTESTS AVERAGE PERCENT CORRECT, BY GROUP The results are presented for each item in the table below. TABLE 10. MATH ASSESSMENT RESULTS, BY ITEM AND STUDY GROUP 2014 ASSESSMENT MATH PROBLEM PERCENT OF STUDENTS WITH CORRECT ANSWERS 1. 0.1 x 100 2. 400,000 ÷ 1,000 1.5% 3.9% 5.2% 0.9% 1.1% 1.4% 0.4% 0.5% 1.6% 0% 1% 2% 3% 4% 5% 6% Computations (5 items) Geometry (2 items) Comparison RISE Mixed RISE formal schools Other (3 items: algebra, percentile and ratio computations) 8% 13% 8% comparison RISE formal RISE mixed 9% 12% 8% comparison RISE formal RISE mixed 47 3. 3,547 - 19 4. Katika mlinganyo 2(2p-7)=31-5p thamani ya p ni 5. Tafuta (-8) – (+4) 6. Shehia ya Pangawe ina watu 4000. Ikiwa 40% ni wanawake, 35% ni wanaume, na waliobakia ni watoto, tafuta idadi ya watoto waliopo katika shehia hiyo. 7. Katika mchoro ufuatao, tafuta thamani ya ‘x’ 8. Mwalimu aliwapa Jaku na Jokha vitabu 100 na akawaambia wavigawe katika mafungu mawili kwa uwiano wa 1:4. Baada ya kuvigawa, kila fungu lilikuwa na vitabu vingapi? 9. Kibati cha kutilia taka kina urefu wa sm 10 na nusu kipenyo cha sm 2. Ikiwa kibati hicho kitazungushiwa karatasi pembeni, eneo la karatasi hiyo litakuwa ngapi? 10. Rahisisha: 2¼ ÷ ¾ x ⅜ + 1½ 29% 32% 16% comparison RISE formal RISE mixed 3% 9% 2% comparison RISE formal RISE mixed 4% 7% 4% comparison RISE formal RISE mixed 0% 2% 0% comparison RISE formal RISE mixed 3% 5% 0% comparison RISE formal RISE mixed 0% 3% 2% comparison RISE formal RISE mixed 0% 1% 0% comparison RISE formal RISE mixed 1% 5% 8% comparison RISE formal RISE mixed sm 5 sm x cm sm 4 sm 10 48 DISCUSSION To help us better understand why some students perform better on the standardized assessment, and some perform worse, the study considered information about students’ background. To what extent did students from literate households achieve higher test scores? Did attending preschool have an effect on test results in Standard 7? Was schooling interruption a factor in student performance? Bivariate correlation41 analysis showed that parental literacy was associated with better scores on the reading test, in both Kiswahili and in English. Interruptions of a month or more were found to be negatively associated with scores: students who experienced such interruptions were found to be scoring statistically significantly lower than their peers who did not experience such interruptions, in all three tested areas. Students who said they read books and the Qur’an were found to have higher results in all three tested areas, although an association between reading books and higher math scores was weak. Attending preschools was found to be statistically significantly associated with higher scores in math and English, but only among students who were in Standard 7 (that is, students who were in the appropriate grade, given their Standard 1 status in 2008). Table 11 shows the results of the statistical analysis. TABLE 11. BIVARIATE CORRELATION BETWEEN STUDENT BACKGROUND CHARACTERISTICS AND TEST RESULTS Kiswahili English Math Attending preschool n/s .099* (7 grade students only) .125* (7 grade students only) Interruptions of schooling for over a month -.210** -.175** -.102** Mother literate .221** .128** .079* Father literate .256** .151** .076* Reading books regularly .296** .229** .138** Reading Qur’an regularly .274** .161** .071* Other variables included in the analysis did not appear to have association with the student performance on the assessment. 41 “Correlation” is a statistical term that describes a degree of relationship between two variables. Two variables are considered correlated when a change in one is associated with a change in another. Correlation does not presume causation since a change in both variables might be caused by the third variable. Correlation coefficient ranges between 0 and 1, with higher value denoting stronger relationship. Correlations in a range between .1 and .3 are considered rather weak, in a range between .4 and .6 are considered moderate, and above .6 are considered strong. 49 CONCLUSION: LEARNER PERFORMANCE The 2014 student assessment found that RISE Formal Group students performed significantly better on standardized grade-level assessments in Kiswahili, English and math, compared to their counterparts from the comparison group and from the RISE mixed group. Six years after the intervention, RISE Formal Group students demonstrated a better mastery of grade-level concepts. However, the overall performance level in English and math was found to be very low among all study groups. The RISE intervention appears to have contributed to sustained advantages for the formal school students who participated in the program. Unfortunately, not enough students from the non-formal or combined programs could be traced to enable the drawing of strong conclusions about the effect of the program on those populations. A significant portion of RISE Mixed Group students either could not be found for inclusion in the study, or had repeated grades and thus could not be included in the analysis. Since those students were particularly disadvantaged, due to very low parental literacy or due to poverty, they were an important focus of RISE Project, and determination of the long-term benefit of the program to their academic performance would be helpful in informing policy decisions about how to target continued program development through the eLearning Division of the MoEVT. Additional follow up or tracer studies that focus particularly on this group of students would be beneficial. 50 CONCLUSION Despite logistical and coordination challenges, which are common when donor-funded projects are transferred to an official counterpart institution, the MoEVT’s goal of using RISE and ZTUR as the foundation for long-term improvements in early education in Zanzibar has largely been achieved. The eLearning Division has been strategic about moving its work forward with the support and guidance of the MoEVT leadership. They have made conscious and noble efforts to sustain and scale-up RISE and ZTUR activities and to ultimately ensure that their policy goals related to early childhood development are advanced. The insistence of the MoEVT leadership that the division remains cohesive, as it was created under the RISE and ZTUR projects, has ensured the sustainability of the team and their work, and encouraged the confidence of new partners who can substantially contribute to continued development of the early education sector. In addition, the RISE model of intervention has demonstrated sustained positive effects on students who participated in the programs, providing them with an advantage in academic performance 6 years after their initial performance was assessed. This is particularly true for students in the RISE Formal Group. However, the assessed achievement of both RISE and comparison participants was still relatively low, when compared to national expectations for grade-level performance in Standard 7. Clearly, as identified in the Zanzibar Education Policy and Zanzibar Education Development Plan, work remains to be done to increase the percentage of Zanzibari students who meet performance standards across pre-primary and primary education. Additional tracer research on the impact of the RISE programming model on the most disadvantaged students (those in the RISE non formal group) would also help ensure the greatest possible impact on that group as programming continues to evolve. Given the Zanzibari context, interactive audio programming (IAI) remains a viable and useful intervention format for increasing access to high-quality early education, particularly in the most challenged districts. Careful attention will need to be paid to the mechanisms necessary for continuing to ensure the quality of TuTu programming (specifically: maintenance of hardware; thoughtful integration of the programs into the increasingly dense school timetable; the provision of continuous support and professional development for teachers and mentors; and sufficient and effective monitoring and evaluation of programs as they are rolled out). With additional investment in these quality assurance processes, and the continued commitment of the MoEVT to the work of the eLearning Division, the improvements begun under RISE and ZTUR can continue to be built upon and sustained. 51 REFERENCES Downie, Bruce and Peter Mwaura. (January, 2010) Building Capacity for Pre-School Teaching: A Status Assessment and Recommendations for Teacher Training Programming. Reported was commissioned by the ZTUR project, with direct support from USAID. ILO. (2008). Good Practices on Social Protection and Coping Strategies used by Low Income Women Workers in the Informal Economy in Zanzibar to Mitigate Against Social and Economic Risks, 6. Accessed on November 2, 2014. http://www.ilo.org/wcmsp5/groups/public/---dgreports/--- gender/documents/publication/wcms_098172.pdf. Kilimanjaro International. (February 18, 2011). Radio Capacity in Zanzibar. Ministry of Education and Vocational Training (MoEVT). (2006). Education Policy. Revolutionary Government of Zanzibar. Ministry of Education and Vocational Training (MoEVT), Revolutionary Government of Zanzibar. (2007). Zanzibar Education Development Programme (ZEDP) 2008/2009 2015/16 Draft 3. p.58 Ministry of Education and Vocational Training, Revolutionary Government of Zanzibar. (2013). Zanzibar EMIS office under the Department of Policy, Planning and Research (DPPR). Ministry of Education and Vocational Training website (access July 21, 2014). http://www.moez.go.tz/index.php?cq=dept&dept=10 National Bureau of Statistics, United Republic of Tanzania. (2013). Population and Housing Census 2012. pp. vii Office of Chief Government Statistician, Zanzibar. (May 2010). Socio-Economic Survey, p. 49, 59. Tanzania Millennium Development Goals. (September 2011). Country Report on the Millennium Development Goals. Dar es Salaam, p. 16 USAID/Tanzania (2009).USAID/Tanzania Education Strategy for Improving the Quality of Education FY 2009-2012, p. 2 Zanzinet (access August 8, 2014), “About Zanzibar,” http://www.zanzinet.org/zanzibar/visiwa.html 52 APPENDIX: LEARNING AND TEACHING MATERIALS TABLE 12. LEARNING AND TEACHING MATERIALS DEVELOPED, PRODUCED AND DISTRIBUTED UNDER RISE # Produced Description Details # Distributed 276 IAI Programs 78 Preschool, 99 Standard 1, 99 Standard 2 837 total aired 3 Mentor’s Guides Preschool, Standard 1 and 2 795 1 Songbook Songs for children in English and Swahili covering literacy, math, language and life skills 8400 2 Learner Workbooks Standard 1 and 2 8400 4 Flashcard Sets Numeracy, alphabet, shapes and animals 748sets 4 Story Cards Pictures and text 748cards 1 Classroom poster set 6 pages (English Alphabet, Kiswahili Alphabet, Body Parts, Domestic Animals, Wild Animals and Marine Environment) 800 1 Learning Kit Colored cubes and dices, string, seeds, seashells, cloth, chalk, slates, rope, sticks, bottle tops, numeracy and literacy flash cards, exercise books, pencils 190 1 Teaching Kit Chalkboard, chalk, tape measure, scissors, cards and paper, rubber bands, story cards and story books 180 Radios Lifeline wind-up and solar charged 591 10 Story Books Zanzibar story books for children (10 titles) donated by Rotary Club 975 10 Story Books Zanzibar story books for children (7 titles) 1000 53 TABLE 13. LEARNING AND TEACHING MATERIALS DEVELOPED, PRODUCED AND DISTRIBUTED UNDER ZTUR Materials # Printed Total Needed NTRC 11 TC MoEVT HQ MoEVT IAI Vuga Benjamin Mkapa MoEVT Pemba 248 Primary 31 Pre-schools EDC Central Library ZMRC Other NGOs TuTu Pre-school Mentors Guide 321 321 2 33 1 1 249 31 1 2 1 Std 1 Teacher's Guides 321 321 2 33 2 1 249 31 1 2 Std 2 Teacher's Guides 324 224 2 33 5 1 149 31 1 2 Std 1 Learner Books 309 309 2 33 21 1 249 1 2 Std 2 Learner Books 489 489 2 33 21 1 429 1 2 Children Storybook 1000 1000 19 319 1 59 1 1 498 93 5 2 1 1 TuTu Kits 13 13 1 11 1 Lifeline radio (Hybrid with MP3s) 652 652 5 77 1 5 1 1 498 62 1 1 IAI Trainer's guides 50 50 9 33 1 1 1 1 1 2 1 IAI Teachers' guides 350 300 9 275 1 8 1 1 1 2 1 1 SDLM Introduction for Trainers 50 50 9 33 1 1 1 1 1 2 1 SDLM Introduction for Teachers 350 300 9 275 1 8 1 1 1 2 1 1 SDLM Trainer's Guides (4 topics) 50 50 9 33 1 1 1 1 1 2 1 SDLM Teachers' guides (4 topics) 350 300 9 275 1 8 1 1 1 2 1 1 Verification sheets (SDLM) 1000 1000 18 924 1 53 1 1 1 1 Verification sheets (IAI) 1000 1000 18 924 1 53 1 1 1 1 Video Equipment User Guide 50 50 9 33 1 1 1 1 1 2 1 Lifeplayer Radio User Guide 1050 1050 23 385 3 67 3 3 498 62 3 3 CDs 500 500 19 319 11 106 5 5 23 2 1 9 ECD L & T Inventory 60 60 9 33 1 1 1 1 1 2 1 10 Songbooks 350 350 19 319 1 5 1 1 1 2 1 ELTP Module* 50 50 9 33 1 2 1 1 1 2 IPod 14 14 1 12 1 Pico projector + extra batteries 14 14 1 12 1 Windup Speakers 24 24 1 22 1 ZTUR bags 400 400 23 330 7 29 2 2 2 2 3 IPod + I trip* 10 10 1 Pico projector + extra batteries* 10 10 1