TIME TO LEARN PROJECT YEAR 4 PERFORMANCE EVALUATION REPORT February 2016 This publication was produced for the USAID Time to Learn Project. It was prepared by Rebecca Frischkorn, Zachariah Falconer-Stout, and Lyn Messner, Time to Learn/EnCompass LLC. USAID Time to Learn Project The Time to Learn project is funded by the United States Agency for International Development in Zambia under contract number AID-611-C-12-00002, awarded on March 1, 2012. Time to Learn is implemented by Education Development Center, Inc., in collaboration with the Campaign for Female Education, EnCompass LLC, and the Forum for African Women Educationalists in Zambia. The project assists the Ministry of General Education through a 5-year national program to provide an equitable standard of education service for vulnerable learners, improve reading skills, and implement practical strategies to strengthen school quality and promote community engagement in community schools. Cover photo by: Kennedy Makulika, Time to Learn TIME TO LEARN PROJECT YEAR 4 PERFORMANCE EVALUATION REPORT February 2016 Project Number AID-611-C-12-00002 DISCLAIMER This evaluation is made possible by the support of the American people through the United States Agency for International Development (USAID). It was produced by Encompass LLC for the Time to Learn project, which is funded by USAID in Zambia under contract no. AID-611-C-12-00002. The contents of this evaluation are the sole responsibility of the author(s) and do not necessarily reflect the views of USAID or the United States Government. Time to Learn Project Year 4 Performance Evaluation Report iv CONTENTS ACRONYMS .................................................................................................................................. V EVALUATION TEAM .................................................................................................................VI EXECUTIVE SUMMARY........................................................................................................... VII 1. EVALUATION PURPOSE AND EVALUATION QUESTIONS .......................................... 1 1.1. EVALUATION PURPOSE............................................................................................................................................................. 1 1.2. EVALUATION CONTEXT AND DESIGN PROCESS.................................................................................................................. 2 1.3. EVALUATION QUESTIONS........................................................................................................................................................ 2 2. PROJECT BACKGROUND ...................................................................................................... 4 2.1. PROJECT CONTEXT.................................................................................................................................................................. 4 2.2. TTL INTERVENTIONS ............................................................................................................................................................... 6 3. EVALUATION DESIGN, METHODS, AND LIMITATIONS ............................................. 11 3.1. EVALUATION DESIGN.............................................................................................................................................................11 3.2. METHODS.................................................................................................................................................................................12 3.3. DATA COLLECTORS ...............................................................................................................................................................13 3.4. DATA ANALYSIS ......................................................................................................................................................................13 3.5. LIMITATIONS ............................................................................................................................................................................14 4. FINDINGS ................................................................................................................................ 15 4.1. EVALUATION QUESTION 1: WHAT SUPPORTS AND HINDERS COMMUNITY SCHOOL TEACHERS IN IMPLEMENTING EFFECTIVE EARLY GRADE LITERACY INSTRUCTIONAL PRACTICE? ...................................................................................15 4.2. EVALUATION QUESTION 2: WHAT KINDS OF LEADERSHIP AND MANAGEMENT ACTIVITIES SUPPORT COMMUNITY SCHOOL TEACHERS TO TEACH EARLY GRADE LITERACY? ...............................................................................................25 5. CONCLUSIONS AND RECOMMENDATIONS.................................................................. 32 5.1. CONCLUSIONS........................................................................................................................................................................32 5.2. RECOMMENDATIONS..............................................................................................................................................................35 ANNEXES .................................................................................................................................... 38 ANNEX 1: EVALUATION MATRIX .....................................................................................................................................................39 ANNEX 2: DATA COLLECTION INSTRUMENTS...............................................................................................................................41 ANNEX 3: REFERENCES......................................................................................................................................................................53 Time to Learn Project Year 4 Performance Evaluation Report v ACRONYMS CPD continuing professional development eEGRA Electronic Early Grade Reading Assessment ELM education leadership and management IR intermediate result MOGE Ministry of General Education PCSC parent community school committee SIR sub-intermediate result SPRINT School Program of In-Service for the Team TTL Time to Learn USAID United States Agency for International Development ZIC zonal in-service coordinator Time to Learn Project Year 4 Performance Evaluation Report vi EVALUATION TEAM Core Team Rebecca Frischkorn, Research and Evaluation Specialist, Time to Learn Zachariah J. Falconer-Stout, Evaluation Specialist, Time to Learn Lyn A. Messner, Senior Research and Evaluation Advisor, Time to Learn Data Collectors Mable Banda, Monitoring and Evaluation Assistant, Time to Learn Sheila Chiile, District Resource Centre Coordinator, Provincial Education Office, MOGE Mark Chilanda, Intern, Examinations Council of Zambia, MOGE Chilanda Chileshe, District Resource Centre Coordinator, Provincial Education Office, MOGE Anthony Chomba, Statistician, Provincial Education Office, MOGE Paul Daka, Provincial Outreach Coordinator, Time to Learn Boniface Lisuba, Senior Exam Specialist, Examinations Council of Zambia, MOGE Kennedy Makulika, Monitoring and Evaluation Specialist, Time to Learn Humphrey Tembo, Monitoring and Evaluation Assistant, Time to Learn Time to Learn Project Year 4 Performance Evaluation Report vii EXECUTIVE SUMMARY Evaluation Purpose and Evaluation Questions The USAID Time to Learn project (TTL) is conducting internal impact and performance evaluations over the life of the project to assess its interventions at different points and from different perspectives, and to provide a holistic understanding of project results over time. The purpose of this internal performance evaluation, conducted 1 year after midline evaluation data collection, is to better understand community school teacher practice, especially in literacy instruction. This evaluation seeks to answer questions and fill knowledge gaps that emerged from the midline impact evaluation about what is working well and what can be improved in TTL’s intervention areas. The key evaluation questions (Exhibit 1) were developed jointly with the TTL project team and are presented. Project Background The 5-year (2012-2017) TTL project aims to improve reading among 500,000 primary grade community school learners in six of Zambia’s 10 provinces. TTL seeks to inform and inspire policy dialogue at the central Ministry of General Education (MOGE) level, creating a favorable environment for effective implementation of the MOGE policy for integrating community schools into the formal education system and providing a range of MOGE actors with an opportunity to understand how to sustain and generalize TTL interventions for project scale-up. TTL’s development hypothesis aims to improve reading among community school learners through four changes: x Increased MOGE support to community schools x Improved community school literacy instruction and educational management x Improved parent community school committee (PCSC) governance, resource mobilization, and advocacy for quality reading instruction and support, including in the home x Increased access to age-appropriate, familiar-language teaching and learning materials. TTL seeks to achieve these results through four types of activities: MOGE and PCSC capacity building, teacher training, and development and dissemination of teaching and learning materials. EXHIBIT 1: KEY EVALUATION QUESTIONS 1. What supports and hinders community school teacher in implementing effective early grade literacy instructional practice? 2. What kinds of leadership and management activities support community school teachers in teaching early grade literacy? Time to Learn Project Year 4 Performance Evaluation Report viii Evaluation Design, Methods, and Limitations EnCompass LLC designed the project year 4 performance evaluation through a consultative process with TTL staff. Drawing on the midline findings, the key theme of interest to project stakeholders was to better understand community school teacher practice, with a particular focus on literacy instruction. In September and October 2015, the evaluation team collected data from 14 schools in three provinces through semi-structured interviews with grade 1 and 2 teachers, head teachers, and MOGE zonal staff; focus group discussions with teachers in each province; and usage data from the early grade reading Stepping Stone mobile phone program. To better understand successes and challenges, the team selected top and bottom schools (with regard to literacy instructional practices) from the midline evaluation dataset. All data collectors were MOGE officials or TTL staff. The evaluation encountered a number of limitations common to the community school setting, including respondents’ struggle to recall specific TTL activities, the effects of possible teacher turnover on the sampled schools, and data collector bias. In addition, since the sample was purposive and not representative, the findings cannot be generalized to the entire TTL intervention area. Findings, Conclusions, and Recommendations Findings Evaluation Question 1: What supports and hinders community school teachers in implementing effective early grade literacy instructional practice? Teachers stated that their literacy lessons focused strongly on phonemic awareness and phonics, while they struggled to incorporate other literacy components. Most teachers saw phonemic awareness and phonics as the most important, but highlighted specific challenges to including other components, such as fluency in the language of instruction, lack of adequate teaching and learning materials, and time constraints. Teachers pointed to low staffing, high enrollment, and absenteeism as further hindering progress in early grade literacy acquisition. Teachers appreciated literacy training by both TTL and MOGE as positively influencing their classroom practice, as well as material support through government-issued textbooks and TTL’s early grade reading Stepping Stone application and phone. Early grade reading Stepping Stone content was reinforced during school-based teacher group meeting. Evaluation Question 2: What kinds of leadership and management activities support community school teachers in teaching early grade literacy? A variety of head teacher training events were instrumental in improving management activities at community schools. Zones provided leadership and management support through zonal literacy training, managing assessment data, and providing materials. In addition, monitoring and observation of literacy classes by zonal in-service coordinators (ZICs) occurred regularly. Head teachers and teachers alike appreciated the training and monitoring activities and wanted them to continue. However, community schools needed even more support from ZICs. The quality and quantity of monitoring activities were affected by distance and the cost of travel to the community schools, particularly remote schools, limited materials, and teacher turnover, as well as the ZICs’ knowledge of literacy components. Time to Learn Project Year 4 Performance Evaluation Report ix Conclusions The project year 4 performance evaluation findings provided additional validation for the midline finding that community school teachers devoted substantially more instructional time to letter sounds than to other literacy skills, and that activities related to comprehension and learners reading during class were almost entirely absent (Falconer-Stout et al. 2015). The performance evaluation independently found that activities that emphasize phonemic awareness, phonics, and syllable and word coding and decoding received the most emphasis during literacy lessons in grades 1 and 2. Activities that offered learners the opportunity to independently read connected text, build comprehension, and practice handwriting all received less emphasis during lessons. Overall, the performance evaluation findings indicated several related reasons that help explain these instructional practices. Explanations include factors related to literacy principles and how those principles have been understood by teachers and in the broader framework of curricular reform in Zambia, as well as a wider range of factors specific to the Zambian community school context. These contextual factors include overcrowding, understaffing, and poor learner attendance, which affect the content and progress of literacy lessons because teachers focus on activities they feel are easier to accomplish, given time constraints and teacher-to-learner ratios. High teacher turnover also hindered the ability to build teachers’ capacity over time. The MOGE’s expanded role in community schools is evident throughout the evaluation findings. Most schools highlighted monitoring activity by MOGE officials, community school teachers’ inclusion in continuing professional development (CPD) programs at zonal and district levels, participation in national literacy assessments, and provision of teaching and learning materials, such as the grade 1 and 2 textbooks. The evaluation found a fairly consistent intended method of handing reports and data through the MOGE structure, although adherence varied. In addition, trained teachers and head teachers on the MOGE payroll were being deployed to community schools. This aligns with evidence from the midline evaluation, which found that the types and amount of MOGE support from zonal and district offices had increased notably since 2012 (Falconer-Stout et al. 2015). However, this MOGE involvement, particularly through monitoring activities, tended to reinforce teachers’ focus on a limited number of literacy skills—namely, letter sounds. Although teacher group meetings were held regularly at community schools, the types of support needed to ensure well-rounded literacy lessons were largely absent. Recommendations 1. TTL and the MOGE should strengthen teacher training modules to clarify how to integrate all five components of literacy instruction into one lesson. 2. TTL should follow up with schools that have received the early grade reading Stepping Stone phone training to better understand and improve usage of the program. 3. TTL should work with the MOGE and head teachers to improve the quality of formative community school monitoring. 4. TTL should work with the MOGE to continue to integrate community schools into the MOGE system. Time to Learn Project Year 4 Performance Evaluation Report 1 1. EVALUATION PURPOSE AND EVALUATION QUESTIONS 1.1. Evaluation Purpose TTL is conducting internal impact and performance evaluations over the life of the project for itself and its stakeholders to measure and understand progress made toward three of the five intermediate results (IRs) for TTL, indicated in Exhibit 2 by the red boxes, and toward USAID/Zambia IR 3.1. EXHIBIT 2: TTL AND USAID RESULTS FRAMEWORKS Taken together, these evaluations reflect a multilevel and sequential mixed-method approach intended to assess TTL interventions at different points and from different perspectives and provide the project with a holistic understanding of results. Exhibit 3 (on the next page) illustrates this approach; the blue boxes indicate the three phases of the impact evaluation and the red boxes indicate performance evaluations. Performance evaluations use qualitative and limited quantitative data to assess how TTL interventions are being implemented and perceived by key stakeholders. Conducted in project years 2 and 4, in between impact evaluation phases, these evaluations help TTL understand why the project has been effective and how it can be improved. Impact evaluation phases are being conducted in project years 1 (baseline), 3 (midline), and 5 (endline) to measure literacy levels among primary learners (see Exhibit 2, USAID/Zambia IR 3.1) and other intermediate outcomes to determine whether teachers, head teachers, and the MOGE are using TTL-promoted techniques. Impact evaluation activities use primarily quantitative data. Time to Learn Project Year 4 Performance Evaluation Report 2 The purpose of this internal performance evaluation, conducted 1 year after midline evaluation data collection, is to better understand community school teacher practice, especially literacy instruction. This evaluation seeks to answer questions and fill gaps in the existing knowledge that emerged from the midline impact evaluation of what is working well and what can be improved in TTL intervention areas. 1.2. Evaluation Context and Design Process On April 15 and 16, 2015, EnCompass LLC facilitated a 2-day meeting for TTL project staff in Lusaka to validate midline impact evaluation findings, gain feedback on the draft report from the midline impact evaluation, and determine priorities for the project year 4 performance evaluation. The meeting helped participants focus on what they wanted to learn from this performance evaluation based on questions and knowledge gaps that emerged from the midline impact evaluation about what is working well and what can be improved in TTL interventions. In this way, the project year 4 performance evaluation contributes to a holistic series of evaluations by building on the midline and informing the TTL interventions that will be assessed at endline, thereby contributing to a more complete understanding of TTL. The midline finding that generated the most interest among project stakeholders was that community school teachers devoted substantially more instructional time to letter sounds than other literacy skills, and that activities that focus on comprehension and offer learners the opportunity to read during class were almost entirely absent (Falconer-Stout et al. 2015). Drawing on this finding, the key theme that emerged from the April meeting was TTL staff’s desire to better understand community school teacher practice, especially the factors driving current literacy instruction practices. This performance evaluation was designed to explore this theme in order to understand why community school teachers have responded to TTL activities in this manner. 1.3. Evaluation Questions Exhibit 4 presents the key evaluation questions and corresponding sub-questions, which were developed jointly with the TTL project team. These questions explore three crosscutting themes: (1) how teachers and head teachers understand and implement early grade literacy instructional practice, (2) what supports and hinders early grade literacy instructional practice, and (3) how teachers and head teachers perceive CPD programs. EXHIBIT 3: TTL EVALUATION TIMELINE Time to Learn Project Year 4 Performance Evaluation Report 3 EXHIBIT 4: YEAR 4 PERFORMANCE EVALUATION QUESTIONS Key Evaluation Question Sub-questions 1. What supports and hinders community school teachers in implementing effective early grade literacy instructional practice? a) What enables and constrains community school teachers to incorporate all five core pillars1 of early grade literacy into their lessons? b) In what ways do teaching and learning materials used by teachers support early grade literacy lessons? c) To what extent are teachers using the Early Grade Reading Stepping Stone program? What has enabled and constrained teachers’ use of the Early Grade Reading Stepping Stone program? d) What factors facilitate head teacher classroom observation and teacher group meetings to result in changes in instructional practice? 2. What kinds of leadership and management activities support community school teachers in teaching early grade literacy? a) To what extent have head teachers used educational leadership and management techniques to support teachers and improve instructional practice? b) Which forms of continuing professional development are teachers most interested to see continued by the MOGE after the project ends? c) How is routine monitoring by the MOGE, including use of eEGRA Instruct, being used as a formative tool for teacher development and for promoting changes in classroom practice? 1 The MOGE’s five core pillars of early grade literacy lessons are phonemic awareness, phonics, oral reading fluency, vocabulary, and comprehension. According to the midline findings, teachers incorporated more letter sounds and decoding activities into their literacy lessons than reading and comprehension activities. TTL evaluation activities divide the teaching and learning of literacy into seven domains for measurement purposes, but these domains correspond to the same principles of literacy learning. Time to Learn Project Year 4 Performance Evaluation Report 4 2. PROJECT BACKGROUND The 5-year TTL project endeavors to improve reading among 500,000 grade 1 to 4 community school learners in six of Zambia’s 10 provinces, as illustrated in Exhibit 5 and Exhibit 6. TTL contributes to USAID Education Strategy (2011-2015) Goal 1 (improved reading skills for 100 million children in primary grades by 2015), which establishes early grade reading ability as a key determinant of retention and success in future grades. TTL aims to create a favorable environment for effective implementation of the MOGE policy to integrate community schools into the government education system and provide a wide range of MOGE actors with an opportunity to understand how to sustain and generalize TTL interventions for scale-up. EXHIBIT 5: TTL CATCHMENT AREA Provinces # Districts # Community Schools Central 11 437 Copperbelt 9 342 Eastern 9 386 Lusaka 8 496 Muchinga 7 265 Southern 13 381 TOTAL 57 2,307 Source: 2015 TTL monitoring data. 2.1. Project Context In 2002, Zambia declared free primary education by officially abolishing school fees for grades 1 to 7. According to the 2014 MOGE Educational Statistical Bulletin, male and female enrollment in early grades has increased steadily since 2002, much of which is attributed to community schools. As their name implies, community schools are created and managed by communities. They typically include grades 1 to 7 and are managed through a PCSC, which has the primary responsibility for managing and supporting the school. In 2011, the Zambia Education Act officially recognized community schools as one of four types of primary and EXHIBIT 6: MAP OF TTL PROVINCES Time to Learn Project Year 4 Performance Evaluation Report 5 secondary educational institutions. The MOGE subsequently launched a series of policies to integrate community schools into the public education sector. Today, community schools still serve Zambia’s most vulnerable rural and urban communities, help relieve overcrowding, and provide access to education in the most rural areas, where government schools are too distant for young learners to access. Although accurate numbers are difficult to obtain, a commonly cited estimate puts the number of community schools across Zambia at 3,000, with those learners comprising at least 18 percent of the primary school population. Most community school teachers are volunteers who receive stipends from the community, lack formal teacher training, and generally have no more than a secondary school education. The MOGE now has policies for deploying trained government teachers to serve in community schools and for monitoring community schools. Registered community schools are eligible to receive government assistance in the form of CPD (e.g., in-service training), small grants, books and other materials, and infrastructure. Increasing community school enrollment has helped Zambia to achieve Education for All universal primary enrollment targets, but educational quality has remained a challenge across the primary education sector in government and community schools. Zambian primary school learning outcomes have remained among the lowest in Sub-Saharan Africa. Low levels of literacy attainment are a particular challenge. The 2012 Grade 5 National Assessment Survey found that only one-third of learners met national literacy standards by the end of grade 5 (MOGE 2013c). In recent years, addressing low reading performance has become a national priority. The MOGE recently launched the new Primary Literacy Program, which incorporates a wide range of changes to literacy instruction for grades 1 through 4, including: x Introducing a new approach to literacy within the revised national curriculum (2013) x Adopting a policy of familiar-language instruction for teaching early grade reading in seven Zambian languages (replacing English as the medium of reading instruction), and beginning oral English lessons in Grade 2 (2013) x Developing textbooks in the seven Zambian languages of instruction and distributing these based on official regional languages for each area (2013-2017) x Defining performance-level descriptors (MOGE 2014) that specify achievement standards for core pre-reading and reading competences (2014) x Introducing the National Literacy Framework (MOGE 2013a), which specifies outcomes by each stage of literacy acquisition, provides a series of formative and summative assessments, and includes weekly grade 1 language schedules for the introduction and revision of letter sounds and blends (2013) x Releasing a revised Zambian Languages Syllabus (MOGE 2013b) for grades 1 through 7 that specifies lesson topics and learning outcomes (2013). As part of the rollout for this new literacy program, the MOGE conducted a series of orientation workshops for head teachers and grade 1 and 2 teachers through its teacher education cascade. Commensurate with the MOGE strategy of integrating community schools into the formal education sector, community schools have been included in these literacy orientation workshops, as well as in national literacy assessment activities. Time to Learn Project Year 4 Performance Evaluation Report 6 2.2. TTL Interventions TTL’s development hypothesis, illustrated in Exhibit 7, is that community school learners will have improved reading as a result of the following changes (represented by the gray boxes): x Increased support to community schools from the MOGE x Improved literacy instruction among community school teachers and better educational leadership and management by head teachers x Improved PCSC school governance, resource mobilization, and advocacy for high-quality reading instruction and support x Improved support for literacy in the home x Increased community school access to age-appropriate textbooks and other teaching and learning materials in a familiar language. EXHIBIT 7: TTL DEVELOPMENT HYPOTHESIS TTL endeavors to achieve these results through four strategies, represented by the dark blue boxes: MOGE capacity building, teacher training, PCSC capacity building, and teaching and learning material development and dissemination. Each strategy involves multiple distinct activities, described below. 2.2.1. MOGE Capacity Building TTL works with the MOGE through its existing structures and systems to reinforce its capacity to train, manage, plan, monitor, and evaluate community school progress toward improved education standards, and to diffuse literacy and community school policy updates throughout the MOGE structure. At data collection, TTL supported the MOGE to: x Incorporate reading assessment into routine MOGE monitoring of community schools via eEGRA Instruct, an Education Development Center, Inc., proprietary software program designed for formative monitoring and feedback of reading performance at the school level (2014). As part of this activity, TTL provided a 4-day training for provincial and district MOGE Improved reading among learners in community schools The MOGE provides more support to community schools Community schools have skilled teachers and managers PCSCs are mobilizing school resources Parents are providing a conducive home environment for literacy Community schools have and use teaching and learning materials TTL advocates to the MOGE for increased support to community schools TTL supports the MOGE to train head teachers and teachers TTL builds PCSC capacity in school management and community mobilization TTL supports the MOGE to develop and disseminate teaching and learning materials Time to Learn Project Year 4 Performance Evaluation Report 7 officials on administering eEGRA Instruct and distributed 137 eEGRA Instruct-equipped laptops/netbooks to MOGE administrators who were expected to train ZICs in their districts. x Form a community school steering committee to coordinate MOGE support to community schools across departments and with donors. The steering committee has conducted two national community school symposia (2014 and 2015), revised the 2007 operational guidelines for community schools to improve the policy framework governing them (2013-2014), modified the 2015 National Policy on Education to ensure the inclusion of the Community School Education Act (in progress), and advocated for the inclusion of community schools in the 2016 Zambian National Budget. x Improve early grade reading policy to provide a conducive framework for the teaching and learning of literacy. Through routine interaction with MOGE officials at all levels of the central ministry, TTL technical specialists have advised on a number of strategies, processes, systems, curricula, and teaching and learning materials that support community schools and affect early grade reading. 2.2.2. Teacher Training Designed in collaboration with the central MOGE, TTL conducts training that cascades down through the MOGE teacher education structures from provincial, district, and zonal levels, ultimately reaching community school teachers. The TTL training listed below included variations on this cascade model, as appropriate based on content and resource constraints. Exhibit 9 summarizes the number of individuals who received TTL training at all levels of the cascade. The following training events were conducted between project inception and data collection: x The Quick Start Literacy Program trained head teachers in literacy instruction basics through the following modules: Comprehension, Fluency, Phonemic Awareness and Phonics (word decoding), Read Aloud (oral passage reading), and Vocabulary and Spelling (2013; 2-day training). x Education Leadership and Management (ELM) training provided head teachers and PCSC chairs with skills in managing resources, information, and records; conducting and supervising school-based assessments; assessing effective teaching; school improvement planning; monitoring and evaluating school performance; and providing psychosocial counseling, environment, health, and hygiene education (September 2013 to March 2014; 2-day training). x Zonal Training and Teacher Learning Circles included three modules (1 day each) as part of a CPD program: Reading (read aloud) and Writing in 2013 and Alphabet Sounds (letter sounds) in 2014. Zonal training participants were head teachers who were expected to train their teachers in the same material through teacher learning circles at their schools. x CPD on School-based Assessment for head teachers and teachers focused on how to monitor learner progress and intervene accordingly. The workshop also reviewed teaching methods for literacy instruction and provided an orientation to the MOGE’s Primary Literacy Program (2014; 2-day training). x Early Grade Reading Stepping Stone Program was loaded on Nokia 111 mobile phones, with one phone provided to each school. The Stepping Stone platform is proprietary software from Education Development Center, Inc., that provides video demonstrations of literacy lessons, with short quizzes and other literacy resources for teachers to use in preparing lessons (Exhibit 8). TTL conducted a 2-day training for district resource center coordinators, ZICs, and head teachers on how to use the program to prepare literacy lessons. In turn, head teachers Time to Learn Project Year 4 Performance Evaluation Report 8 were expected to train teachers at their schools. Half of the TTL districts received training in 2015, and there are plans to cover the remaining districts in 2016. EXHIBIT 8: EARLY GRADE READING STEPPING STONE PROGRAM TRAINING CONTENT 2.2.3. PCSC Capacity Building The PCSCs that manage community schools generally comprise parents, teachers, and prominent community members. Community school teachers are accountable to PCSCs, and this accountability is widely regarded as a major strength of community schools. TTL has built capacity of PCSCs through three primary activities: x Orientation on the Operational Guidelines for Community Schools was provided through a 2-day training for two members of each PCSC after the revision of the guidelines was completed (see Section 2.2.1 on MOGE capacity building). The orientation gave an overview of the policy framework that governs community schools in order to better position the PCSCs to advocate with the MOGE on behalf of their schools (2013-2014). x Community Mobilization to Support Literacy was a 2-day workshop to help PCSC members promote more supportive home literacy environments within their communities. Two members of many PCSCs participated, after which they were expected to conduct community￾level training to increase parents’ awareness of the importance of education and how they could help their children learn to read (2013-2014). x Design of School Improvement Plans by PCSCs took the form of support from ZICs at planning workshops, as a follow-on to the 2013 and 2014 ELM training (see Section 2.2.2). ZICs guided the PCSCs through a self-assessment process, needs identification, and planning for future improvement. TTL plans to continue supporting ZICs to make regular community school visits to help PCSCs monitor their schools’ implementation of improvement plans (2015). Grade 1-2 Video Lessons • Alphabet sounds • Letter caps • Syllable sliders • Word-building • Read aloud • Question cubes • Teacher/parent checklist • Group picture • Stories from pictures • Class vs. teacher Grade 3-4 Video Lessons • Alphabet sounds • Letter caps • Syllable sliders • Word-building • Read aloud • Question cubes • Teacher/parent checklist • Group picture • Stories from pictures • Class vs. teacher • Look and say • Language experience Letter Sound Recordings • CiNyanja • iCiBemba • Chitonga • Body alphabet: • CiNyanja • iCiBemba • ChiTonga TLA Tutorials • Syllable slider • Question cubes • Making a small book • Pictures from stories Time to Learn Project Year 4 Performance Evaluation Report 9 EXHIBIT 9: INDIVIDUALS WHO RECEIVED TTL TRAINING2 Training Head Teacher Teacher PCSC ZIC MOGE Unknown Total Quick Start 514 553 10 425 94 45 1,641 Writing 662 977 18 107 4 29 1,797 Reading 441 713 13 81 7 7 1,262 Alphabet Sounds 194 285 5 15 2 9 510 ELM 1,016 287 69 36 52 90 1,550 Operational Guidelines for Community Schools 1,240 317 1,394 27 23 258 3,259 CPD on Assessment 907 3,350 8 340 3 19 4,627 Community Mobilization to Support Literacy 92 112 1571 4 14 1 1,794 eEGRA Instruct 3 1 0 27 196 20 247 Stepping Stone 812 1663 0 48 33 0 2,752 TOTAL 5,881 8,258 3,088 1,110 428 478 19,439 Source: 2015 TTL monitoring data. 2.2.4. Teaching and Learning Materials Development and Dissemination To date, TTL has developed and disseminated the following low-cost, easily replicable supplementary reading materials and instructional resources to improve reading instruction in community schools: x Reading/learning kits for learners included story cards, short-story books in local languages (Maiden readers), and leveled CASAS readers3 (2013-present). x TOTAL community library book boxes with an array of books in English and the language of instruction. Book boxes were distributed to 150 community schools through a public-private partnership with TOTAL Zambia (2013-2014). x Instructional resources for teachers included flashcards, the 2012 Zambian Basic Education Syllabi, teaching guides such as the language schedule, and video and audio content in the early grade reading Stepping Stone phone. These resources have been included in training materials disseminated as part of the CPD program for community school teachers (2013-present). 2 These numbers reflect the total individuals trained in each TTL content areas at all levels of the cascade. Individuals may have attended multiple training events. 3 TTL readers are referred to by their publishing company: Maiden Publishing House and Center for Advanced Studies of African Society (CASAS). Time to Learn Project Year 4 Performance Evaluation Report 10 x Educational management materials, including attendance logs, enrollment forms, and daily and monthly continuous assessment booklets, were distributed along with the ELM training (2013-present). According to TTL monitoring data (Exhibit 10), TTL had distributed 474,858 teaching and learning materials as of 2015. The TTL performance monitoring plan specifies a target to distribute 400,000 teaching and learning materials by the end of the project. EXHIBIT 10: TEACHING AND LEARNING MATERIALS DISTRIBUTED Province Flash Cards Continuous Assessment Story Cards Maiden Readers CASAS Readers Total Daily Monthly Central 464 928 1,856 7,424 2,784 102,528 115,984 Copperbelt 296 592 1,184 7,104 1,776 71,568 82,520 Eastern 389 778 1,556 9,336 1,945 83,040 97,044 Lusaka 450 900 1,808 10,848 2,260 23,904 40,170 Muchinga 216 432 864 6,192 1,296 46,512 55,512 Southern 323 646 1,292 7,752 1,615 72,000 83,628 TOTAL 2,138 4,276 8,560 48,656 11,676 399,552 474,858 Source: 2015 TTL monitoring data. Time to Learn Project Year 4 Performance Evaluation Report 11 3. EVALUATION DESIGN, METHODS, AND LIMITATIONS This internal performance evaluation integrated participatory and utilization-focused approaches with qualitative methodologies. 3.1. Evaluation Design Informed by the midline impact evaluation, this performance evaluation was designed to understand why teachers are more successful in incorporating some TTL-promoted practices into early grade literacy lessons than other practices. The evaluation team applied qualitative methods to better understand the “why” behind teachers’ actions, purposeful sampling to target schools that provided a range of teaching practices, appreciative evaluation to identify successful practices on which to build, and a participatory design process to engage key stakeholders to ensure a relevant and useful evaluation. 3.1.1. Sample This performance evaluation used purposeful sampling to generate data that could yield the richest information in response to the evaluation questions. The sampling frame was drawn from the midline evaluation dataset, which included information on classroom literacy instructional practices, leadership and management practices, and learner performance for 102 schools. The sampling procedure used an iterative, four-step process (Exhibit 11) whereby the number of schools in the frame was further restricted at each step until it yielded the final sample: x Step 1: The schools were sorted into quintiles, based on data from the midline evaluation classroom observation protocol, to identify top- and bottom-performing schools in terms of early grade literacy instructional practices. Top schools were defined as the first quintile and bottom performing schools were defined as the last quintile. Schools outside these two quintiles were removed from the frame. This step identified the most extreme cases to intentionally examine the factors that facilitate and impede changes in teaching literacy. The result was an updated sampling frame of 40 schools. x Step 2: To respond to evaluation sub-question 2c, regarding the Stepping Stone platform, the Step 1 sampling frame was analyzed based on schools that had received this intervention in 2015. EXHIBIT 11: SAMPLING STEPS Time to Learn Project Year 4 Performance Evaluation Report 12 x Step 3: The Step 2 sampling frame was analyzed against TTL monitoring and midline evaluation data to ensure that all schools in the frame had received TTL teaching and learning materials and at least one TTL teacher training. x Step 4: Fourteen schools were selected from those remaining in the sampling frame. Intentional selection included a mix of top- and bottom-performing schools and took into account school accessibility for the data collection plan. Exhibit 12 presents the final sample. EXHIBIT 12: SCHOOL-LEVEL SAMPLE OVERVIEW Province Performance Quintile Total (number of schools with Top Bottom Stepping Stone) Eastern 3 2 5 (3) Muchinga 4 0 4 (1) Southern 2 3 5 (4) TOTAL 9 5 14 (8) 3.2. Methods The evaluation team collected data in September and October 2015 using the following methods: semi￾structured interviews with grade 1 and 2 teachers, head teachers, and MOGE zonal staff; focus group discussions with teachers in each province; and usage data for the early grade reading Stepping Stone program. The list below summarizes these methods and tools; Annex 2 presents the tools. The evaluation team also referenced several documents, which are listed in Annex 3. x Semi-structured interviews: Available grade 1 and 2 teachers, head teachers, and zonal MOGE officers participated in semi-structured interviews at each sample school. Each interview focused on three key areas—classroom practice, classroom management, and teacher support—emphasizing how respondents understood literacy and implemented or supported literacy instruction. Exhibit 13 presents the number of interviews by province and stakeholder group, drawing a distinction between head teachers and teachers who were volunteers and those who were paid by the government. x Focus group discussions: Teachers from nearby community schools participated in one focus group discussion in each of the three provinces. The discussions explored literacy instructional practice, including methods and activities used in literacy lessons and the types of tools and resources needed to improve literacy instruction. Exhibit 14 details the number of respondents who participated in focus groups. x Early grade reading Stepping Stone usage data: Where available at the schools visited, the evaluation team downloaded Stepping Stone usage data logs from the Nokia 111 phones’ memory cards. The logs provided basic information on which video lessons users had accessed and for how long. The evaluation team triangulated this information with interview data. Time to Learn Project Year 4 Performance Evaluation Report 13 EXHIBIT 13: SEMI-STRUCTURED INTERVIEW RESPONDENTS Province Head Teacher Teacher MOGE Total Government Volunteer Government Volunteer Eastern 3 2 2 3 5 15 Muchinga 2 2 2 2 4 12 Southern 4 1 3 4 5 17 TOTAL 9 5 7 9 14 44 EXHIBIT 14: FOCUS GROUP DISCUSSION PARTICIPANTS Province Head Teacher Teacher Total Government Volunteer Government Volunteer Eastern 2 0 4 4 10 Muchinga 0 1 0 6 7 Southern 0 0 11 0 11 TOTAL 2 1 15 10 28 To capture semi-structured interview and focus group discussion data, the evaluation team used a dedicated note-taker and an audio recording device. The team independently validated the notes by comparing them with the audio recording and producing a final electronic version before beginning the analysis. When a language other than English was used, a data collector fluent in that language translated the data into English; as needed, notes were validated by a third party fluent in the language. 3.3. Data Collectors All data collectors were MOGE officials or TTL staff. They received 4 days of training on the purpose of the performance evaluation, the role and value of qualitative research, basic techniques and best practices in qualitative methods (including interview and focus group facilitation techniques, probing skills, and appropriate note-taking), research ethics and informed consent, tool-specific skill development (including piloting the tools for 1 day in urban community schools in Lusaka), and an introduction to the fieldwork procedures specified in the field guide. 3.4. Data Analysis The evaluation team coded and analyzed transcripts from individual interviews and focus group discussions in Dedoose (an online, cross-platform application for qualitative and quantitative analysis) for both content and thematic analysis. To enable analysis by key subpopulations, all data were disaggregated by sex, province, district, stakeholder group (MOGE, head teacher, and teacher), top or bottom school, years in position, and volunteer or government-paid teacher. Data from all sources and stakeholders were triangulated to verify emerging themes. The early grade reading Stepping Stone program usage data were analyzed using Microsoft Excel. Time to Learn Project Year 4 Performance Evaluation Report 14 3.5. Limitations This evaluation encountered a number of limitations. When asked about training received, respondents were often unclear, simply citing the training they had most recently attended, and were often unable to discern or recall differences between TTL training and training conducted by other projects or organizations. Both limitations were more pronounced with older TTL training (e.g., Quick Start and the CPD series that occurred more than 2 years ago) than its more recent technology-based training (e.g., Stepping Stone). Qualitative data generated from interviews and focus groups are highly dependent on the skills of the data collector. Although TTL’s commitment to engaging MOGE officials and TTL staff builds their capacity over time, using data collectors who have not had substantial prior field experiences could affect data quality in the short term. Performance of teachers and head teachers during literacy lessons was used as sampling criteria to identify “high performing” and “low performing” teachers. However, a number of grade 2 teachers who were part of the midline evaluation sample were no longer at the schools visited for this evaluation. The data collection methods captured respondent perceptions and cannot determine statistical causality. The sampling procedure was designed to capture the breadth of factors affecting TTL and not to infer findings to the population of TTL schools. Time to Learn Project Year 4 Performance Evaluation Report 15 4. FINDINGS The project year 4 performance evaluation findings, presented below and organized by the two key evaluation questions, provide an understanding of: x How teachers and head teachers understand and implement early grade literacy instruction x What supports and hinders early grade literacy instructional practice x How teachers and head teachers perceive CPD programs x What improves early grade literacy instructional practices and ELM x What can be improved to inform what TTL should continue and do more of, less of, or differently? Exhibit 15 presents the key terms referenced in the findings and used by the MOGE in Zambia (and internationally) to describe and define the key components of literacy instruction. EXHIBIT 15: KEY COMPONENTS OF LITERACY Skill Description Phonemic Awareness Ability to “hear” sounds and manipulate them orally (e.g., put sounds together, break words apart into sounds, and identify rhyming words, likenesses, and differences in spoken words) Phonics Ability to associate written letters (symbols) with their corresponding sounds Oral Reading Fluency Ability to read orally with accuracy, speed, and expression Vocabulary Ability to understand the meanings of words and use them orally and in writing Comprehension Ability to understand the meaning of what is read or heard Source: MOGE 2013a. 4.1. Evaluation Question 1: What Supports and Hinders Community School Teachers in Implementing Effective Early Grade Literacy Instructional Practice? Exhibit 16 maps the findings to the sub-questions they help answer for key evaluation question 1. Some findings correspond to more than one sub-question. Time to Learn Project Year 4 Performance Evaluation Report 16 EXHIBIT 16: FINDINGS VIS-À-VIS EVALUATION QUESTION 1 Sub-question Findings a) What enables and constrains community school teachers to incorporate all seven core pillars of early grade literacy into their lessons? 1: Phonemic awareness and phonics were seen as the most important and well￾understood, but knowledge of other literacy components was generally weak. 2: Many teachers’ lack of oral and written fluency in their locale's language of instruction hindered accurate teaching of letter sounds and comprehension. 3: Comprehension and writing were viewed as time-consuming, resource￾intensive, and difficult to teach. 11: Teachers reported being behind schedule due to a variety of factors. 18: In general, ZICs’ knowledge of the literacy components was limited, but ZICs considered phonemic awareness and phonics the most important. b) In what ways do teaching and learning materials used by teachers support early grade literacy lessons? 9: Government-issued textbooks and teacher’s guides were the most-used resources in preparing for literacy lessons, whereas locally sourced materials were used to supplement the limited number of classroom resources. 10: A lack of teaching and learning materials impeded teachers’ ability to teach literacy. c) To what extent are teachers using the Stepping Stone mobile phone platform? What has enabled and constrained teachers’ use of the Stepping Stone mobile phone platform? 4: ZICs, head teachers, and teachers greatly appreciated the content of the early grade reading Stepping Stone program, particularly for syllables and letter sounds. 5: The extent of usage of the Stepping Stone phones was difficult to discern from the data. 6: Teachers did not access the full range of early grade reading Stepping Stone content over the 7 or 8 months following the training. d) What factors facilitate head teacher classroom observation and teacher group meetings to result in changes in instructional practice? 7: Trainings and the new literacy method have supported teachers to improve literacy instruction. 8: Teacher group meetings were held regularly and fostered peer learning. 13: ELM and literacy training have helped head teachers engage the community and enhance overall school supervision. 4.1.1. Teachers’ Knowledge, Skills, and Attitudes Finding 1: Phonemic awareness and phonics were seen as the most important and were well-understood, but knowledge of other literacy components was generally weak. During interviews and focus group discussions teachers and head teachers, especially volunteers, found it difficult to list and describe the five key literacy components (see Exhibit 15), even when prompted by the interviewer. For example, when explaining comprehension, teachers sometimes described it as telling a story with the sound of the day and did not include questions to ensure that the learners understood the story; this would, therefore, be considered a phonemic awareness activity. Some grade 1 and 2 teachers considered any instance of storytelling to be listening comprehension even without ensuring learner understanding. Time to Learn Project Year 4 Performance Evaluation Report 17 The literacy lessons teachers described varied little between grades 1 and 2. However, teachers and head teachers described phonemic awareness and phonics as the most important literacy skills and the focus of grade 1. Comprehension was identified as less important overall, but a few teachers and head teachers said it had a greater role in grade 2 classes, where learners would begin to read independently. Teachers rarely spoke about fluency or vocabulary and these components’ role in a literacy lesson. When they were mentioned, they were often discussed inaccurately. … of course, we are looking at the letter sounds [for grade 1]. When they know the letter sounds, it will be very easy for them to read … The same are also important for grade 2. For grade 1, you are building a step for the rest of the grade. —Government head teacher All the skills are important [for grade 1], but phonics skills are very important because once the child can identify the sounds, the letter becomes easy to read. For grade 2, they also follow the same major skills. The same methodology. Mostly of the sounds they have acquired. —Government head teacher Grade 1 and 2 teachers alike spoke most often about introducing the sound of the day, combining the sound of the day with vowels to create syllables, and then combining syllables to create words. Teachers said they routinely used songs, stories, poems, and locally available resources to introduce the sound of the day. Very few teachers and head teachers included comprehension activities when describing their literacy lessons. Those who did include comprehension were primarily government teachers; they described it as asking learners to recall a story read to them to see if they could answer questions. Teachers mostly spoke about comprehension as a listening activity, rather than reading activity. The literacy lesson plan, the first thing, I start with a song, then revise the previous day’s work, which is the sound we learned the previous day. Then the sound of the day is introduced, then we go through the sound, write on the board, then after writing, teach the letters as sound. Now to syllables, add sound and a vowel to come up with syllables … No comprehension at this stage. Maybe words with two and three syllables words, they can read. —Government grade 1 teacher When asked to reflect on why they were incorporating more letter sounds and decoding activities in their lessons than reading and comprehension activities, teachers spoke of the importance of letter sounds as basic to learning literacy. Some respondents elaborated that when learners know the sounds, they will be interested in reading. Because letter sounds are very important. It’s the foundation. The children they have to know letter sounds. Because when they know the letter sounds, they will be able to make syllables to make words even sentences. So the letter sounds are very important. —Volunteer grade 1 teacher You can’t just go from nothing to reading. You had to start with letter sounds, after that they can write and read. —Government grade 2 teacher Grade 1 and 2 teachers spoke most often about introducing the sound of the day, combining the sound of the day with vowels to create syllables, and then combining syllables to create words. Time to Learn Project Year 4 Performance Evaluation Report 18 Finding 2: Some teachers’ lack of oral and written fluency in their locale’s language of instruction hindered accurate teaching of letter sounds and comprehension. Several government teachers said that not knowing the language of instruction for the region where they were working impeded their ability to sound letters properly, read fluently to their learners, and know the meaning of some words. Government teachers deployed to schools in a region where they were not familiar with the local language particularly mentioned this challenge. For example, one head teacher spoke of observing some teachers struggling with the language of instruction. The teachers seemed unaware that they were misleading or confusing learners. Some teachers aren’t Tonga so how can they read fluently and teach [in Tonga]? —ZIC The teacher is not familiar with the local language or the words in local language. So when they read a story, the pronunciation of the words from the story sometimes it is not clear. So students, some of them, they don’t understand clear pronunciation. —Volunteer head teacher ZICs from Southern and Eastern provinces particularly expressed concerned about teachers’ ability to sound letters accurately. Even volunteer and government teachers from the area where they were teaching were observed struggling to translate their oral understanding of the language into a phonological approach to teaching literacy. Some teachers noted that they had never taught grade 1 or had any training on phonological and phonemic awareness. One government teacher said that sounds had not been covered when she was studying for her primary teaching certificate. In addition, during a 1- day workshop on the revised curriculum for all grade 1 teachers in the district, she noticed that teachers had difficulty with sounds when asked to demonstrate lessons. Pronunciation of the sound itself is a challenge to most of the teachers, even government teachers … You will find that the pupil will pronounce the letter sound different, instead the pupil will pronounce the syllable. What children portray is what teachers teach. —ZIC Finding 3: Comprehension and writing were viewed as time-consuming, resource￾intensive, and difficult to teach. Teachers, head teachers, and ZICs from all three provinces said they found conducting comprehension activities challenging. Teachers said that lack of reading materials was a major factor hindering their inclusion of comprehension in a literacy lesson, while a number of ZICs and teachers said that teachers needed to be more creative, develop their own decodable stories, and use locally available materials. One ZIC said that the new grade 1 textbooks did not include specific comprehension activities so teachers had to devise their own, which required teachers to use creativity and initiative. Regardless of the factors, head teachers and teachers viewed comprehension as time-consuming and burdensome. Reading needs material, sometimes it is time-consuming. You may not take every learner to read, looking at enrollment levels. Difficult to go to each pupil to see if they can read. Write on the board, then only 5 minutes to go around to each learner. —Teacher focus group discussion participant Respondents often described writing exercises as challenging because they took up a significant portion of the lesson. Grade 1 and 2 teachers said that letter formation was a major challenge for learners. Teachers said some learners made simple mistakes (e.g., switching letters such as ‘b’ and ‘d’), while others did not write clearly or would scribble and hold the books upside down. Because learners had difficulty writing, several teachers reported spending 50 to 60 percent of the lesson on a writing Time to Learn Project Year 4 Performance Evaluation Report 19 exercise, since it took a very long time for learners to copy from the board and for teachers to check their work. Writing is still a problem for the learners. They are not writing properly. Sometimes they are turning the books upside down, so I have to go around and correct them. —Volunteer grade 1 teacher I spend most the time on the same writing exercise, but I would love them to be reading. It is not correct for them to always be writing. They should be reading. They take time writing because they normally come late to school. —Volunteer grade 2 teacher ZICs, head teachers, and teachers all highlighted broader challenges affecting classroom literacy activity, such as low staffing and high enrollment, which limited teachers’ class and preparation time. The challenge is from where I come from we are only the two of us against seven classes … I wish you had come to our place so you could see how busy we become. So we feel the phonic part is quite easy for us … I like to concentrate on the reading issue, but you need to take your time. We need to prepare even at home. —Teacher focus group discussion participant 4.1.2. Experiences Using the Early Grade Reading Stepping Stone Program Finding 4: ZICs, head teachers, and teachers greatly appreciated the content of the early grade reading Stepping Stone program, particularly for syllables and letter sounds. During interviews, ZICs, head teachers, and grade 1 and 2 teachers from the eight sample schools who were part of the early grade reading Stepping Stone training (see Exhibit 12) expressed a high degree of appreciation for the content and the phone. Many comments focused on the utility of content related to letter sounds, syllables, and word￾building. Respondents reported that they used the program primarily to prepare for lessons and for instruction on how to make their own teaching materials, but they also mentioned using the phone during lessons. Teachers and head teachers spontaneously reported using the phone’s videos to teach word-building, syllables, and letter sounds, highlighting the phone’s utility for teaching sounds. Respondents credited the program with facilitating literacy in their schools, and, in at least one school, with learners’ learning to pronounce sounds better than government school learners. Several found the program useful in the absence of materials for the new literacy program, because it had “everything the teacher needs to teach the correct sound.” Learners were seen to have benefited because the teacher could teach the right sounds and syllables. The school teacher uses the phone with other activities … and when he notices that he is missing the point he checks in the phone when teaching. —Volunteer head teacher When teaching learners about letter sounds, I get the phone and operate it for learners to hear the sound; then I sound it and ask learners to repeat. —Government head and grade 1 teacher The Stepping Stone module on the use of letter sounds was spontaneously mentioned as useful more than any other activity. Time to Learn Project Year 4 Performance Evaluation Report 20 Teachers from all three provinces spoke about the usefulness of the early grade reading Stepping Stone videos for preparing and teaching literacy lessons. The module on the use of letter sounds was spontaneously mentioned more than any other activity, followed closely by the syllable slider. Teachers who were unfamiliar with the local language found the letter sounds useful; more government teachers than volunteer teachers reported this. Blended sounds were mentioned as particularly difficult for teachers unfamiliar with the local language. Teachers and head teachers said they had used the early grade reading Stepping Stone content to practice the letter sounds on their own and with other teachers so that they would not make mistakes when teaching. Related to lesson preparation, teachers also appreciated content on how to create teaching materials. Many teachers specifically mentioned that the syllable sliders video was useful and that the teaching tool was a fun and practical (i.e., easy to make) way to teach syllables. Sometime we used to make mistakes pronouncing sounds. It [the phone] has really helped us. —Government grade 1 teacher Syllables sliders have helped. It has given new ideas, no longer a boring way to teach literacy. Can change activities. —Government grade 1 and 2 teacher Finding 5: The extent of usage of the Stepping Stone phones was difficult to discern from the data. Data from the mobile phones’ memory cards indicated regular and ongoing use of the early grade reading Stepping Stone program in some schools, while data from other schools showed that the phone had been used during training and to learn to make teaching and learning materials. In a third group of schools, usage findings were inconclusive. Of the eight sample schools that were included in TTL’s early grade reading Stepping Stone phone training, the phone was present and functioning at three schools. Only one school had the phone available and charged when the evaluation team visited. Two phones were not charged and therefore could not be turned on or demonstrated. Usage data from these three schools indicated that two schools had used the Stepping Stone application no more than 10 times in the 8 months since the TTL training. At the third school, data showed that a range of activities had been accessed repeatedly on about 46 occasions. At the other five schools, the phone was not available. Head teachers gave the following explanations for the unavailability of the phone at time of visitation: two phones had been stolen, two were left at the head teachers’ homes, and one had experienced a critical hardware failure. Some teachers also complained that the phone was often locked in the head teacher’s office or said they could not operate the phone without the head teacher’s assistance. The extent of usage was difficult to discern based on the interviews. Some respondents stated that they used the phone frequently, but other respondents were unclear as to whether they were describing a small handful of instances immediately following training or repeated and ongoing use of the phones. Finding 6: Teachers did not access the full range of early grade reading Stepping Stone content over the 7 or 8 months following the training. Interviews, focus group discussions, and usage data indicated that a small portion of the early grade reading Stepping Stone program’s early grade reading content had been accessed in the 7 or 8 months following the training. Comments from grade 1 and 2 teachers, head teachers, and ZICs in all eight Time to Learn Project Year 4 Performance Evaluation Report 21 schools focused primarily on content related to phonics and phonemics. Some teachers were unaware of programs other than letter sounds on the phones. According to the TTL program staff, during the early grade reading Stepping Stone phone training, Head teachers were instructed to conduct 16 hours of training with teachers covering at least phone orientation, letter sound recordings, alphabet sounds, syllable slider, word-building, read aloud, and question cubes so teachers would at least access that content (and with the hope that they would try other modules). The usage logs indicate that two schools accessed letter sounds, alphabet sounds, word building, and syllable sliders over the course of one training session. The third school accessed the suggested modules regularly, as well as some additional content. However, usage and interview data indicated little exploration of the full range of content. Exhibit 17 highlights the small portion of content that was used (red text), compared with the overall content available. 4.1.3. Factors that Support Literacy Instruction Finding 7: Training and the new literacy method supported teachers to improve literacy instruction. Grade 1 and 2 teachers from all three provinces said training has positively influenced their classroom practice, namely how to prepare lesson plans, develop and use weekly schedules, use timetabling, teach alphabet sounds and blending syllables, and work with slow learners. Teachers specifically mentioned recent training events that had helped them learn how to assess their learners, measure their own “effectiveness and weaknesses,” submit week 5 and week 10 assessments, and use the “red level” tracker. Training cited most was also the most recent training, specifically TTL’s Stepping Stone training (2015); TTL’s CPD program, which covered assessment and the Primary Literacy Program (2014); and MOGE’s training on “Grade 1 Numeracy and Literacy Assessment” (2015) and “Primary Literacy Program Approach to Teaching Reading in Local Languages” (2014). Grade 1-2 Video Lessons •Alphabet sounds •Letter caps • Syllable sliders •Word-building • Read aloud • Question cubes • Teacher/parent checklist • Group picture • Stories from pictures • Class vs. teacher Grade 3-4 Video Lessons • Alphabet sounds • Letter caps • Syllable sliders • Word building • Read aloud • Question cubes • Teacher/parent checklist • Group picture • Stories from pictures • Class vs. teacher • Look and say • Language experience Letter Sound Recordings •CiNyanja •iCiBemba •Chitonga • Body alphabet: • CiNyanja • iCiBemba • ChiTonga TLA Tutorials • Syllable slider • Question cubes • Making a small book • Pictures from stories EXHIBIT 17: EARLY GRADE READING STEPPING STONE PROGRAM CONTENT ACCESSED (RED TEXT) VS. AVAILABLE Time to Learn Project Year 4 Performance Evaluation Report 22 The training on literacy was some help to me. It gave me an idea on how it should be done, how to teach literacy and numeracy. After the training, I was a person transformed into a person who really wanted to teach in the way I was told. —Volunteer grade 1 teacher Because when I was starting I didn’t know how to go about a literacy lesson, so it was from there [training] I learned how, even how to prepare a lesson plan, even the weekly schedule, and how to use the weekly schedule. I didn’t know how to go about it. After the training, [I] make two lessons per week. —Government grade 1 and 2 teacher MOGE officials said teacher training generally provided a strong foundation for community schools’ literacy instructional practice and consequently influenced learner outcomes. ZICs highlighted that because of the training, teachers were able to transition to the revised curriculum, which emphasized familiar language instruction and the phonetic approach to literacy. One ZIC said that this approach to teaching literacy was effective and easy for teachers to implement because it was straightforward and simple to follow. ZICs also spontaneously mentioned the Stepping Stone mobile phone training, particularly for helping teachers understand the new approach to literacy while providing tools for its implementation. After bringing the methodologies and new curriculum some [teachers] have come out in the open to say those methodologies are helping the learners to read. Because the same methodologies used to teach a Grade 1 can be used to teach other learners from other grades who don’t know how to read. —ZIC Finding 8: Teacher group meetings were held regularly and fostered peer learning. A number of government-supported head teachers in top and bottom schools in all three provinces said that teacher group meetings were held every 2 weeks at their schools, and some specifically followed the MOGE’s School Program of In-service for the Team (SPRINT). Volunteer head teachers commonly reported that teacher meetings were held irregularly throughout the year, each term, weekly, or, for a small few, never. We do have lesson study. We have this chart [SPRINT program] I once did with them like the syllable slider, when I find something I always try to share with them and give them a guide. When these two teachers came, they didn’t have any methodology, they just acquired [it] here. —Government head teacher Every term we have such meetings. The other term we had a lesson demonstration. The outcome was teachers were able to see where they had problems. —Volunteer head teacher Teachers and head teachers stated that teacher learning circles or teacher group meetings allowed for learning from fellow teachers, discussion of challenges, and sharing of materials and knowledge from training. Head teachers particularly noted that these meetings focused mainly on lesson plan preparation, schemes of work, lesson presentation, learner performance, and administrative issues. Several teachers also mentioned that such meetings were an opportunity to discuss the Primary Literacy Program and its components, often a result of attendance at a zonal level training. The majority of respondents described teacher meetings as forums for sharing training materials and skills, such as discussing the revised curriculum or Stepping Stone content. Respondents at one top school in Muchinga Province said that meeting topics included alphabet sounds, syllables and word building, making syllable sliders, how to teach in local languages, and decodable stories. However, this type of specific and regular focus on literacy components in meetings was less common. Time to Learn Project Year 4 Performance Evaluation Report 23 Meetings tended to cover a wide range of topics. A few head teachers discussed conducting more intensive workshops for teachers on similar topics, stating that “1 hour [in a teacher group meeting] is not enough to grasp concepts.” They really help us in terms of performance in pupils when you see how fellow teachers use certain objects. Then you go straight to your class and use as well. We can learn from each other. —Government grade 1 teacher The teacher meetings we have had them just the two of us like I said earlier on that we discuss how best we can improve on some challenges we may have. —Volunteer grade 1 and 2 teacher Finding 9: Government-issued textbooks and teacher’s guides were the most-used resources in preparing for literacy lessons, whereas locally sourced materials were used to supplement the limited number of classroom resources. Teachers and head teachers spontaneously said that the MESTVEE grade 1 and 2 textbooks, including their teacher’s guide (when available), were the most useful for teaching literacy lessons, because they provided details for each lesson, activities to use in the classroom, and a weekly schedule to follow. Even though many schools received a limited number or were missing several term books (e.g., grade 2, term 3 textbooks), teachers relied heavily on these materials to understand and prepare daily lessons. They were seen as clearly articulating the MOGE’s methodology for literacy instruction. Teachers and head teachers reported that literacy lessons were supplemented with teacher-made flash cards and locally available resources to introduce letter sounds, associate sounds with familiar items, and allow learners to create nonsense and real words out of syllables. Respondents expressed pride in using locally available materials as a way to supplement the limited availability of books and materials. The literacy books for grade 1 and 2 [are] the most useful and they are straight forward because in the teacher’s guide the lesson plans are there so you just go through the lesson plan and prepare so the work is not very difficult. —Volunteer head teacher The book [government textbook] is the one I depend on. It is all I have. It is the floor of the lesson. —Government grade 1 and 2 teacher Flashcards make children remember easily what was learned the previous day and for this I like them because children do not forget what they were taught. —Volunteer grade 1 and 2 teacher 4.1.4. Factors that Hinder Literacy Instruction Finding 10: A lack of teaching and learning materials impeded teachers’ ability to teach literacy. Teachers, head teachers, and ZICs all spoke about the need for more textbooks and story books to enable them to help children learn to read. There seemed to be more expressions of this need in Eastern and Southern provinces than in Muchinga. Respondents singled out a shortage of textbooks as a Respondents expressed pride in using locally available materials as a way to supplement the limited availability of books and materials. Time to Learn Project Year 4 Performance Evaluation Report 24 major challenge in all three provinces. For example, one class received 10 copies for 50 learners in grade 1. Respondents at other schools reported an even higher ratio of learners to textbooks. Teachers further observed that grade 1 textbooks did not include specific comprehension activities; teachers had to come up with their own. The books are few. The work becomes easier when there are many books. I give the pupils books so that they are also able to see what is in the books exercise … That is a challenge also because we have few books. —Volunteer grade 1 teacher Although they reported being comfortable using government-issued textbooks and teacher’s guides, teachers reported being much less familiar with supplementary reading materials. A number of grade 1 and 2 teachers were completely unaware of any supplemental materials, including those provided by TTL, even though they were available at their schools. Some teachers said that the level of text in the materials was too complicated for grade 1 and 2 learners, and some grade 1 teachers stated that they never used supplementary reading materials because they were not appropriate for grade 1. One ZIC stated that teachers preferred MOGE-provided books to TTL books because the MOGE books included specific steps to guide the teachers and learners through the lesson. Respondents still requested additional story books to use during literacy lessons. Beyond the question of books in the need for teaching and learning materials, markers and flip charts were cited most often as lacking, especially in Southern Province. Needed supplies that were cited less frequently were manila card stock (to make flashcards and syllable sliders), word cards, illustrated charts, bottle caps, and pens and pencils. I need flash cards with already written syllables, even with pictures. Cards with words would help. Markers and flip charts, Teacher’s Guide. Now just teach however I feel, but maybe the Teacher’s Guide has other suggestions on how to teach. —Government grade 1 and 2 teacher Finding 11: Teachers reported being behind schedule due to a variety of factors. When asked, teachers and head teachers said that they were behind schedule in their grade 1 and 2 literacy lessons more often than they reported being on schedule. When referencing the schedule, most teachers clearly referred to the Zambian National Literacy Framework and the associated weekly schedules, although some teachers referenced other documents or it was not clear what schedule they were referencing. More teachers and head teachers from top-performing schools were able to discuss their progress in terms of the schedule, regardless of whether they felt on or behind schedule, than teachers from bottom-performing schools. The reasons given for being behind schedule were gaps in staffing, teacher turnover, repeating lessons because learners were behind, and materials arriving too late at the school. Several teachers discussed taking on a class that had been out of session due to lack of teachers. ZICs, head teachers, and teachers routinely noted learner absenteeism, or starting the school year or term late, as a major hindrance to literacy acquisition. If a learner missed 3 or 4 days of classes, they could fall behind quickly. The reasons given for being behind schedule were gaps in staffing, teacher turnover, repeating lessons because learners were behind, and materials arriving late at the school. Time to Learn Project Year 4 Performance Evaluation Report 25 We are a bit behind, we are not moving with the syllabus because teachers are forced to repeat lessons where learners have not understood. It could also be because teachers are slow in teaching because of their own educational background. —Government head teacher If the children have not performed well I do repeat the sounds, the letter sounds so that [they] understand that is when I can move a step ahead. I don’t just move to another sound. I make sure they understand. —Volunteer grade 1 teacher 4.2. Evaluation Question 2: What Kinds of Leadership and Management Activities Support Community School Teachers to Teach Early Grade Literacy? Exhibit 18 maps the findings to the sub-questions that they help answer for key evaluation question 2. Some findings correspond to more than one sub-question. EXHIBIT 18: FINDINGS VIS-À-VIS EVALUATION QUESTION 2 Sub-question Findings a) To what extent are head teachers using educational leadership and management techniques to support teachers and improve instructional practice? 12: Zones support community schools and literacy by coordinating zonal trainings, managing the assessment data, and providing materials. 13: ELM and literacy training have helped head teachers engage the community and enhance overall school supervision. b) Which forms of continuing professional development are teachers most interested to see continued by the MOGE after the project ends? 7: Training and the new literacy method supported teachers to improve literacy instruction. 15: Teachers and head teachers would like ongoing training and monitoring. c) How is routine monitoring by the MOGE, including use of eEGRA Instruct, being used as a formative tool for teacher development and for promoting changes in classroom practice? 12: Zones support community schools and literacy by coordinating zonal trainings, managing the assessment data, and providing materials. 14: Literacy classes were monitored and observed and feedback was generally provided. 16: Distance, limited materials, and teacher turnover and education levels hindered ZICs training support to community schools. 17: Distances to community schools are a leading barrier to monitoring and classroom observation. 18: In general, ZICs’ knowledge of the literacy components was limited, but ZICs considered phonemic awareness and phonics the most important. Time to Learn Project Year 4 Performance Evaluation Report 26 4.2.1. Leadership and Management Practices that Support Literacy Finding 12: Zones support community schools and literacy by coordinating zonal trainings, managing the assessment data, and providing materials. ZICs, head teachers, and teachers spoke about multiple forms of leadership and management support that zones have provided to community schools, in addition to monitoring and observation (covered under Finding 14). Leadership and management practices discussed included those intended to support community schools in general and early grade literacy instruction in particular, and included coordinating zonal training, managing assessment data, and providing teaching and learning materials or free basic materials. Respondents who discussed leadership and management support were from all provinces sampled. More respondents who discussed these forms of support were from top-performing schools, and they discussed leadership and management support in the context of providing this support (ZICs and head teachers). However, many teachers and head teachers also shared forms of support that they received. ZICs reported coordinating in-service activities in community schools in their zones, including participating in MOGE and TTL literacy training and often training head teachers and teachers at community schools. ZICs reported that they were more confident monitoring classroom activity at community schools and providing feedback to teachers as a result of their participation in MOGE and TTL training, and that they had a stronger knowledge base for working with community schools on literacy. This training that I attended, in a way it has developed esteem. When I go to observe the teacher, I know exactly what I go for. Then for a teacher to convince me, maybe he has taught the sound wrongly, I feel more confident to convince teachers they are wrong. —ZIC Then there were activities that needed follow up like assessment. Receiving those trainings helped me do those follow ups. —ZIC When discussing monitoring learners’ progress, both at the classroom level and at the zonal or district level, the most commonly mentioned form of assessment was the 5/10/13 week assessments. This was reflected in comments not only from teachers, who discussed using the assessments at the classroom level, but also from ZICs, who discussed receiving these data as part of their management work. Head teachers discussed both processes, reflecting the fact that many teach early grade classes and play a role in the data management process. Respondents also discussed using the “red level” tracker, a means to categorize learners’ competencies, to summarize assessment results and track aggregate learner progress. The red level tracker was most discussed by ZICs, although classroom teachers also referred to it. No respondents explicitly linked red level tracker data to the 5/10/13 week assessments (or any other assessment). Respondents shared how reports and data were handled through the MOGE structure or, in some cases, how such material was supposed to be handled. Respondents identified a fairly consistent chain, whereby information flowed up from teachers, to head teachers, to zones, and then to the district. Some respondents acknowledged making decisions or giving feedback to their subordinates based on ZICs reported that they felt more confident monitoring classroom activity at community schools and providing feedback as a result of their participation in MOGE and TTL training. Time to Learn Project Year 4 Performance Evaluation Report 27 this information (or, in the case of classroom teachers, receiving feedback based on information they had submitted). Finding 13: ELM and literacy training have helped head teachers engage the community and enhance overall school supervision. Head teachers from all provinces and from top- and bottom-performing schools reported receiving training from TTL or the MOGE on early grade literacy, management skills, and activities needed to run a community school. Head teachers spoke consistently about specific ways such training enhanced their overall supervision of school activities, design and assessment of the lesson plans, schemes of work, and weekly forecasts of their teachers. Head teachers also stated that training taught them how to engage the community, which some said had resulted in increased learner enrollment at their schools, more volunteer teachers recruited, and community involvement in building teacher housing. Head teachers said training provided them with the basic knowledge to handle finances, human resources, and recordkeeping at community schools, particularly in the context of MOGE policies. Being the only government teacher, if I had not received such training I don’t think I would have lasted this long because communities are difficult to deal with. But with these trainings, I have been able to hold fruitful meeting with them. — Government head teacher Before training, I did not know how to run a school, but I now do. Because of that training [ELM] I’m able to work with my fellow teachers … Previously, I would not delegate duties whenever I left the school, but can do so now as a result of that training. — Government head teacher Head teachers said that early grade literacy training, especially the recent MOGE training introducing the Primary Literacy Program and the National Literacy Framework, had provided the content-area knowledge they needed to apply monitoring skills from management training to grade 1 and 2 literacy lessons. In addition, training from MOGE reinforced assessment guidelines and the foundation of literacy lesson preparation and implementation. Head teachers said that TTL’s Stepping Stone training had provided them with the components of a literacy lesson, such as how to pronounce letter sounds and develop teaching and learning aids to train teachers. The trainings have helped me in the sense I am able to help the teacher in areas where they do not understand … Maybe I was made to demonstrate how to write the weekly focus and the lesson plan. And also we had a demo lesson here, how a lesson is to be handled. —Government head teacher Finding 14: Literacy classes were monitored and observed and feedback was generally provided. ZICs, head teachers, and teachers all said that community school monitoring and literacy lesson observation was happening. More respondents from top-performing schools said they were being monitored and observed than those from bottom-performing schools. Teachers said they had been observed by the head teacher or by the MOGE, but not by both, and that monitoring occurred rarely. One teacher stated that monitoring did not include classroom observation. Head teachers spoke consistently about specific ways the training enhanced their own overall supervision of schools, design and assessment of lesson plans, schemes of work and weekly forecasts of their teachers. Time to Learn Project Year 4 Performance Evaluation Report 28 According to head teachers, teachers, and ZICs, the frequency of monitoring varied quite a bit. In general, ZICs said they monitored once or twice each term, although some said it was less often. I make sure at least two times, two or once. If they are lucky, I will visit them two [times], but other schools it is once because of the same that I’ve already explained, the issue of transport. Once in a term, I at least visit them. If I fail, that is what the ZEST [Zone Education Support Team] is for. —ZIC I think in a term I can manage two times in one community school. At times it’s just once. Like last term, I visited each of the community school[s] once. —ZIC ZICs, head teachers, and teachers in all three provinces—though more from Southern Province—stated that feedback was provided either after their most recent monitoring or observation visit or more generally as a part of every such visit. In the majority of instances, respondents stated this in response to a direct question, although some respondents (most from top-performing schools) mentioned feedback spontaneously. A minority of head teachers and teachers reported that they had not received feedback when they were observed, and one ZIC said he did not provide feedback after using eEGRA Instruct to monitor community schools. One teacher’s response indicated potential differences among respondents’ understandings of what constituted feedback. The teacher explained that the classroom observer merely noted what happened in the lesson, as opposed to giving specific feedback on the strengths and weakness of the lesson. The deputy head came, but never gave any feedback … The observation was of help because I put in more effort when the deputy was there. —Volunteer grade 2 teacher Feedback has not been there because I’ve been very busy with other schedules. —ZIC Respondents said that feedback was usually oral; written feedback was less common. One MOGE official said that they had provided both written and oral feedback. One ZIC said they had used a specific form to guide the observation note-taking process. A head teacher said they had sat with the teacher observed to write stories together that could be used during lessons, as a way to overcome weaknesses the head teacher had observed in the literacy lesson. In another instance, a head teacher used literacy materials during the feedback session to illustrate how activities should be conducted. In at least two cases of no feedback, the teacher noted that the observer had made written feedback, but that a copy was not provided to the teacher. Respondents who gave or received feedback generally discussed strengths and weaknesses, but a smaller number who gave feedback concentrated more on weaknesses to help the teacher focus on areas most in need of improvement. One head teacher discussed providing feedback using the “sandwich technique” in which weaknesses are placed in between strengths in order to start and end on a positive note. The feedback was verbal. First, I praised her before talking about her weaknesses… those are skills we learned at the Educational Leadership and Management training. —Government head teacher Almost all teachers and head teachers who said they received feedback on their lessons said that they found the process helpful. Time to Learn Project Year 4 Performance Evaluation Report 29 Then we sit and discuss the whole lesson. During my observation, I focus mostly on negatives so that person can improve. If you focus on positive things you hinder that teacher. From those negatives, the teachers can learn. —ZIC Core issues identified for improvement included preparation for lessons and lesson planning; managing the classroom to make sure all learners were involved in the lesson, paying attention, and being called on; and details specific to literacy instruction. Within the latter category, most feedback focused on teaching foundational skills such as decoding, building words from syllables, and phonemic awareness. Comprehension was mentioned less often. However, a number of respondents were unable to specify concrete issues discussed during feedback, stating simply that it had focused on providing “encouragement” or “appreciating the teacher’s work.” So we end up discussing this with the teacher, of course, in a humble manner. Because these teachers, if you are harsh, they end up abandoning the children. So even the way you are talking to them, sometimes they take in a way they should not realize that they are making a lot of mistakes. So let’s encourage them to say, “that is good, but do this next time.” —ZIC Almost all teachers and head teachers who said they received feedback on their lessons said that they found the process helpful. Many pointed to concrete changes they had made in literacy lessons as a result. Changes encouraged by the feedback included using group work and pace groups, making greater efforts to involve all learners, and using formative assessment. Some teachers said that just the act of being observed encouraged them to teach better, surprise observations forced them to be prepared, and they put more effort into lessons that are observed. Finding 15: Teachers and head teachers would like ongoing training and monitoring. Teachers and head teachers frequently mentioned the need for any training, additional training, and refresher training to improve their ability to teach and teach literacy well, and for ongoing monitoring to observe their work and provide feedback. When probed for specific training content they feel is needed, however, most respondents had only unspecific requests for “literacy training.” Just more on methodologies about literacy … to have more people who are experienced whether the ZIC to be coming more often to teach us about literacy. —Volunteer grade 1 teacher 4.2.2. Factors that Hinder Support to Community Schools Finding 16: Distance, limited materials, and teacher turnover and education levels hindered ZICs training support to community schools. ZICs and head teachers described a number of factors that made it difficult for them to fulfill their supporting role. These challenges were mentioned spontaneously and in response to direct questioning about difficulties. The challenges mentioned most often were the distance to community schools and the lack of stable volunteer teachers. Distance was a challenge both in terms of consolidating teachers at a central location for training and for ZICs to make post-training follow-up visits to schools. In one case, the community school’s location posed a communication challenge, and in another the school was required to pick up teaching and learning materials directly from the district center, without assistance from the zone. ZICs spoke of this challenge in terms of distance, lack of funds, or means of transportation. Time to Learn Project Year 4 Performance Evaluation Report 30 Head teachers and ZICs said that the high rate of volunteer teacher turnover was frustrating in terms of training. Respondents spoke of the frequency in which they have trained a teacher in literacy instruction only to see the teacher leave, sometimes within just a few months. I have oriented more than four people; all of them are not working with that community school so it’s like my labor is going in vain. So I am only disappointed with that community school. —ZIC We give a period of 2 weeks then we follow-up … I haven’t visited the community schools because of the challenges of transport. —ZIC ZICs reported that they had limited materials to use for cascade training in their zones. For example, they may have been trained with a projector and laptop, but when they returned to their zones they could not conduct the training the same way. Others wished for standard training materials, such as flipcharts and markers. Respondents also pointed to a lack of sufficient teaching and learning materials that correspond to the new literacy methodology. ZICs also expressed difficulties with the low education of teachers, which they perceived as slowing down the training process and requiring follow-up to confirm that teachers were implementing what they had been taught and to establish whether retraining was needed. Respondents pointed out that most volunteer teachers had not completed secondary school and some had completed only grade 7, facts that require trainers to proceed very slowly when presenting content. Related to this, ZICs mentioned challenges in understanding training content. In some instances, ZICs noted that the core training content was often misunderstood (even by themselves) and wished that all volunteer teachers would have completed at least grade 12. The challenges was some of these guys, their education background is not very good. You find that you address them. You may think they have gotten the concept, but at the end of the day when you make a follow up, you find they do something else … Some have not gone to grade 9, some just to grade 7. —ZIC Finding 17: Distances to community schools are a leading barrier to monitoring and classroom observation. ZICs and head teachers spontaneously mentioned lack of transport to community schools as a challenge to monitoring the schools more frequently. The quality of roads, particularly in rainy season, and the distances were both noted as parts of the problem. ZICs mentioned having to find their own solutions to the transportation challenge, such as using personal or borrowed bicycles and motorbikes. This left the ZICs to make the time-consuming arrangements on their own, and some mentioned that they had not received funds for fuel when they used motorbikes. Additional but less-mentioned challenges included ZICs’ and head teachers’ teaching loads, leaving little time for supervisory activities, and a lack of TTL monitoring forms. Both factors were mentioned spontaneously. Respondents spoke of the frequency in which they have trained a teacher in literacy instruction only to see the teacher leave, sometimes within just a few months. Time to Learn Project Year 4 Performance Evaluation Report 31 There was no funding, so I had to use my own funds to monitor to see if they implemented. Sometimes I just ride my bicycle. Some community schools are a bit far, 12 kilometers from here. Sometimes I go on a sacrificial basis. —ZIC Finding 18: In general, ZICs’ knowledge of the literacy components was limited, but ZICs considered phonemic awareness and phonics the most important. Almost all ZICs understood basic approaches to teaching letter sounds, syllables, and word-building. They highlighted phonics and phonemic awareness as the most important aspect of grade 1 and 2 classes and as an area of concern in terms of teacher skill. A large number of ZICs, despite the number of years in their positions, were unable to articulate and describe the five key literacy components. ZICs who were more familiar with the five components could describe more concrete feedback and advice given to teachers across all components. I would tell them to mainly focus on story reading for their learners to be very good readers, they have to read themselves. Maybe from January, even from grade 1 to 7. It is better for a teacher to read to the learners and for grade 1 and 2, it is better they start reading with the learners from the beginning and at the end of the lesson. —ZIC Areas where they do less, that is when it comes to reading the sentences. They just end up with syllables and words. Also reading of stories of learners themselves … They waste a lot of time on syllables, neglect how to write a sentence or allow learners to read on their own. —ZIC Maybe because it is because of the way the new methodology has come, which is just concentrating on sounds. Even them, they’ve just been drilled sounds, sounds, sounds, forgetting even comprehension and writing is very important. —ZIC Time to Learn Project Year 4 Performance Evaluation Report 32 5. CONCLUSIONS AND RECOMMENDATIONS The project year 4 performance evaluation conclusions and recommendations are consolidated in this section. The conclusions synthesize the evaluation findings with information from other available sources, and are organized in two sections, corresponding to the performance evaluation questions. The findings and conclusions were validated at a feedback session held with TTL staff in January 2016. The recommendations included below are based on the conclusions, findings, and content generated during the feedback session. 5.1. Conclusions Evaluation Question 1: What supports and hinders community school teachers in implementing effective early grade literacy instructional practice? The project year 4 performance evaluation findings provide additional validation for the midline classroom observation that community school teachers devoted substantially more instructional time to letter sounds than other literacy skills, and that activities related to comprehension and learners reading during class were almost entirely absent (Falconer-Stout et al. 2015). The performance evaluation independently found that activities emphasizing phonemic awareness, phonics, and syllable and word coding and decoding receive the most emphasis during literacy lessons in grades 1 and 2. Further reflecting this pattern, teachers’ use of Stepping Stone content mostly related to the phonic and phonemic components of literacy. Activities that offered learners the opportunity to read connected text independently, build comprehension, and practice handwriting all received less emphasis during lessons. Overall, the performance evaluation findings indicate several related reasons that help explain these early grade literacy instructional practices. Explanations include factors related to (1) literacy principles and how they have been understood by teachers and in the broader framework of curricular reform in Zambia, and (2) a wider range of factors specific to the Zambian community school context. 5.1.1. Factors Related to the Zambian Literacy Framework Findings indicated a widespread attitude among stakeholders that phonemic awareness and letter sounds are more important than other literacy components during grades 1 and 2, and that these components must be mastered before learners can proceed to work on fluency and comprehension. This hierarchy is reflected in some teachers’ and head teachers’ comments that comprehension had a greater role in grade 2 classes only when learners should begin to read independently. Teachers’ and head teachers’ concerns about being behind schedule also reflected this linear and hierarchical approach to teaching literacy, in which lower-order skills must be mastered before a learner can proceed to higher-order skills. The fact that teachers and ZICs demonstrated this attitude suggests that it is being reinforced by stakeholders at different levels. Moving at the pace of learners was a positive teaching practice that TTL and MOGE training content promoted. However, in the context where learners are failing to master content and teachers are neglecting some literacy components in grades 1 and 2, it may contribute to literacy instruction getting “stuck” at letter sounds and syllables and never making it to comprehension and fluency. Time to Learn Project Year 4 Performance Evaluation Report 33 The evaluation found that the prioritization of phonics was also reinforced by the teaching and learning materials available to community school teachers. For example, teachers generally found that MOGE￾issued grade 1 and 2 textbooks were the most useful materials available, but that these textbooks focused on letter sounds and word decoding and did not include activities to support comprehension. Findings show that teachers were less sure of how to use supplementary reading materials, which are the primary form of teaching and learning materials TTL provided to schools. The MOGE textbook content, which teachers preferred over supplementary reading materials, align with the content teachers deemed more important in Grades 1 and 2. On the other hand, the early grade reading Stepping Stone program included substantial content related to all five literacy components. Even with the broad content offerings, however teachers’ use of early grade reading content in the Stepping Stone program mostly related to the phonic and phonemic components. This nuance illustrates that the role of different teaching and learning materials in teachers’ initial attitude formation is unclear: it is possible that materials contributed to teachers’ prioritization of phonics and phonemic awareness, but it is likewise possible that teachers preferred these materials because they emphasized the components teachers already believed were the most important. Related to this linear understanding of literacy acquisition was a broader misunderstanding of the five literacy components. Many stakeholders, from classroom teachers to ZICs, could not accurately define the five components, and teachers often misidentified the component supported by specific activities. In a context where ZICs (who, as zonal officials, interact most closely with teachers and are, therefore, the “frontline” MOGE support for community schools) were themselves unclear on this core content, it is unsurprising that classroom teachers also struggled to understand this content. This misunderstanding was not ubiquitous, but it was widespread enough to offer a plausible explanation for why some literacy components received less attention in the classroom than others. These factors align with the recent internal TTL report on reading progress in Zambia community schools, which found that: A major impediment … is that the construct of reading is, first, deeply misunderstood and that attempts to teach reading are understood to be sequential rather than holistic; where each strategy is taught in isolation before moving on to the next strategy and where it is hoped that fluency and comprehension will appear at the end of the competency tunnel (Lewis 2015). 5.1.2. Factors Related to Community Schools Findings revealed several factors related to community schools and the MOGE structure supporting them that also helped build understanding of the forces driving instructional practices. Overcrowding and low staffing in community schools resulted in high learner-to-teacher ratios that made perceived time-intensive activities, such as writing and comprehension, difficult for teachers to cover during lessons. High teaching loads meant that most teachers were instructing more than one class, which left limited time to prepare for lessons. In the context of changes to literacy curriculum, it was natural that teachers would need increased time to prepare, but findings showed that current teaching loads did not allow for extra preparation time. Poor learner attendance was also found to hamper literacy acquisition. Instances when learners were absent and teachers had to repeat lessons contributed to the pattern of falling behind schedule. This aligned with observed learner attendance rates documented in the midline evaluation, which found an average attendance rate of just 70 percent among the grade 2 classes sampled (Falconer-Stout et al. 2015). Time to Learn Project Year 4 Performance Evaluation Report 34 Findings also showed that the high rate of teacher turnover among volunteer community school teachers had made it difficult for ZICs to consistently build teachers’ capacity for high-quality literacy instruction over time. Instead of helping teachers improve, ZICs had to retrain new volunteer teachers on a regular basis because previously trained teachers had left. As found in the midline evaluation, volunteer teachers constituted almost 60 percent of the teaching workforce in community schools; on average, teachers were found to have served about 3 years at their community schools (Falconer-Stout et al. 2015). Evaluation Question 2: What kinds of leadership and management activities support community school teachers to teach early grade literacy? The MOGE’s expanded role in community schools was evident throughout the findings. Most schools highlighted monitoring activity by MOGE officials, inclusion of community school teachers in CPD programs at the zonal and district levels, participation in national literacy assessments, and provision of teaching and learning materials, such as the grade 1 and 2 textbooks. Findings showed a fairly consistent way that reports and data were intended to be handled through the MOGE structure. In addition, trained teachers and head teachers on the MOGE payroll were being deployed to community schools. This aligns with evidence from the midline evaluation, which found that the types and amount of MOGE support from zonal and district offices had increased significantly (Falconer-Stout et al. 2015). 5.1.3. Monitoring and Observation Activities Findings showed that monitoring, observation, and feedback were generally happening in community schools, although with inconsistent frequency. Teachers and head teachers found the monitoring, observation, and feedback valuable, indicating that these activities were serving their intended purpose. Head teachers and ZICs saw monitoring and observation as a key component of their roles and responsibilities. Findings showed that many ZICs were not monitoring termly, as they wished, due to demands on their time (e.g., high teaching loads in addition to the functions of their position), distance to community schools, lack of transportation, and lack of funds for transportation. Head teachers also wanted to observe teachers and provide feedback more often, but were frequently pulled away due to high teaching loads and management functions. Feedback was provided verbally or in writing, but inconsistently. Findings showed that ZICs’ feedback tended to be general, with many simply providing appreciation or encouragement. ZICs who did articulate concrete feedback focused on issues of lesson preparation and classroom management. When ZICs gave feedback specific to literacy methods, it most often concentrated on phonemic awareness, phonics, and word coding and decoding. These were the same areas teachers had covered most often and in which ZICs expressed the strongest understanding. It was unclear, however, if ZICs were reinforcing this singular focus in response to what they were observing or because these were the areas in which the ZICs, too, had the strongest understanding. 5.1.4. Administrative and Supervisory Activities The findings point to a number of factors influencing community school management. Government head teachers and volunteer head teachers alike have used the knowledge, skills, and tools they gained from TTL and MOGE training to better supervise the schools generally, assure the delivery of high-quality literacy lessons, observe literacy lessons, and foster school-based learning and discussion among teachers. Teacher group meetings were being held, and were useful opportunities for teachers to learn from each other and address specific school or classroom issues. Government-trained teachers in Time to Learn Project Year 4 Performance Evaluation Report 35 particular have gained valuable approaches to managing the limited resources of community schools and working with the surrounding communities. Occasionally, the distribution of knowledge from training broke down. For example, findings indicated that the early grade reading Stepping Stone phones were often underused. Reported use of the phone was uniformly limited to a small portion of the content, while understanding by teachers and head teachers of how, when, and why to access the Stepping Stone platform to support literacy instruction varied across schools. 5.2. Recommendations 1. TTL and the MOGE should strengthen teacher training modules to clarify how to integrate all five components of literacy instruction into one lesson. x TTL should work with the MOGE to identify ways to strengthen teachers’ ability to include all five literacy components in their lessons and how lessons should progress through grades 1 and 2. Teachers’ attitudes may also need to be targeted to improve their appreciation for the importance of all five components. This could be accomplished by augmenting existing training modules, teaching and learning materials, or monitoring activities. In the community school context, with challenges such as learner absenteeism and limited materials, needs to be considered in order to provide targeted support to teachers on how to negotiate these challenges while progressing through literacy content. Particular focus should be placed on integrating comprehension activities and on how to allocate time to the different components, such as writing. Teachers may benefit from practical ideas for how to conduct comprehension, writing, and learner reading activities in classes with high learner-to-teacher ratios. x TTL should work with the MOGE to create simple aids, which describe the structure and steps of a standard literacy lesson in a simple and straightforward way. TTL could use existing tools, such as the early grade reading Stepping Stone platform, and add content, which shows a full lesson integrating all five components of literacy instruction. x TTL should work with the MOGE to integrate the use of supplemental teaching and learning materials, including the TTL-provided Nokia 111 phone and supplementary reading materials, with MOGE textbooks to create a well-rounded lesson. TTL could use ZICs’ monitoring exercises of teachers to provide detailed instruction on how to support the use of MOGE textbooks in conjunction with other materials. A focus on supplemental materials and their specific uses in a literacy lesson may also need to integrated into additional teacher training activities. 2. TTL should follow up with schools that have received the early grade reading Stepping Stone phone training to better understand and improve usage of the program. x TTL should investigate why the phones are not being used more, why some content is being used more frequently, and why phones are not being made more accessible to teachers. TTL should work with the MOGE to explore factors that are leading to a focus on content limited to phonics and phonemic awareness, as well as head teachers’ approaches to sharing training content and teaching and learning materials equitably with their staff. Time to Learn Project Year 4 Performance Evaluation Report 36 3. TTL should work with the MOGE and head teachers to improve the frequency and the quality of formative community school monitoring. x TTL should work with the MOGE to strengthen ZICs’ knowledge of the five literacy components and build their skills in providing practicable monitoring and observation feedback to teachers. The ability to make concrete recommendations for lesson improvement, such as specific activities and materials to include based on what is observed, is a core skill. As ZICs’ supervisors, zonal heads could also benefit from this content, which could in turn provide legitimacy and support for ZICs’ monitoring of community schools. This could be accomplished by revisiting training for ZICs on literacy instruction and ways to provide targeted feedback to teachers during monitoring exercises, and by extending this training to zonal heads. TTL could also provide specific guides or instruments that support monitoring activities and ensure that all five literacy components are being integrated into a lesson. x TTL should improve its community school monitoring tool to reflect the quality of classroom observation and feedback and develop a system to routinely review data from this tool in order to provide ZICs with feedback on ways to improve the quality of their monitoring activities. Just as teachers need feedback to improve their lessons, supervisors need formal support to improve the quality of the feedback they provide. This should be done in collaboration with the MOGE and include its monitoring tools. x TTL should investigate the most viable mechanisms to produce the desired monitoring of community schools by ZICs. This could include piloting whether increased support to ZICs through travel and lunch allowances or incentives for outstanding performance would lead to commensurate increases in quantity and quality of monitoring activities. TTL should further work with the MOGE to alter ZIC job descriptions and responsibilities to ensure that sustainable mechanisms exist to guarantee community school monitoring activities post-TTL. x TTL should continue to work with head teachers to ensure that classroom observation and teacher group meetings are done regularly and include high-quality feedback and discussion of all five literacy components. x TTL and the MOGE should work with PCSCs to encourage coordination with ZICs and head teachers on monitoring literacy lessons and the school. PCSCs should be encouraged to hold head teachers accountable in the use of resources TTL has provided to community schools (e.g., the Nokia 111 phone and other teaching and learning materials) and to continue to advocate for MOGE support for schools that are missing certain resources, such as textbooks. 4. TTL should work with the MOGE to continue to integrate community schools into the MOGE system. x TTL and the MOGE should investigate ways to ensure that community school teachers regularly attend zonal CPD. The distance between community schools and zonal centers, as well as community school teachers’ limited resources, should be considered when identifying mechanisms for their inclusion. x TTL and the MOGE should establish procedures for ensuring that teaching and learning materials, including MOGE textbooks, are being distributed to community schools and contain the number of books needed. x TTL should work with the MOGE to identify ways of providing financial support to volunteer community school teachers and increase deployment of government teachers to community schools to reduce teacher turnover. A strengthened financial relationship between community Time to Learn Project Year 4 Performance Evaluation Report 37 schools and the MOGE would increase confidence in the management of community schools by head teachers and PCSCs, thereby increasing access to future MOGE grants and materials. Time to Learn Project Year 4 Performance Evaluation Report 38 ANNEXES ANNEX 1: EVALUATION MATRIX......................................................................................... 39 ANNEX 2: DATA COLLECTION INSTRUMENTS ................................................................ 41 ANNEX 3: REFERENCES .......................................................................................................... 53 Time to Learn Project Year 4 Performance Evaluation Report 39 ANNEX 1: EVALUATION MATRIX Key Evaluation Questions Illustrative Measures Data Sources 1. What supports and hinders community school teachers in implementing effective early grade literacy instructional practice? a) What enables and constrains community school teachers to incorporate all seven core pillars of early grade literacy into their lessons? x Teachers’ perception and knowledge of core pillars (and source of knowledge) x Teachers’ expressed confidence and perceived skill in teaching core pillars x Teachers’ stated instructional approaches x Teachers’ stated use of resources and support (lesson plans, syllabi, external guidance, materials, trainings) x Semi-structured interviews with teachers, head teachers, and MOGE x Focus group discussions with teachers and TTL Lusaka￾based staff x Review of student assessments and records (teacher￾and school-level); review and audit of teaching and learning materials (syllabi, curricula, learning schedules); MOGE and head teacher classroom observation records b) In what ways do teaching and learning materials used by community school teachers support early grade literacy lessons? x Teachers’ perception of usefulness of the existing materials x Teachers’ stated use of the existing materials (which ones, how, how often) x Teachers’ and head teachers’ perceptions of the extent to which the use of materials has had an impact on instructional practice x Semi-structured interviews with teachers and head teachers x Focus group discussions with teachers and TTL Lusaka￾based staff x Audit of available teaching and learning materials at schools, and review of TTL monitoring records of distribution and trainings c) To what extent are teachers using the Stepping Stone mobile phone platform? What has enabled and constrained teachers’ use of the Stepping Stone mobile phone platform? x Teachers’ perception of the Stepping Stone x Teachers’ access to and use of the Stepping Stone x Teachers’ and head teachers’ reports of application in the classroom x Teachers’ and head teachers’ perceptions of how the Stepping Stone has improved or changed instructional practice x Semi-structured interviews with teachers and head teachers x Observation of the Stepping Stone use x Stepping Stone usage data x Focus group discussions with TTL Lusaka-based staff x Review of TTL monitoring records of distribution and trainings d) What factors facilitate head teacher classroom observation and teacher group meetings to result in changes in instructional practice? x Teachers’ and head teachers’ perceptions of classroom observation and teacher group meetings x Documented frequency and content of classroom observations and teacher group meetings x Teachers’ and head teachers’ perceptions of how x Semi-structured interviews with teachers and head teachers x Focus group discussion with teachers x Review of head teacher records (classroom observation, teacher group meetings), and TTL Time to Learn Project Year 4 Performance Evaluation Report 40 Key Evaluation Questions Illustrative Measures Data Sources classroom observations and teacher group has improved or changed instructional practice monitoring records on educational leadership and management trainings and materials 2. What kinds of leadership and management activities support community school teachers in teaching early grade literacy? a) To what extent are head teachers using educational leadership and management techniques to support teachers and improve instructional practice? x Head teachers’ and teachers’ perceptions of type and content of leadership and management activities at school (instructional and managerial) x Head teachers’ and teachers’ reports of application in the classroom x Teachers’ and head teachers’ perceptions of how activities have improved or changed instructional practice x Semi-structured interviews with teachers and head teachers x Focus group discussion with teachers Review of head teacher records (assessment tools, classroom observation, teacher group meetings), teacher records (student assessments), and TTL monitoring records on educational leadership and management trainings and materials b) Which forms of continuing professional development are teachers most interested to see continued by the MOGE after the project ends? x Teachers’ and head teachers’ perceptions of professional development activities x MOGE perception of activities x Semi-structured interviews with teachers, head teachers, and MOGE x Focus group discussion with teachers Review of TTL monitoring records of trainings c) How is routine monitoring by the MOGE, including use of eEGRA Instruct, being used as a formative tool for teacher development and for promoting changes in classroom practice? x Head teachers’ and teachers’ perceptions of MOGE monitoring activities x MOGE perception of monitoring activities x Documented frequency and content of activities x Teachers’ and Head teachers’ perceptions of how MOGE monitoring has improved or changed instructional practice x Semi-structured interviews with teachers, head teachers, and MOGE x Focus group discussion with teachers Review of MOGE classroom observation records (including eEGRA instruct), and TTL monitoring records of trainings and implementation DRAFT Time to Learn Project Year 4 Performance Evaluation Report 41 ANNEX 2: DATA COLLECTION INSTRUMENTS Semi-structured Interview Guides At the beginning of all interviews, the evaluators introduced the evaluation and an informed consent statement. While the introduction was altered slightly depending on the stakeholder involved, the informed consent statement was uniform throughout the qualitative data collection. As an illustration, below is the introduction and informed consent statement given to respondents during semi-structured interviews. Introduction and informed consent statement Thank you very much for setting time aside for us today. My name is ________ and I am part of the USAID-funded Time to Learn project (TTL), which collaborates with the MOGE to improve reading in community schools and increase equitable access to education for orphans and other vulnerable children. We are conducting an evaluation to understand how to strengthen the TTL project with a particular focus on support to teachers. We want to understand what helps teachers in the classroom. For this evaluation, we are doing interviews at 14 schools in three provinces: Eastern, Muchinga, and Southern. This evaluation is not an evaluation of the Ministry, school, teachers, or PCSC. We would like your perspective so that we can make the TTL project serve you better. I also want to tell you that I am not directly involved in any funding decisions for TTL. I am here simply to listen to your opinion. Before we begin this interview, I want to let you know that no information we discuss during this interview will be connected to a specific person. You and this school will not be identified in any reports. Your participation is completely voluntary and you are free stop the interview at any time or not to answer any questions, and this will not affect your relationship with TTL or the MOGE. The interview will take about 60 minutes. If you do not mind, I would like to record this conversation solely for the purposes of having a backup of what you say in case I am not able to write everything down. Is that all right? (If not, take written notes only.) Before we begin, what questions do you have about this interview? DRAFT Time to Learn Project Year 4 Performance Evaluation Report 42 Interview Guide: Grade 1 and 2 Teachers Background 1. Perhaps you could start by telling us about yourself. Probe for: x Role at the school, years as a teacher, educational preparation for being a teacher. x Why are you a teacher? What do you enjoy about teaching? x How long have you been affiliated with the school? Who hired you? x How is your relationship with the head teacher, other teachers, community, PCSC, Ministry officials? What do you like about this school? Classroom Orientation 2. Please show us around your classroom and tell us about how you teach your literacy activities here. Probe for: Where are materials and books kept and which ones are there? How is the classroom organized/set up? Where do the learners sit during different lessons? How are they organized? What are the strengths of the space? What are the difficulties? Classroom Practice 3. Now we want to talk about your classroom activities around reading and writing. What is your experience with teaching literacy? How do children respond to literacy lessons compared to other subjects, like mathematics or science? Probe: Is literacy easier to teach or more difficult? Can you talk more about that? 4. Tell us about how you teach literacy to your learners. What methods do you use? Allow the teacher to explain his or her methods. Probe deeper into what specific literacy skill their methods or activities are trying to teach (phonetic awareness, phonics, fluency, comprehension, vocabulary). Ask additional questions regarding what is missing (orientation to print, letter sounds, vocabulary, decoding, oral reading, reading comprehension, listening comprehension, writing): What about these other activities (i.e., reading aloud to learners, asking questions about what they read, sounding out words, learners reading text)? Why do you do these other activities? Or why don’t you? Come to a consensus about the method to teach literacy. 5. Which step do you think is the most important for your class when learning literacy? What do you focus on? Probe: Why do you think that is the most important thing to focus on? Can you tell me more about that? 6. During a typical day, what does your literacy lesson look like? What do you do? Probe: How much time per day is spent on literacy? What step or activity do you spend the most time on? What do you spend the least time on? Why do you design your literacy lesson this way? Are there any additional activities that you do on weekly basis? DRAFT Time to Learn Project Year 4 Performance Evaluation Report 43 7. When you are planning out your day or week, how do you prepare for a literacy lesson? Probe: How much time is spent preparing for class? What materials do you use to prepare (such as training packets, syllabi, newsprint, draft story cards, and annexes to training packets containing MOGE schedule for introducing letter sounds and flash cards that teachers can cut out)? 8. What activities do you use to teach literacy? Can you show me? Allow the teacher to demonstrate some activities on the board or with a book. Probe: What materials do you use (such as training packets, syllabi, newsprint, draft story cards, and annexes to training packets containing MOGE schedule for introducing letter sounds and flash cards that teachers can cut out)? How have these materials helped you teach? 9. Has anything changed in the way that you teach these activities? Probe: What type of changes? Did you receive any TTL trainings, Stepping Stone instruction, head teacher observation, teacher learning circles, etc. on this topic? When did you make changes in how you teach? 10. It sounds like you spent the most time on A and B. Is that correct? Now I want to talk about the other steps we discussed earlier: X, Y, and Z. Why do you spend more time on A and B than on X, Y, and Z? Probe: Go deeper into the areas/activities where the teacher is less comfortable. How much time is spent on these other steps? What activities do you use? What is the most difficult to teach? What is the easiest to teach? What is the most important to teach? 11. Has anything changed in how you think about X, Y, and Z? Probe: Did you teach this the same way last year? How has it changed? Did you receive TTL trainings, Stepping Stone instruction, head teacher observation, teacher learning circle, etc. on this topic? 12. What type of tools or resources would you need to improve teaching X, Y, and Z? Probe: What specific types of materials or trainings? 13. Last year we conducted a study that shows that teachers are incorporating more letter sounds and decoding activities into their lessons than reading and comprehensions activities. Why do you think that is? Ensure that this is not stated as any kind of judgment. We just want their reflections. Probe for their perceptions around approach to teaching literacy, student ability, teacher ability, availability of materials, what should be the priority in literacy lessons, etc. Classroom Management 14. Let’s look at these learners’ exercise books you’ve brought us. Please tell us about the literacy activities they have been working on. Walk through the book with the teacher. Probe for: What have the children been writing? What activity were you teaching when the learner wrote this? How did you teach about that? What activities did you use? Can you show us on the chalkboard? Based on these learners, how well did the learners understand the lesson? DRAFT Time to Learn Project Year 4 Performance Evaluation Report 44 15. Please share a time when one of your learners was doing very well and received good grades. Probe: Why was the learner doing well? What did you do to help that student? Did that experience influence your teaching practice? In what case would you talk to the learner, head teacher, parents of the learner, or parent committee? 16. Please share a time when one of your learners didn’t understand an activity or received poor grades. Probe: Why was the learner struggling? What did you do? When a student (or students) is struggling, does that influence how and what you teach? In what instances would you talk to the learner, head teacher, parents of the learner, or parent committee? Have you set up remedial groups? What do those consist of? 17. Overall, what do you expect your learners to have learned by the end of Grade 1 or 2? Probe: How is their progress? Do you think your learners are on schedule or behind schedule? What is causing them to move quickly or slowly? What have your learners completed so far this year? 18. Describe your familiarity with the Zambian National Literacy framework, including the weekly literacy teaching schedule or the Zambia Basic Education Syllabus. How do you think your learners compare to what is expected by the Zambian government for the start of Term 3? Probe: What is hindering or enabling adhering to this schedule and using the syllabus? Why are learners behind the schedule? Why are learners ahead of the schedule? How do you decide when to move onto the next step (letter sound, etc.)? 19. Please show me the tools you use to keep track of learners’ progress. How do you use these tools? Ask about specific management tools, including continuous assessments, attendance records, and assessment sheets (to record results). Note what the assessments look like, how they describe the learner and learner progress. Probe: When did you start using them? What do you do with this information? Do you receive any feedback from the head teacher or Ministry officials on the grades and reports you submit? Do these assessments inform your instructional practice (such as continuous assessment)? Teacher Support 20. We noticed you only mentioned a few materials you use regularly. Are there any other materials at the school that you have used? Probe for other materials you have seen at the school. These could include other TTL materials, Stepping Stone, posters/talking walls, textbooks, supplementary reading materials, flashcards, exercise books, or slates. Have you used these other things before? When was the last time you used them? What did you like the most about each item? Can you show me? Why don’t you use other materials? What’s special about the ones you do use? 21. How did you learn about these teaching and learning materials and how to use them? Probe for: TTL trainings, Stepping Stone, other teachers (like senior teachers), head teacher observation, teacher learning circle. DRAFT Time to Learn Project Year 4 Performance Evaluation Report 45 22. What instruction or training have you received to help you teach literacy? Prompt the teacher with relevant TTL trainings, like in-service training, Grace meetings, teacher learning circles (Alphabet sounds, Writing I, Writing II, Reading, and Read Alouds), Continuing Professional Development (CPD), workshops, Inset, Stepping Stone, or other. 23. Have you used Stepping Stone mobile phone before? If yes, probe: Describe how you have used the phone. Which lesson do you use most often and why? What other lessons have you used and why? How has the phone influenced your classroom practices? How would you improve the phone to be more useful for you and your learners? 24. What did you appreciate the most about these trainings? Probe: What did you learn? Tell me more about the training – when, where, how long, how. How useful and practical was the information provided? What was the duration of training (too long, too short)? 25. How did you use the information you received, for example in X training, in your classroom? What did you do differently as a result of the training? Probe for behavior, attitudes, teaching and reading in local languages, children reading more, collaboration among teachers in reading lesson planning, teachers reading out loud in class, etc. 26. What has prevented you from making even more changes in the classroom? Probe for limited support from the head teacher, limited skills or understanding of other teaching techniques, etc. 27. What other types of support have you received that are important? Probe for other NGOs working in the areas, non-TTL trainings, etc. 28. Please describe the teacher learning circles at your school. Probe: Do you have them? How often they meet? What is a specific example of something you discussed? How did the discussion help you? How can they be improved? 29. Describe a time when someone at the school monitored your literacy class. Probe: Who monitored your class (head teacher, deputy head teacher, senior teacher)? When did this happen? What type of feedback was given? What did the feedback comment on (specific activity, learner behavior, etc.)? How often are your literacy classes monitored? Was it helpful? Did you do anything different as a result? Is it a type of mentorship with another teacher? 30. Who else has observed your literacy class before? What did s/he do? Probe: Was it Zonal, District, Provincial, or Central Education Ministry official? What happened? What type of feedback did you receive? Did you do anything differently as a result? Did anyone follow up on feedback to make sure it was implemented? 31. What type of support would you like to get from your school or the Ministry to help improve your literacy classes? Probe: What type of feedback would be helpful you? DRAFT Time to Learn Project Year 4 Performance Evaluation Report 46 Conclusion 32. If you were granted three wishes to improve literacy instruction at your school, what would they be? a. b. c. 33. Is there anything else that you want to tell me, but didn’t because I didn’t ask the right question? 34. What questions do you have for me? Thank you for your participation! Do you have any additional questions about the interview? Interview Guide: MOGE Officers Note: This can include ZICs or Standards Officers (all those involved in monitoring schools and literacy lessons). You may need to have more than one interview. Background 1. Perhaps you could start by telling us about yourself. Probe for: Role in the Ministry, years in the position, major components of the position, educational preparation, and professional background. 2. Now we would like to know more about your relationship to community schools. Probe for: How many schools do you support and types? What are the main activities you do in regard to community schools? What types of support do you offer? How often do these types of support occur? How often do you visit? What do you do during your visits? Teacher Support 3. Now we want to talk specifically about literacy instruction and the support provided for Grade 1 and 2 teachers. What trainings have you attended and what materials have you received regarding literacy instruction? 4. In what ways have these trainings influenced your leadership and management at the schools? What materials have you found the most useful? 5. Please share a specific example when you trained head teachers or teachers based on a training you received. Probe: What was the most difficult about that process? What would have made the process easier? How did the teachers respond? How did you follow up on the trainings? 6. Please share a specific example about one training that really helped literacy instruction at a school. Probe: How did it influence the teachers? What was beneficial about it? DRAFT Time to Learn Project Year 4 Performance Evaluation Report 47 7. When have you monitored literacy lessons being taught at a community school? Probe: Do you have documents regarding those observations? Can we see them? What was the last literacy lesson you monitored? Can you describe it? What did you notice? Did you give feedback to the teacher? How was feedback given? Did you follow up on that feedback at the school? 8. When have you used eEGRA Instruct to monitor a community school classroom? Probe: Do you have data regarding those observations? Can we see them? How do you use the tool and when? What was the last literacy lesson you monitored? Can you describe it? What did you notice? How did you give feedback to the teacher? Did you follow up on that feedback? What was beneficial about using eEGRA Instruct? What was difficult? Classroom Practice 9. How familiar are you with literacy instruction in Grade 1 and 2? What methods are used to help children read? What are the steps involved? Allow the respondent to explain the necessary steps to literacy without help. Probe deeper into what specific literacy skills are being taught (phonetic awareness, phonics, fluency, comprehension, vocabulary). Ask additional questions depending on what is missing (orientation to print, letter sounds, vocabulary, decoding, oral reading, reading comprehension, listening comprehension, writing). Come to a consensus about the method to teach literacy. 10. Which do you think is the most important step for Grade 1? And for Grade 2? What should these grades focus on? Probe: Why do you think this is the most important thing to focus on in Grade 1? And in Grade 2? Can you tell me more about that? 11. What do Grade 1 and 2 teachers focus on the most in their literacy instruction? Why do they focus on these areas? 12. What do you think Grade 1 and 2 teachers are most successful at teaching in terms of literacy? Why do they excel at this activity? 13. When it comes to teaching reading and writing, what areas do Grade 1 and 2 teachers do the least? Why do you think that is? 14. What type of tools or resources do teachers use most often? 15. What type of tools or resources do teachers need to teach literacy even better? What would be helpful for them? Classroom Management 16. What types of information and reports do you receive about literacy? Ask about specific management tools, such as attendance records, continuous assessment, enrollment forms, and grades. Probe: What do you do with this information? Do you give any specific feedback to the schools? 17. What do you do if you find that a teacher or class is doing very well? What do you do if you find that a teacher or class needs to improve? Probe: Talk to head teacher, parent committee, or Ministry. DRAFT Time to Learn Project Year 4 Performance Evaluation Report 48 Conclusion 18. If you were granted three wishes to improve literacy instruction at community schools, what would they be? 19. Is there anything else that you want to tell me, but didn’t because I didn’t ask the right question? 20. What questions do you have for me? Thank you for your participation! Do you have any additional questions about the interview? Interview Guide: Head Teachers Note: Head teacher duties may have been distributed between the head teacher, deputy head teacher, and senior teachers. Find out who has been in charge of materials, who has been doing teacher observations, who has received various TTL trainings, and who has trained Grade 1 and 2 teachers. Assemble all individuals who have worked with Grade 1 and 2 teachers for a group interview. Background 1. Perhaps you could start by telling us about yourself. Probe for: x Role at the school, years as a teacher, years as head teacher, educational preparation. x How long have you been affiliated with the school? Who hired you? What do you like about this school? What do you enjoy about being a head teacher? 2. Please tell us about the current Grade 1 and 2 teachers? Probe: What is their background? How long have they been at the school? How are they performing? What are they doing well? What are your concerns? How many learners are in each classroom? How are the learners performing? Teacher Resource Orientation 3. Please show us key materials available for Grade 1 and 2, including TLM, logbooks, assessments, and records to send to the DEBS office. Look for: Flash cards, Story cards, CASAS books (story books), Maiden readers, Assessment sheets, Assessment booklets/guides, School registers, Enrollment Forms, TOTAL community library book box, Language Schedule, Reading/module reading, Writing/ module writing, Alphabet/module alphabet, CDP training module on assessment, Zambia Basic Education Syllabi, Radios, Operational Guidelines for Community Schools, ELM training manual, Teachers’ Guide on School-based Assessment, Literacy activity handbook, and any materials from other sources or created at the school. Probe for: When did you receive each item? How do the teachers access these materials? When do they use them and how often? Is there a logbook to check items in and out? How are records kept at the school? What is done with them when completed? What records do teachers give to the head teacher? What is done with those records? DRAFT Time to Learn Project Year 4 Performance Evaluation Report 49 Classroom Practice 4. How familiar are you with literacy instruction in Grade 1 and 2? What methods do your teachers use to teach learners how to read and write? Probe deeper into what specific literacy skills are being taught (phonetic awareness, phonics, fluency, comprehensions, vocabulary). Ask additional questions depending on what is missing (orientation to print, letter sounds, vocabulary, decoding, oral reading, reading comprehension, listening comprehension, writing). Come to a consensus about the method to teach literacy. 5. Which do you think is the most important step for Grade 1? And for Grade 2? What should these grades focus on? Probe: Why do you think this is the most important thing to focus on in Grade 1? And in Grade 2? Can you tell me more about that? 6. Please describe further the literacy activities of your Grade 1 and 2 teachers. Probe: What do their daily literacy lessons look like? What are they focused on? How well are they teaching these activities? How do they prepare for their classes? What materials do they use (such as training packets, syllabi, newsprint, draft story cards, and annexes to training packets containing MOGE schedule for introducing letter sounds and flash cards that teachers can cut out)? 7. What about the other aspects of literacy instruction you described, such as A, B, and C? Probe: Go deeper into areas/activities that teachers are less focused on. How do teachers teach these? It sounds like these are done less often, why? 8. What do you think Grade 1 and 2 teachers are most successful at teaching in terms of literacy? Why do they excel at this activity? 9. What do you think Grade 1 and 2 teachers are least successful at teaching in terms of literacy? Why is that? 10. What type of tools or resources do teachers need to teach literacy even better? What would be helpful for them? Classroom Management 11. How often are learners in Grades 1 and 2 assessed on their reading ability? What methods are used? Probe: Ask about specific management tools (attendance records, assessment sheets (to record results), Assessment booklets/guides (instructional), Teachers Guide on School-based Assessment, school register, enrollment forms). How do you keep track? Do you receive information from teachers? 12. What do you do with these results? Can you give me a specific example of working with a teacher as a result of these tools? Probe: Work with teacher, discuss in teacher learning circles, and talk to learner, parents, or parent committee about results? 13. How did you learn how to use these tools? When did you start using them? Probe for TTL trainings, instruction from ZICS or other Ministry official, etc. DRAFT Time to Learn Project Year 4 Performance Evaluation Report 50 14. Describe your familiarity with the Zambian National Literacy Framework, including literacy teaching schedule or the Zambia Basic Education Syllabus. How do you think your learners compare to what is expected by the Zambian government for the start of Term 3? Probe: How are your teachers using these materials to plan their classes? What is hindering or enabling them to adhere to this schedule or syllabus? Why are learners ahead or behind? Teacher Support 15. What trainings have you attended? In what ways have these trainings affected your leadership and management at the school? Probe: How have you been better able to manage and support your school? Can you provide a specific example? What kind of impact did the trainings have on teachers at your school and literacy instruction? 16. We’ve already talked about teaching and learning materials you have received, but what materials have you found the most useful? Probe: How have these materials been used at your school? Which ones have been used the most? Which ones have been used the least? Why? 17. Please share a specific example when you trained teachers at your school using the information, tools, or resources you received from training. Probe: How did you share this information? How was it received? What was the most difficult about that process? What would have made the process easier? How did the teachers respond? How did you follow up on the trainings? What did the teachers find the most helpful? 18. Have you used Stepping Stone mobile phone at your school? Probe: If yes, download the Stepping Stone use data. Ask: How has it been used at the school? Why has it been used this way? How were teachers trained at your school? How has the phone been helpful? What would make the phone more useful for the school and for the teachers? 19. Describe a time you monitored a literacy lessons being taught? Probe: What was the last literacy lesson you monitored? Please describe it? What did you notice? Did you give feedback to the teacher? How was feedback given? 20. Describe a time when you organized a meeting with teachers to specifically address teaching quality? Probe: When was the last meeting? What was discussed? What was the outcome? How often do you organize meetings with the teachers? What do you usually discuss? What would make these meetings more productive? 21. When has a Ministry official observed Grade 1 and 2 classrooms at the school? Probe: Use the visitor logbook to identify dates of Zonal, District, Provincial, or Central Education Ministry official visits. What feedback did you receive? Was the feedback helpful? How often do Ministry officials visit the school? What do they do when they visit? DRAFT Time to Learn Project Year 4 Performance Evaluation Report 51 Conclusion 22. If you were granted three wishes to improve literacy instruction at your school, what would they be? 23. Is there anything else that you want to tell me, but didn’t because I didn’t ask the right question? 24. What questions do you have for me? Thank you for your participation! Do you have any additional questions about the interview? Focus Group Discussion Guide At the beginning of all focus group discussions, the evaluators gave an introduction to the evaluation and an informed consent statement, which is outlined below: Introduction Thank you very much for setting time aside for us today. My name is ________. We are conducting the project year 4 performance evaluation to understand how to strengthen the TTL project with a particular focus on support to teachers. We want to understand what helps teachers in the classroom. For this evaluation we are doing interviews at 14 schools in three provinces: Eastern, Muchinga, and Southern, and conducting three focus group discussions. We would like your perspective so that we can make the TTL project serve community schools better. Before we begin this discussion, I want to let you know that no information we discuss during this interview will be connected to a specific person. You will not be identified in any reports. Your participation is completely voluntary and you are free stop the discussion at any time or not to answer any questions. The interview will take about 90 to 120 minutes. If you don’t mind, I would like to record this conversation solely for the purposes of having a backup of what you say in case I am not able to write everything down. Is that alright? (If not, take written notes only.) Before we begin, what questions do you have about this discussion? Focus Group Discussion Guide: Teachers Instructional Practice 1. Please tell us your first name and how long you have been involved in community schools? DRAFT Time to Learn Project Year 4 Performance Evaluation Report 52 2. What is the method you use to help someone read? What are the steps involved? Let’s make a list on the wall. Probe: What specific literacy skills do these methods or activities try to teach (phonetic awareness, phonics, fluency, comprehension, vocabulary)? Ask additional questions about what is missing (orientation to print, letter sounds, vocabulary, decoding, oral reading, reading comprehension, listening comprehension, writing). 3. Which steps/activities do you do most often? Why? Probe: What about Grade 1? Grade 2? How are these different? How are they similar? 4. Which steps/activities do you do less often? Why? Probe: What about Grade 1? Grade 2? How are these different? How are they similar? 5. Last year we conducted a study that shows that teachers are incorporating more letter sound and decoding activities into their lessons than reading and comprehension activities. Why do you think teachers are only doing those types of activities? Probe for views of learner ability and teacher ability, availability of materials, what should be part of literacy lessons and what should not in Grade 1 and 2, etc. 6. What advice would you give to teachers who are not doing reading and comprehension activities? 7. What type of tools or resources do teachers need to improve their literacy teaching? What would be helpful for them? Thank you for your participation! Do you have any additional questions about the interview? DRAFT Time to Learn Project Year 4 Performance Evaluation Report 53 ANNEX 3: REFERENCES Falconer-Stout, Z., L. Messner and V. Wedekind. 2015. USAID Time to Learn Project: Midline Impact Evaluation. Lusaka, Zambia: Time to Learn Project. Lewis, Carrie. 2015. Recommendations for Continued Reading Progress in Zambia Community School. Lusaka, Zambia: Time to Learn Project. MOGE. 2013a. National Literacy Framework. Lusaka, Zambia: MOGE. MOGE. 2013b. Zambian Languages Syllabus: Grades 1-7. Lusaka, Zambia: MOGE. MOGE. 2013c. Zambia’s National Assessment Survey Report – 2012. Lusaka, Zambia: MOGE. MOGE. 2014. Reading Performance Level Descriptors for Grade 1. Lusaka, Zambia: MOGE. MOGE. 2015. 2014 Educational Statistical Bulletin Tables. Lusaka, Zambia: MOGE. Time to Learn Project Plot Number 203B, Off Kudu Road Kabulonga, Lusaka Zambia