TIME TO LEARN MIDLINE IMPACT EVALUATION 6HSWHPEHU 2015 This publication was produced for the USAID Time to Learn Project. It was prepared by Zachariah Falconer-Stout, Lyn Messner, and Vera Wedekind, Time to Learn/EnCompass, LLC. Time to Learn Project Time to Learn project (TTL) is funded by the United States Agency for International Development (USAID) in Zambia under contract number AID 611-C-12-00002, awarded on March 1, 2012. Time to Learn is implemented by Education Development Center, Inc. (EDC), in collaboration with the Campaign for Female Education (Camfed), EnCompass LLC, and the Forum for African Women Educationalists in Zambia (FAWEZA). The project assists the Ministry of Education, Science, Vocational Training and Early Education (MESVTEE) through a 5-year national program to provide an equitable standard of education service for vulnerable learners, improve reading skills, and implement practical strategies to strengthen school quality and promote community engagement in community schools. Cover photo by: Zachariah Falconer-Stout, EnCompass LLC TIME TO LEARN MIDLINE IMPACT EVALUATION September 8, 2015 Project Number AID-611-C-12-00002 DISCLAIMER This evaluation is made possible by the support of the American people through the United States Agency for International Development (USAID). It was produced by Encompass LLC for the Time to Learn project, which is funded by USAID in Zambia under contract no. AID- 611-C-12-00002. The contents of this evaluation are the sole responsibility of the author(s) and do not necessarily reflect the views of USAID or the United States Government. Time to Learn Midline Impact Evaluation 4 CONTENTS ACRONYMS ................................................................................................................................................ 5 EVALUATION TEAM ................................................................................................................................. 6 EXECUTIVE SUMMARY ............................................................................................................................ 8 EVALUATION PURPOSE AND EVALUATION QUESTIONS ................................................................................................................. 8 PROJECT BACKGROUND ..................................................................................................................................................................... 8 EVALUATION DESIGN, METHODS, AND LIMITATIONS..................................................................................................................... 8 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS .................................................................................................................. 9 1. EVALUATION PURPOSE AND EVALUATION QUESTIONS ................................................... 12 1.1. EVALUATION PURPOSE .......................................................................................................................................................12 1.2. EVALUATION QUESTIONS ..................................................................................................................................................13 2. PROJECT BACKGROUND .............................................................................................................. 14 2.1. PROJECT CONTEXT ............................................................................................................................................................14 2.2. TTL INTERVENTIONS..........................................................................................................................................................15 3. EVALUATION DESIGN, METHODS, AND LIMITATIONS ........................................................ 18 3.1. EVALUATION DESIGN.........................................................................................................................................................18 3.2. METHODS.............................................................................................................................................................................19 3.3. DATA COLLECTORS ...........................................................................................................................................................21 3.4. ANALYSIS..............................................................................................................................................................................21 3.5. LIMITATIONS ........................................................................................................................................................................21 4. DESCRIPTION OF THE SAMPLE ................................................................................................... 22 4.1. CHARACTERISTICS OF SCHOOLS IN THE SAMPLE............................................................................................................22 4.2. CHARACTERISTICS OF LEARNERS IN EGRA SAMPLE.......................................................................................................24 4.3. CHARACTERISTICS OF TEACHERS PARTICIPATING IN LITERACY LESSON CLASSROOM OBSERVATIONS.................26 4.4. CHARACTERISTICS OF COMMUNITY SCHOOL HEAD TEACHER QUESTIONNAIRE RESPONDENTS ..........................27 5. FINDINGS .......................................................................................................................................... 28 5.1. EVALUATION QUESTION 1: TO WHAT EXTENT HAVE TTL INTERVENTIONS IMPROVED EARLY GRADE LITERACY ACHIEVEMENT AMONG BOYS AND GIRLS IN COMMUNITY SCHOOLS ACROSS SIX PROVINCES COMPARED TO BASELINE? .................................................................................................................................................................................28 5.2. EVALUATION QUESTION 2: HOW HAS MESVTEE COMMUNITY SCHOOL ENGAGEMENT CHANGED SINCE BASELINE? ...............................................................................................................................................................................................33 5.3. EVALUATION QUESTION 3: TO WHAT EXTENT ARE MALE AND FEMALE TEACHERS IMPLEMENTING TTL-SUPPORTED LITERACY TEACHING METHODS? ...........................................................................................................................................39 6. CONCLUSIONS AND RECOMMENDATIONS ............................................................................ 44 6.1. CONCLUSIONS ....................................................................................................................................................................44 6.2. RECOMMENDATIONS..........................................................................................................................................................46 ANNEXES .................................................................................................................................................. 49 ANNEX 1. EGRA SUMMARY TABLES BY PROVINCE AND LANGUAGE..................................................................................50 ANNEX 2. SAMPLE, METHODS, INTER-RATER RELIABILITY, AND LIMITATIONS ....................................................................58 ANNEX 3. EVALUATION QUESTIONS BY DATA SOURCE AND INDICATOR.........................................................................66 ANNEX 4. DOCUMENTS REVIEWED..........................................................................................................................................67 ANNEX 5. EVALUATION TOOLS ................................................................................................................................................69 Time to Learn Midline Impact Evaluation 5 ACRONYMS EDC Education Development Center, Inc. EdData II Education Data for Decision Making mechanism EGRA Early Grade Reading Assessment IR Intermediate result MESVTEE Ministry of Education, Science, Vocational Training and Early Education PCSC Parent community school committee SCOPE Standards-based Classroom Observation Protocol TTL Time to Learn USAID U.S. Agency for International Development Time to Learn Midline Impact Evaluation 6 EVALUATION TEAM CORE TEAM Zachariah Falconer-Stout, Team Lead Lyn A. Messner, Senior Research and Evaluation Advisor Vera Wedekind, Research and Evaluation Specialist Ashley Strahley, Evaluation Associate Clerisse Lemke, Evaluation Intern Armando Levinson, Statistician DATA COLLECTORS Province indicated by superscript; see key at right Team Leads Maeve Bartlett1, 3, 4 Vera Wedekind2, 3, 4, 6 Zachariah Falconer-Stout3, 5 EGRA Assessors Anthony Chomba, MESVTEE, Chinsali District5 Brenda Kalaluka5 Chileshe Bwalya2 Conrad Bwalya2 Faustina Kamfwa4 Francis Phiri3 Godfrey Chitalu4 Idah G. Mwanza4 Jane Lisimba1 Janet Monde, MESVTEE Southern Province6 Mbololwa Nalumino, MESVTEE Southern Province6 Key 1: Central Province 2: Copperbelt Province 3: Eastern Province 4: Lusaka Province 5: Muchinga Province 6: Southern Province Time to Learn Midline Impact Evaluation 7 Michael Musonda1, 5 Paul Daka5 Tabo Mayapi4 6 Titus Ernesto Miti, MESVTEE Eastern Province3 Classroom Observers Chanda Chileshe, MESVTEE Chadiza District3 Felistus Moono, MESVTEE Lusaka Province4 Kabby Mabili, MESVTEE Lusaka Province4 Martin Chibulika, MESVTEE Central Province1 Maurice Mulopo, MESVTEE Southern Province6 Pardon Tesho, MESVTEE Muchinga Province 5 Valentine Yumba, MESVTEE Copperbelt Province2 Head Teacher Questionnaire Administrators Conrad Bwalya2 Francis Phiri3 Godfrey Chitalu4 Michael Musonda1, 5 Paul Daka5 Tabo Mayapi4 6 Time to Learn Midline Impact Evaluation 8 EXHIBIT 1: KEY EVALUATION QUESTIONS 1. To what extent have TTL interventions improved early grade literacy achievement among boys and girls in community schools across six provinces compared to baseline? 2. How has MESVTEE community school engagement changed since baseline? 3. To what extent are male and female teachers implementing TTL-supported literacy teaching methods? EXECUTIVE SUMMARY Evaluation Purpose and Evaluation Questions The purpose of this internal midline impact evaluation (conducted 2 years after the intervention began) is to assess the degree to which the USAID Time to Learn project (TTL) is on track to achieve its intended impact and outcomes, and identify corrections needed to achieve targets in the project’s remaining 2 years. The key audience is the TTL project team, which includes USAID\Zambia and the Ministry of Education, Science, Vocational Training, and Early Education (MESVTEE) as key stakeholders. Impact evaluation activities are being conducted in project years 1 (baseline), 3 (midline), and 5 (endline) to measure literacy levels among learners, and other intermediate outcomes to determine if teachers, head teachers, and the MESVTEE are using TTL-promoted techniques. The 2012 TTL evaluation design, developed before baseline data collection, conceptualizes five impact evaluation questions. For the midline evaluation, the evaluation team refined these questions with the TTL project team to the three key questions listed in Exhibit 1. Project Background TTL is a 5-year (2012-2017) USAID-funded project that aims to improve reading among 500,000 primary grade community school learners in six of Zambia’s 10 provinces. TTL aims to inform and inspire policy dialogue at the central MESVTEE level, creating a favorable environment for effective implementation of the MESVTEE policy for integrating community schools into the formal education system, and providing a range of MESVTEE actors with an opportunity to understand how to sustain and generalize TTL interventions for project scale-up. TTL contributes to USAID Education Strategy Goal 1: Improved reading skills for 100 million children in primary grades by 2015. TTL’s development hypothesis aims to improve reading among community school learners through four changes: 1) increased MESVTEE support to community schools, 2) improved community school literacy instruction and educational management, 3) improved parent community school committee (PCSC) governance, resource mobilization, and advocacy for quality reading instruction and support, including in the home, and 4) increased access to age-appropriate familiar-language teaching and learning materials. TTL seeks to achieve these results through four interventions: MESVTEE and PCSC capacity building, teacher training, and teaching and learning material development and dissemination. Evaluation Design, Methods, and Limitations The 2012 impact evaluation design (for baseline, midline, and endline) was developed through a consultative meeting with TTL and various MESVTEE offices. In 2014, the evaluation team developed a Time to Learn Midline Impact Evaluation 9 midline implementation plan with the TTL project team to detail modifications made to the 2012 evaluation design in response to changes in project activities, baseline challenges and successes, and updated USAID Education Strategy Goal 1 measurement guidance. This evaluation used an independently pooled, repeated cross-sectional design. Key evaluation questions 1 and 2 are answered through comparison with baseline data to provide the counterfactual. A single cross section is used to answer evaluation question 3 because comparative baseline data is unavailable. Baseline and midline data were collected at the same point in the school year. The evaluation used a two-stage sampling design, with an intermediary stage applied in schools with more than one grade 2 class. In order to provide provincially-disaggregated estimates of early grade reading assessment (EGRA) data, target sample sizes were calculated for the provincial level and separate samples were drawn for each province. This procedure is consistent with the sampling methodology employed at baseline, reflecting the cross-sectional design. Sample targets were met in all provinces except Muchinga Province. Between September and December 2014, the evaluation team collected midline data through a school￾based survey that used the following tools and methods: EGRA, classroom observation protocol, head teacher questionnaire, MESVTEE district education board self-administered survey questionnaire, and document review. The evaluation team first analyzed all data using descriptive statistics, with particular attention to group distributions, and then proceeded to bivariate and inferential techniques; all data were analyzed by sex. Unless specifically noted, differences should not be taken as statistically significant. This evaluation is limited by the following factors: under-sampling in Muchinga Province, no baseline data for EGRA task 4 orientation to print and task 6 English language listening comprehension, restrictions in EGRA design that limit comparability across the three languages of assessment, a lack of clear definitions for language specific definitions of grade level text, the absence of a non-intervention comparison group, and recall issues among head teachers. Findings, Conclusions, and Recommendations Findings Baseline and midline EGRA data comparison shows a small, but statistically significant positive trend at both provincial and language-group levels (three official languages of instruction). These are two different, but interrelated aspects: (1) the size of literacy gains from baseline to midline are positive, although small in absolute terms, and (2) these gains are generally significant in statistical terms, meaning that there is a high degree of confidence that these gains are not a result of random chance due to sampling. While most learners cannot complete EGRA tasks as evidenced by the high proportion of learners scoring zero on most tasks, the 2-year trends are positive, with a decrease in the proportion of learners scoring zero on all comparable tasks. Grade 2 learners have moved closer to the MESVTEE standard for grade 2 reading competences since 2012. Familiar language listening comprehension scores were relatively higher than other EGRA tasks, but still low overall. This indicates that low degrees of oral comprehension may be holding learners back. Nevertheless, learners performed significantly better in listening comprehension in the languages of instruction than in English, providing evidence in support of recent changes to language of instruction policy. Boys significantly outperformed girls on most tasks, although the size of those differences remains small in absolute terms and neither group has a large proportion of learners who are reading with comprehension. Time to Learn Midline Impact Evaluation 10 MESVTEE district offices’ and head teachers’ responses differ greatly on the amount of MESVTEE support provided to community schools in both 2012 and 2014. The percentage of head teachers who reported receiving MESVTEE support increased in five of the seven areas since 2012: monitoring (increase of 61 percent), teaching and learning materials (increase of 58 percent), free basic materials (increase of 47 percent), teacher trainings (increase of 23 percent), and deployed government teachers (increase of 11 percent). The percentage of community schools receiving grants reduced, while grant amounts remained relatively stable. Overall, head teachers reported receiving significantly more types of MESVTEE support and more support from zonal and district offices in 2014 than in 2012. The classroom observation protocol included seven domains that form the core pillars of early grade literacy lessons. According to TTL, the project’s head teacher and teacher training content covered all seven domains. Teachers were observed most often conducting letter sound and orientation to print activities, and were least often observed practicing reading and listening comprehension. The average length of literacy lessons increased significantly, by over 30 percent, since baseline. Additionally, the majority of literacy lessons observed were conducted in the official language of instruction although, as reported by head teachers, more often in grade 1 than grade 2. While there was some variation from province to province and, to a lesser extent, by teacher sex, literacy lessons generally emphasized similar domains. Conclusions Learners have improved significantly on the majority of EGRA tasks since baseline, although Lusaka Province showed backsliding on some tasks. Overall, the largest improvements were in the letter sounds task, which aligns with classroom observation data that letter sounds occupy a high proportion of instructional time. Baseline findings were inconclusive regarding differences between girls and boys, but midline findings indicate a significant trend of boys doing better than girls in a number of areas. In all six provinces learners performed significantly better in listening comprehension in the official language of instruction than in English. This aligns with Zambia’s current transition to familiar language instruction and TTL’s efforts to translate this policy into practice in community schools. The Southern and Eastern African Consortium for Monitoring Educational Quality (SACMEQ) testing consortium showed grade 6 reading levels declining in Zambia from 1995 to 2007 (from SACMEQ I to SACMEQ III). Therefore, TTL’s efforts have not only contributed to improving reading, but also to reversing a negative trend. Although the MESVTEE and head teachers’ perceptions of the amount of MESVTEE support provided to community schools in 2012 and 2014 differ, head teachers report a marked increase in and more forms of MESVTEE support since 2012. Head teachers also report more MESVTEE support from zonal levels in 2014 than in 2012. This could be an indication of MESVTEE decentralization or closer relationships and communication between community schools and zonal MESVTEE officials. There is a large discrepancy in perceptions about the percentage of schools receiving grants, the average grant size, and other types of support. Head teachers may be unclear about what support can be attributed to the MESVTEE versus other sources outside of the Government of Zambia, and the MESVTEE may be over-reporting the support provided. Current literacy teaching practices among teachers are positive, but insufficient to reach minimal MESVTEE standards for learner performance. Teachers appear to be including new literacy techniques, although there is still far to go to meet the standards. This may indicate inconsistencies in TTL training implementation or that core training content is not being fully absorbed by teachers. Overall, teachers are applying at least some of the knowledge and skills in which they have been trained, such as capacity to apply phonics-based fundamentals. The TTL baseline did not systematically collect data on the Time to Learn Midline Impact Evaluation 11 presence or absence of letter sound teaching, but anecdotal data by data collectors indicate that the approach to literacy was not phonics-based, in which case the extent of letter sound instruction in 2014 would point to a positive change. There is also a gap between classroom focus on reading passages and the type of comprehension questions linked to those passages that build learners’ ability to read with understanding. In order for readers to achieve fluent reading, i.e., reading with both speed and accuracy, it is fundamental that they observe this behavior modeled by successful readers and have the opportunity to read more than single words. Recommendations 1. TTL should work with the MESVTEE and head teachers to accelerate and improve the quality of formative community school monitoring. 2. TTL and the MESVTEE should identify the barriers head teachers and teachers are facing in providing comprehensive and equitable early grade literacy lessons. 3. TTL and the MESVTEE should strengthen teacher training on comprehension and oral passage reading where teachers are struggling. Time to Learn Midline Impact Evaluation 12 1. EVALUATION PURPOSE AND EVALUATION QUESTIONS 1.1. Evaluation Purpose The purpose of this internal midline impact evaluation (conducted 2 years after interventions began) is to assess the degree to which the USAID Time to Learn project (TTL) is on track to achieve its intended impact and outcomes, and what corrections should be made to achieve targets in the project’s subsequent years. TTL is conducting internal impact and performance evaluations over the life of the project for itself and its key stakeholders to understand progress made towards three of the five TTL intermediate results (IRs) indicated in Exhibit 21 by the red boxes. EXHIBIT 2: TTL RESULTS FRAMEWORK IN THE CONTEXT OF USAID’S RESULTS FRAMEWORK Combined, these evaluations reflect a multilevel and sequential, mixed-method approach so that TTL can assess its interventions at different points and from different perspectives, and to provide TTL with a holistic understanding of project results over time. This approach is illustrated in Exhibit 3 where the 1 All internal report links are indicated by red, underlined text. Time to Learn Midline Impact Evaluation 13 EXHIBIT 3: TTL EVALUATION TIMELINE blue boxes indicate the three phases of the impact evaluation and red boxes indicate performance evaluations. Performance evaluations use qualitative and limited quantitative data to assess how TTL interventions are being implemented and perceived by key stakeholders. Conducted in project years 2 and 4 in between impact evaluation phases, these performance evaluations help TTL understand why the project has been effective and how it can be improved. Impact evaluation phases are being conducted in project years 1 (baseline), 3 (midline), and 5 (endline) to measure literacy levels among primary learners (see Exhibit 2, USAID/Zambia IR 3.1), and other intermediate outcomes to determine if teachers, head teachers, and the Ministry of Education, Science, Vocational Training, and Early Education (MESVTEE) are using TTL-promoted techniques. Impact evaluation activities use primarily quantitative data. 1.2. Evaluation Questions The 2012 TTL evaluation design, developed before baseline data collection, conceptualizes five impact evaluation questions. For the midline evaluation, the evaluation team refined these to three key questions and corresponding sub-questions listed in Exhibit 4. These modifications were made with the TTL project team (documented in the midline evaluation implementation plan) to align the evaluation questions with project implementation since baseline, TTL’s information needs, and USAID priorities. EXHIBIT 4: MIDLINE EVALUATION QUESTIONS Key Evaluation Question Sub-Questions 1. To what extent have TTL interventions improved early grade literacy achievement among boys and girls in community schools across six provinces compared to baseline? (TTL IR 2) a) To what extent, and for how many learners, have TTL interventions increased reading skills among boys and girls across six provinces? (TTL indicator 2.1.1) b) What proportion of male and female learners across six provinces, in TTL-supported community schools, can read and understand the meaning of grade-level text after 2 years of primary schooling? (TTL indicator 2.1) 2. How has MESVTEE community school engagement changed since baseline? (TTL IR 1) a) How many community schools receive more support from the MESVTEE compared to baseline? (TTL indicator 1.3) b) To what extent has the MESVTEE improved their monitoring of community schools compared to baseline? (TTL indicator 1.2) 3. To what extent are male and female teachers implementing TTL-supported literacy teaching methods? (TTL IR 2) Time to Learn Midline Impact Evaluation 14 EXHIBIT 5: MAP OF PROVINCES WHERE TTL WORKS 2. PROJECT BACKGROUND TTL is a 5-year (2012-2017) USAID-funded project that endeavors to improve reading among 500,000 primary grade community school learners in six of Zambia’s 10 provinces as illustrated in Exhibit 5 and Exhibit 6. TTL aims to inform and inspire policy dialogue at the Central MESVTEE level, creating a favorable environment for effective implementation of MESVTEE policy for integrating community schools into the government education system, and providing a wide range of MESVTEE actors with an opportunity to understand how to sustain and generalize TTL interventions for project scale-up. 2.1. Project Context Zambia declared free primary education in 2002 and, according to the 2013 MESVTEE Educational Statistical Bulletin, male and female enrollment in early grades has increased steadily since, much of which is attributed to community schools. Created by communities, community schools typically include grades 1 to 7 and are managed by the community through a parent community school committee (PCSC), which has the primary responsibility for supporting the school. As the HIV epidemic swept through Zambia and in the decade before primary school fees were abolished through free basic education, community schools absorbed orphans and other vulnerable children who were not attending government schools. Today, community schools still serve Zambia’s most vulnerable communities by helping relieve overcrowding, serving vulnerable populations in urban areas, and providing access to education in the most rural areas where government schools are too distant to be accessed by young learners. The 2013 MESVTEE Educational Statistical Bulletin counts approximately 3,000 registered community schools whose learners comprise at least 18 percent of the primary school-going population in Zambia. Most community school teachers are volunteers who receive stipends from the school, lack formal teacher training, and may only have a secondary school education or less. The MESVTEE now has policies for deploying trained government teachers to serve in community schools and monitoring community schools. Additionally, registered community schools are eligible to receive government assistance in the form of continuing professional development/in-service training, small grants, books or other materials, infrastructure, and trained government teachers on deployment. In recent years, the MESVTEE has introduced a new literacy curriculum for primary school grades 1 through 4 that adopts familiar language instruction for the teaching of early grade reading, with seven Zambian languages replacing English as the medium of instruction. Time to Learn Midline Impact Evaluation 15 EXHIBIT 7: TTL DEVELOPMENT HYPOTHESIS EXHIBIT 6. TTL CATCHMENT AREA POPULATION (SOURCE: TTL MONITORING DATA) Provinces # Districts # Zones # Community Schools # Teachers # Learners Central 11 97 454 1,304 140,302 Copperbelt 10 79 306 1,296 68,881 Eastern 9 106 348 516 59,309 Lusaka 8 37 346 1,286 94,142 Muchinga 7 134 198 413 33,406 Southern 13 91 362 1,152 67,978 TOTAL 58 544 2,014 5,967 464,018 2.2. TTL Interventions The TTL project contributes to Goal 1 of the USAID Education Strategy (2011-2015): Improved reading skills for 100 million children in primary grades by 2015. This strategy establishes early grade reading ability as a key determinant of retention and success in future grades. TTL’s development hypothesis, illustrated in Exhibit 7, is that community school learners will have improved reading (represented by the light blue box) as a result of the following changes (represented by the grey boxes): x Increased support to community schools from the MESVTEE x Improved skills in literacy instruction among community school teachers and better educational leadership and management by head teachers x Improved PCSC school governance, resource mobilization, and advocacy for high quality reading instruction and support, including in the home x Increased access by community schools to age-appropriate textbooks and other teaching and learning materials in a familiar language. TTL endeavors to achieve these results through a series of interventions represented by the four dark blue boxes, described below. They are MESVTEE capacity building, teacher training, PCSC capacity building, and teaching and learning material development and dissemination. 2.2.1. MESVTEE Capacity Building TTL works with the MESVTEE through its existing structures and systems to reinforce its capacity to train, manage, plan, monitor, and evaluate community school progress toward improved education standards, and to diffuse literacy and community school policy updates throughout the MESVTEE structure. At data collection, TTL had supported the MESVTEE to: Improved reading among learners in community schools The MESVTEE provides more support to community schools TTL advocates to the MESVTEE for increased support to community schools Community schools have skilled teachers and managers TTL supports the MESVTEE to train Head teachers and teachers PCSCs are mobilizing resources for the school TTL builds PCSC capacity in school management and community mobilization Community schools have and use teaching and learning materials TTL supports the MESVTEE to develop and disseminate teaching and learning materials Time to Learn Midline Impact Evaluation 16 x Incorporate reading assessment into routine MESVTEE monitoring of community schools via eEGRA Instruct, EDC’s proprietary software program to monitor reading performance (2014). x Form a community school steering committee that contributed to: forming of a national community school symposium (2014), revising of the 2007 operational guidelines for community schools to improve the policy framework governing community schools (2013- 2014), and modifying of the national policy on education to ensure the inclusion of community schools (in process). x Improve early grade reading strategy and policy. 2.2.2. Teacher Training Designed in collaboration with the Central MESVTEE, TTL trainings cascade down through MESVTEE teacher education structures from provincial, district, to zonal levels and ultimately to community school teachers. The following trainings were conducted between project inception and midline data collection: x Quick start literacy program on literacy instruction basics for head teachers included the following modules: Comprehension, Fluency, Phonemic Awareness and Phonics (word decoding), Read Aloud (oral passage reading), and Vocabulary and Spelling (2013; 2-day training). x Education leadership and management for head teachers and PCSC Chairs in managing resources, information and records; conducting and supervising school-based assessments; assessing effective teaching; school improvement planning; monitoring and evaluating school performance; and providing psychosocial counseling, environment, health and hygiene education (September 2013 to March 2014; 2-day training). x Zonal Training and Teacher Learning Circles have included Reading (read aloud) and Writing modules in 2013, and the Alphabet Sounds (letter sounds) module in 2014; each module is 1 day. Zonal training participants were head teachers who trained their teachers in the same material through Teacher Learning Circles. x School-based Assessment Training for head teachers and teachers on how to monitor learner progress and intervene accordingly. This included a primary literacy program and teaching methods for literacy instruction (2014; 2 days). At the time of data collection, TTL monitoring data indicated that the project had trained over 3,000 head teachers and MESVTEE Officials and over 7,000 community school teachers. 2.2.3. PCSC Capacity Building Community schools are managed by PCSCs that are generally comprised of parents, teachers, and prominent community members. Teachers are accountable to PCSCs, and this accountability is widely regarded as a major strength of community schools. TTL reports having trained more than 4,000 PCSCs (at least two representatives from each registered PCSC) in the six TTL provinces on community literacy mobilization and the operational guidelines for community schools. 2.2.4. Teaching and Learning Material Development and Dissemination TTL develops and disseminates low-cost, easily replicable textbooks and instructional resources to improve reading instruction in community schools such as: Time to Learn Midline Impact Evaluation 17 x Reading/learning kit for learners (e.g., story cards, short story books in local language, English language books through TOTAL book boxes, flashcards, and levelled readers) x Instructional resources for teachers (e.g., basic education syllabi, teaching guides) x Management materials (attendance logs, enrolment forms, and continuous assessment booklets). At the time of this report’s finalization, the TTL performance monitoring plan specified a target to distribute over 600,000 teaching and learning materials by the end of the project. Time to Learn Midline Impact Evaluation 18 EXHIBIT 8: KEY EVALUATION QUESTIONS 1. To what extent have TTL interventions improved early grade literacy achievement among boys and girls in community schools across six provinces compared to baseline? 2. How has MESVTEE community school engagement changed since baseline? 3. To what extent are male and female teachers implementing TTL￾supported literacy teaching methods? 3. EVALUATION DESIGN, METHODS, AND LIMITATIONS This midline evaluation, designed as a learning and adaption tool for TTL and its stakeholders, integrated participatory and utilization-focused approaches with quantitative methodologies to engage key stakeholders in evaluation planning; data collection, sharing and processing; formulating findings and recommendations; and implementing recommendations. 3.1. Evaluation Design The initial impact evaluation design — for baseline, midline, and endline — was developed in July 2012 through a consultative meeting with TTL and the MESVTEE (Directorate of Standards and Curriculum [including the Curriculum Development Center], Directorate for Planning and Information, district- and provincial-level officials, and the Examinations Council of Zambia). The evaluation team developed a midline implementation plan with the TTL project team that details adaptations to the July 2012 evaluation design to respond to changes in project activities, challenges, and successes encountered during the baseline, and updated USAID guidance for measurement of USAID Education Strategy Goal 1. In keeping with USAID-recommended approaches for measuring reading gains, the evaluation team used an independently pooled, repeated cross-sectional design. Key evaluation questions 1 and 2 (see Exhibit 8) are answered through comparison with baseline data collected before project implementation to provide the counterfactual. A single cross-section is used to answer evaluation question 3 (see Exhibit 8) because comparative baseline data is unavailable (see Section 3.2 classroom observation protocol). Baseline and midline data were collected at the same point in the school year. See Annex 3 for the list of evaluation questions by data source and indicator. 3.1.1. Sample The evaluation used a two-stage sampling design, with an additional intermediary stage applied in schools with more than one grade 2 class. In order to provide provincially-disaggregated estimates of early grade reading assessment (EGRA) results, target sample sizes were calculated for the provincial-level and separate samples drawn for each province. x Stage 1: A sample of schools was drawn for each province using probability proportional to size with schools serving as clusters. In Lusaka Province, the sample was additionally stratified by rural and urban areas.2 2 Lusaka Province is the only province with a large enough number of urban schools to sample proportionally on rural and urban characteristics. Time to Learn Midline Impact Evaluation 19 1. Language of Instruction Listening Comprehension 2. Letter Sound Knowledge 3. Non-word Decoding 4. Orientation to Print 5a. Oral Passage Reading 5b. Reading Comprehension 6. English Language Listening EXHIBIT 10: EGRA TASKS x Intermediary Stage: The vast majority of schools sampled had only one section of grade 2. For schools with multiple grade 2 sections (“streams”), one stream was selected though simple random sampling.3 x Stage 2: A sex-stratified simple random sample of grade 2 learners was drawn at each school or from the selected class for schools with multiple grade 2 streams. This procedure is consistent with the sampling methodology employed at baseline, reflecting the cross￾sectional design. All stages employed random selection. See Annex 2 for more details on the Sample, including target sample size and power calculations. Exhibit 12 presents the achieved midline sample. The evaluation was designed to respond to the TTL performance monitoring plan which specifies EGRA measurement by province. In Zambia, each province has an official language of instruction, as shown in Exhibit 9 where a red asterisk (*) indicates district-level exceptions. Two provinces in the TTL intervention area have more than one language of instruction: Central and Muchinga. The primary language of instruction in Muchinga Province is iCiBemba, with one district in which CiNyanja is used. In Central Province, the primary language of instruction is iCiBemba, with four districts where CiTonga is used. The midline sample only drew from iCiBemba-speaking districts in Central and Muchinga Provinces. As a result, EGRA results from those provinces only reflect the population of iCiBemba-language of instruction community schools. 3.2. Methods Between September and December 2014, the evaluation team conducted a school-based survey that used the following tools and methods: EGRA, classroom observation, head teacher questionnaire, self-administered survey questionnaire for MESVTEE District Education Boards, and a document review. Brief descriptions of these tools and methods are below and additional detail on EGRA, the classroom observation protocol, and the community school head teacher questionnaire is provided in Annex 2. The complete set of tools is provided in Annex 5. EGRA: The TTL EGRA includes a learner interview to capture key demographic information about respondents and an assessment component that measures four core literacy competencies—alphabetic principle awareness, phonetic awareness, reading 3 See Exhibit 12 in Section 4: Description of Sample for the number of schools to which this additional sampling stage was applied. EXHIBIT 9: OFFICIAL LANGUAGES OF INSTRUCTION FOR INTERVENTION Language of Instruction Province iCiBemba Central,* Copperbelt, Muchinga* CiNyanja Eastern, Lusaka, Muchinga* CiTonga Central,* Southern Time to Learn Midline Impact Evaluation 20 fluency, and reading comprehension—through seven tasks illustrated in Exhibit 10. EGRA was administered in three languages—CiNyanja, CiTonga, and iCiBemba—corresponding to the official language of instruction of each province (see Exhibit 9). Analysis of the extent to which these languages of instruction (and EGRA testing) correspond to the actual language spoken at school and in the home for the midline learners sampled are provided in Exhibit 21 in section 4.2.2., language characteristics of the EGRA sample. Classroom observation protocol: The TTL classroom observation protocol contained 23 distinct criteria that are indicators of early grade literacy lesson quality. Lessons were observed for fulfillment of these 23 criteria in 3-minute intervals. Based on the MESVTEE standard that primary grade literacy lessons should be 60 minutes, this produces a maximum of 20 intervals during a lesson lasting for the full hour. These criteria were grouped into seven domains, listed in Exhibit 11, which aggregated criteria into broader categories that reflect the core pillars of early grade literacy lessons. These seven domains correspond to the Zambia EGRA and the TTL intervention model. EXHIBIT 11: CLASSROOM OBSERVATION DOMAINS Domain 1: Orientation to Print Domain 4: Oral Passage Reading Domain 2: Letter-Sound Knowledge Domain 5: Reading Comprehension Domain 3: Word Decoding Domain 6: Listening Comprehension Domain 7: Writing Community school head teacher questionnaire: A face-to-face interview-based questionnaire was administered to a representative of every school in the sample. The questionnaire captured data related to the school profile, head teacher’s background, school funding and resources, instructional practices, educational leadership and management, parent community school committee, and absenteeism and attrition. In order to improve validity of respondents’ answers pertaining to TTL teaching and learning material and training support received, each data collection team had a complete set of TTL teaching and learning materials, which were shown to respondents during question probing to confirm the materials were actually received by the school. Where possible, presence of TTL-disseminated materials was physically verified. MESVTEE self-administered survey questionnaire: Provided to the MESVTEE District Education Board Secretary during the evaluation team’s initial courtesy call in each district, the questionnaire asked how the MESVTEE had supported the community schools. These data were used to triangulate with community school head teachers’ perceptions of support. Because this method was not used at baseline, the MESVTEE District Education Board Secretaries were asked to report on support in 2012, as well as 2014 in order to compare changes over time. To provide for accuracy in triangulating 2014 MESVTEE support between this source and the community school head teacher questionnaire, this tool only targeted schools in the midline sample. The District Education Board Secretary was given responsibility for ensuring that the relevant district MESVTEE officers were consulted for each question. For example, District Education Board Secretaries were to consult finance officers for information pertaining to grants provided to community schools in the sample, human resources officers for the number of teachers deployed to the schools sampled, and so on. Offices were provided with at least 1 day to collect the requested information. Out of the 42 districts in which data collection took place, 35 returned the survey (83 percent), providing information on about 81 of the 102 schools surveyed in the midline (79 percent). Document review: The evaluation team received 21 TTL training documents from the TTL team— training manuals, QuickStart literacy program PowerPoint presentations and materials, and lists of Time to Learn Midline Impact Evaluation 21 training materials—and reviewed them during data analysis for training content implemented through fiscal year 2014. The team also reviewed other documents relevant to this evaluation such as the 2007 and 2014 operational guidelines for community schools (the latter will replace the former once they are signed). See Annex 4 for list of documents reviewed. 3.3. Data Collectors All data collectors were MESVTEE officials, University of Zambia School of Education students, or TTL staff. They received 1 week of training per evaluation tool they would administer (i.e., EGRA, Classroom Observation Protocol, head teacher questionnaire), as well as data quality and evaluation ethics. Training on assessor bias mitigated against risk of TTL staff bias. The evaluation team conducted inter-rater reliability scoring for all potential EGRA assessors and classroom observation protocol administrators during the September 2014 data collectors’ trainings to ensure that assessors were administering the tools consistently. Descriptions of the inter-rater reliability testing and results are provided in Annex 2. 3.4. Analysis The evaluation team analyzed the midline data in SPSS (Statistical Package for the Social Sciences) version 21. All output files were systematically archived in order to provide replicability. In general, this report documents the test statistics upon which indicators are based. The evaluation team first analyzed all data using descriptive statistics, with particular attention to group distributions, before proceeding to bivariate and inferential techniques. Descriptive and group distributions are included along with population estimates and all data were analyzed for similarities and differences between males and females. Head teacher questionnaire and classroom observation data were compared at the provincial level, but the sampling was not designed to detect significant differences at that level. Similarly, provincial-level EGRA results are disaggregated by sex, but the sample was only designed to provide accurate estimates at the sex-aggregated provincial level. Unless specifically noted in the text, differences should not be taken as statistically significant. Where noted, differences are significant at the 5 percent level. Analysis of EGRA data included zero score analysis because, as expected at this stage of an intervention, many learners still failed to complete most EGRA tasks. 3.5. Limitations The evaluation encountered the following limitations: engagement of MESVTEE officials as data collectors, premature school closings in Muchinga Province, no baseline data for EGRA task 4 orientation to print and task 6 English language listening comprehension, limitations in EGRA design to compare across languages and grade-appropriate interpretations, and recall issues among head teachers. A full description of limitations is in Annex 2. Time to Learn Midline Impact Evaluation 22 EXHIBIT 13: TOTAL SCHOOL ENROLLMENT 4. DESCRIPTION OF THE SAMPLE The overall sample for this midline evaluation is shown in Exhibit 12. Sample targets were met (or surpassed) in all provinces except Muchinga Province, where slightly fewer than the targeted number of learners were reached. EXHIBIT 12: MIDLINE EVALUATION SAMPLE OVERVIEW Schools: sample stage 1 (parantheticals denote schools with more than 1 grade 2 class) Learners participating in EGRA: sample stage 2 Teachers participating in classroom observation Head teacher questionnaire respondents (parentheticals denote participants with other titles) Province Male Female Total Male Female Unknown Total Male Female Total Central 17 (3) 147 131 278 8 8 1 17 11 6 17 Copperbelt 17 (6) 150 158 308 5 11 1 17 10 (2) 7 17 (2) Eastern 17 (2) 146 146 292 10 3 4 17 12 (2) 5 17 (2) Lusaka 18 (2) 141 141 282 4 13 1 18 10 (1) 8 (1) 18 (2) Muchinga 16 (1) 107 123 230 11 4 1 16 14 2 16 Southern 17 (1) 143 150 293 12 3 2 17 15 2 17 Total 102 (15) 834 849 1683 50 42 10 102 72 (5) 30 (1) 102 (6) 4.1. Characteristics of Schools in the Sample 4.1.1. Learner Enrollment Across the 102 schools, average total school enrollment was 268 learners, but the variation was very wide (range 54 - 1,197), as presented in Exhibit 13. No difference in school enrollment was identified between male and female learners, which aligns with the MESVTEE data on enrollment by sex in primary grades in Zambia (2013 Educational Statistical Bulletin), which shows parity. Average school enrollment overall and by sex varied by province, as shown in Exhibit 14. Average enrollment by grade decreased as grade level increased (see Exhibit 15), indicating that a substantial number of learners are either failing to progress or are exiting community schools as they progress to higher grades. Overall, the sample indicates that, on average, slightly more girls than boys enrolled in community schools in every province, but analysis by grade shows that, on average, in grades 6 and 7 more boys are enrolled among sampled schools. This aligns with the MESVTEE statistics, which indicate a higher dropout rate among girls than boys and suggest that, while more girls enroll in grade 1, they face challenges related to retention. 15 39 18 16 4 2 0 1 3 1 1 2 0 5 10 15 20 25 30 35 40 45 Number of Schools in Sample School Size (Total Enrollment) Time to Learn Midline Impact Evaluation 23 EXHIBIT 14: AVERAGE TOTAL SCHOOL ENROLLMENT BY PROVINCE AND SEX EXHIBIT 15: AVERAGE ENROLLMENT BY GRADE These exhibits present averages only among the schools actively teaching those grades. Most of the schools in the sample teach grades 1-7, as shown by Exhibit 16. This indicates that the mode of highest grade taught is 7 for the sample; the median is also grade 7. All the schools teach at least up to grade 3 and some continue beyond grade 7. EXHIBIT 16: HIGHEST GRADE TAUGHT IN SAMPLED SCHOOLS 274 377 213 297 193 250 131 180 99 147 93 127 143 194 119 155 100 128 0 50 100 150 200 250 300 350 400 Central Copperbelt Eastern Lusaka Muchinga Southern Number of Enrolled Learners Province Mean School Enrollment Mean Male Enrollment Mean Female Enrollment 50 45 45 41 36 35 31 0 10 20 30 40 50 60 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Number of Learners (Total) Grade Level 3 13 7 13 62 4 0 10 20 30 40 50 60 70 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Higher than Grade 7 Number of Schools in Sample Grade Level Time to Learn Midline Impact Evaluation 24 EXHIBIT 17: STATUS OF TEACHERS WITHIN SAMPLE SCHOOLS 4.1.2. Teachers and School Personnel Across the 102 community schools sampled, the average number of teachers was 6.26 (median 4, about evenly divided among male and female teachers). Copperbelt and Lusaka Provinces had the highest mean and median number of teachers (14.59 and 6.56 as means, and 7 and 6 as medians), while the other provinces had medians of 4 or lower. The majority of teachers at the 102 sampled community schools were volunteers (see Exhibit 17). The average learner-to￾teacher ratio was 60-1 across all 102 schools, but reached as high as 171-1 at one school. Average learner-to-teacher ratios varied from a low of 45-1 in Lusaka Province to a high of 70-1 in Southern Province. 4.1.3. Age and Location of Schools Schools in the sample had been founded between the years 1989 and 2013, and average age of participating schools was approximately 12 years. Limited variation in age of schools was reported between provinces. Schools varied widely among themselves and across provinces in the distance to the nearest MESVTEE district office (see Exhibit 18), with schools in Copperbelt and Lusaka Provinces having markedly shorter distances, and Southern Province having the longest mean distance. EXHIBIT 18: DISTANCE TO THE NEAREST DISTRICT EDUCATION BOARD SECRETARY OFFICE MESVTEE District Office Central Copperbelt Eastern Lusaka Muchinga Southern Mean Distance (in kilometers) 55.8 19.4 62.76 25.0 48.2 83.94 4.2. Characteristics of Learners in EGRA Sample 4.2.1. Sociodemographic Characteristics The EGRA sample was designed to assess an equal proportion of male and female learners. A total of 1,683 learners were assessed for their reading skills, almost evenly divided among males (49.6 percent) and females (50.4 percent). The mean age of learners was 9.55 years (median 9, see Exhibit 19), with no substantial variation by province or sex. However, 13.1 percent of the sample was 12 years of age or older. The official age of grade 2 children in Zambia is 8 years of age, indicating that community schools may have older children. Further investigation is warranted to determine if this is a result of children delaying the start of schooling or a failure to progress, as both factors could have implications for early grade reading acquisition. Volunteer teacher 59% Retired government teacher 3% Employed government teacher 26% Trained and waiting for posting 12% Time to Learn Midline Impact Evaluation 25 4 In an effort to gauge the socio-economic status of participants, learners were asked to identify household assets they owned from a basket of 10 items appropriate to the Zambian context. Learners in the EGRA sample generally come from households with a low socio-economic status (Exhibit 20), with an overall sample mean of 3.02 out of 10 possible items. Limited variation was observed across provinces. 4.2.2. Language Characteristics EGRA tested learners in the official MESVTEE language of instruction, which is determined at the national level; in most cases the language of instruction is the same throughout a province. However, due to the high degree of linguistic diversity in Zambia, not all learners in the sample spoke the official language of instruction at home or in their school (see Exhibit 21). Of the sampled learners, 90.3 percent spoke the language of instruction at school, and 81.8 percent spoke the language of instruction at home. This indicates that most community schools in the sample were using the official MESVTEE language of instruction. However, there was wide variation in these percentages across provinces (Exhibit 21). Eastern Province has by far the largest disparity between language spoken at home and the language of instruction; 39.4 percent of sampled children did not speak the language of instruction at home. This was followed by Copperbelt, Lusaka, and Muchinga Provinces where approximately one of every five learners did not speak the language of instruction at home (range: 17.7-21.7 percent). The discrepancy between language of instruction and language spoken in school was similarly large for Eastern Province (37.3 percent), while less pronounced in other provinces: 8.1 percent in Copperbelt, 10.0 percent in Muchinga, and less than 1.5 percent for all the other three provinces. 4 Age data was missing for 151 learners; data cleaning additionally removed 3 cases aged 4 and under. 8 15 102 308 338 385 173 122 44 25 6 3 0 100 200 300 400 500 5 6 7 8 9 10 11 12 13 14 15 16 Number of Learners in Sample Age in Years 52 281 412 442 200 104 79 65 35 7 6 0 100 200 300 400 500 0 1 2 3 4 5 6 7 8 9 10 Number of Learners in Sample Number of Assets (Out of 10) EXHIBIT 19: AGE OF LEARNERS IN EGRA SAMPLE (N=1529)4 EXHIBIT 20: DISTRIBUTION OF LEARNERS IN EGRA SAMPLE BY NUMBER OF HOUSEHOLD ASSETS Time to Learn Midline Impact Evaluation 26 EXHIBIT 21: PERCENT OF LEARNERS IN SAMPLE WHO SPEAK THE LANGUAGE OF EGRA TESTING …at School …at Home Central 100% 95% Copperbelt 92% 80% Eastern 63% 61% Lusaka 99% 82% Muchinga 90% 78% Southern 99% 94% TOTAL 90% 82% 4.2.3. Attendance Patterns of Learners Attendance in the Grade 2 class sampled on the day of data collection averaged 69.58 percent of the class enrollment as provided by the school with no variation between boys and girls (70.36 percent for boys and 69.33 percent for girls). Attendance varied by province ranging from 62 percent of learners attending class on the day of data collection in Lusaka and Southern Provinces to 69 percent in Central Province and around 75 percent in Copperbelt, Eastern, and Muchinga Provinces. Apart from Muchinga and Lusaka Provinces, boys’ attendance was slightly higher than girls’ across provinces. Exhibit 22 shows attendance as a percentage of enrollment based on physical counts conducted by data collectors during school visits. EXHIBIT 22: GRADE 2 ATTENDANCE AS PERCENTAGE OF ENROLLMENT ON DAY OF SCHOOL VISIT Learners were also asked about their attendance via the EGRA learner questionnaire. Of the learners sampled, 77.2 percent reported attending school every day. There is some variation by province, with around 67 percent of learners responding in Copperbelt, Eastern, and Lusaka Provinces and 96.8 percent responding in Central Province. Twelve (12) percent of learners reported attending 4 out of 5 days, 5.9 percent 3 days, 1.6 percent 2 days, and 0.4 percent 1 day. 4.3. Characteristics of Teachers Participating in Literacy Lesson Classroom Observations The evaluation team observed a total of 102 literacy lessons (one observation per school). Of the teachers observed, 54.3 percent were male and 45.7 percent were female (parity in teacher sex was not sought in the sampling design). The average age of teachers was 29.2 (median 28, range 19-50) years of age. Information on academic and professional qualifications was available for 75 percent of those observed. Province Total Male Female Central 69% 70% 65% Copperbelt 75% 77% 73 % Eastern 75% 77% 73% Lusaka 62% 64% 65% Muchinga 74% 70% 77% Southern 63% 64% 63% Total 70% 70% 69% Time to Learn Midline Impact Evaluation 27 EXHIBIT 23: HIGHEST PROFESSIONAL QUALIFICATIONS ACHIEVED BY OBSERVED TEACHERS The majority (69.3 percent) of teachers had passed the Grade 12 exam, 9.3 percent the Grade 9 exam, and 8 percent completed either Grade 10 or 11. Exhibit 23 shows the professional qualifications received and indicates that only 37 percent had any formal professional training. The average number of years teaching among those observed was 3.76 years (median 3, range 0-17), with 2.99 (median 2, range 0-10) at the present school. Most teachers were volunteers (80 percent), as seen in Exhibit 24. Of those observed, 8 percent were senior teachers, and 10.6 percent were head teachers or deputy head teachers. EXHIBIT 24: EMPLOYMENT STATUS OF OBSERVED TEACHERS 4.4. Characteristics of Community School Head Teacher Questionnaire Respondents The community school head teacher questionnaire collected demographic information on the participant who answered on behalf of the school in addition to the vital school data presented in Section 4.1. The questionnaire was designed to be administered to the head teacher, but in cases where a head teacher was unavailable, a deputy was allowed to answer on the head teacher’s behalf. In 96 of the 102 schools sampled, the head teacher served as the respondent. The highest academic qualifications of head teacher questionnaire respondents were higher than those of classroom teachers observed, with more having passed the Grade 12 exam (82.8 percent of respondents compared to 69.3 percent of the classroom teachers observed). More than half of the head teachers had no pre-service training (61.4 percent), with 31.3 percent having some kind of teaching certificate. Head teachers were generally male (71 percent) and, although the pattern of more male than female head teachers is consistent across all regions, the percentages are much higher in Muchinga and Southern Provinces (both at 88 percent). Community school head teacher respondents had worked an average of 10 years as a teacher, and 4 years as a head teacher at the school where they were working at the time the questionnaire was administered. Seventy-three (73) percent of them had been a teacher before becoming a head teacher, but only 10 percent had been a head teacher at another school before working as a head teacher at their current school. Teaching certificate/ Early Childhood certificate/ diploma 35% Bachelors of Primary Education/ Bachelors of Education 1% None 63% Other 1% (n=75; 27 with missing data) Government teacher (deployed) 12.0% Trained teacher not deployed 8.0% Volunteer teacher 80.0% (n=75; 27 with missing data) Time to Learn Midline Impact Evaluation 28 EXHIBIT 25: TTL DEVELOPMENT HYPOTHESIS 5. FINDINGS The midline evaluation findings provide an understanding of changes since baseline in three areas of the development hypothesis (Exhibit 25): 1. Reading among community school learners represented by the light blue box at the top (evaluation question 1) 2. MESVTEE support to community schools represented by the top left grey box outlined in red (evaluation question 2) 3. Literacy instruction among community schools represented by the grey box outlined in red, second from the left (evaluation question 3). The findings are presented below and are organized by the three evaluation questions listed in Exhibit 8. 5.1. Evaluation Question 1: To what extent have TTL interventions improved early grade literacy achievement among boys and girls in community schools across six provinces compared to baseline? These findings are based on the midline EGRA data and, where applicable, comparison to baseline EGRA data. This section presents results graphically; for EGRA results in tabular form see Annex 1. KEY STATISTICAL TERMS Mean The  average  of  the  numbers;;  a  calculated  “central”  value  of  a  set  of  numbers Median The point where 50 percent of cases are above and 50 percent are below Mode The most common (frequent) response p-value A measure of statistical significance ranging between 0 and 1; lower values are more significant Statistical When there is a high degree of confidence that gains are real and not simply a result Significance of random chance due to sampling Zero Scores The percent of learners who completed no items correctly on an EGRA task; or, the percent of teachers who did not fulfill any criteria correctly within a classroom observation domain Improved reading among learners in community schools The MESVTEE provides more support to community schools TTL advocates to the MESVTEE for increased support to community schools Community schools have skilled teachers and managers TTL supports the MESVTEE to train Head teachers and teachers PCSCs are managing and advocating effectively TTL builds PCSC capacity in school management and community mobilization Community schools have and use teaching and learning materials TTL supports the MESVTEE to develop and disseminate teaching and learning materials Time to Learn Midline Impact Evaluation 29 Finding 1: While most learners cannot complete EGRA tasks, the 2-year trends are positive and significant. Comparisons of baseline and midline EGRA data along the four comparable tasks show a small, but statistically significant positive trend both at the level of provinces and language of instruction). In other words, EGRA data show two different but interrelated aspects: (1) the size of the gains in literacy as measured by EGRA from baseline to midline are positive, although small in absolute terms, and at the same time (2) these gains are generally significant in statistical terms, meaning that there is a high degree of confidence that these gains are real and not simply a result of random chance due to sampling. These results are illustrated in the EGRA tables in Annex 15 , as well as all of the charts in this section. In letter sounds, every province except Lusaka improved significantly. In decoding non-words, every province except Muchinga improved significantly. In reading comprehension, significant gains were seen in all provinces except Lusaka and Muchinga. The percent of learners who completed zero items on a task correctly is referred to as “zero scores.” Midline EGRA data show that on four of seven EGRA tasks the majority of learners could not complete a single item. However, comparison of baseline and midline EGRA data show that zero scores decreased across all comparable tasks as shown in Exhibit 26. These findings are consistent with early phases of an intervention – few children are breaking through to achieve fluent reading, but many more are only taking the first steps. The tasks with which most learners were able to engage in 2014 were orientation to print and language of instruction listening comprehension. This was followed distantly by the letter sounds and English listening comprehension tasks where approximately half of learners could provide at least one correct answer. Nonsense word decoding, oral passage reading, and reading comprehension all had over 80 percent zero scores. EXHIBIT 26: PERCENT OF LEARNERS SCORING ZERO BY TASK (ALL PROVINCES) 5 Annex 1 presents the full midline EGRA results, including disaggregation by province, sex, and language of instruction; results according to the MESVTEE performance level descriptors for oral passage reading and reading comprehension tasks; and comparison to the 2012 baseline EGRA data. 15% 61% 89% n/a 92% 96% n/a 11% 50% 82% 14% 82% 92% 53% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1. Language of Instruction Listening Comprehension 2. Letter Sound Knowledge 3. Non-word Decoding 4. Orientation to Print 5a. Oral Passage Reading 5b. Reading Comprehension 6. English Language Listening Tasks 2012 2014 Time to Learn Midline Impact Evaluation 30 One exception to this positive trend is Lusaka Province where learners showed a significant decline in letter sounds and reading comprehension between baseline and midline. The percentage of zero scores for the decoding non-words task also increased in Lusaka Province from 2012 to 2014 even though the mean words decoded also increased. The reading comprehension decline may be explained in part by the performance of a few exceptional learners in 2012. Nevertheless, the change is surprising given the strong improvement seen in Lusaka Province on the language of instruction listening comprehension task. Although EGRA results are not comparable across languages, Southern Province ranked last among all six provinces on letter sounds, decoding non-words, oral passage reading, and reading comprehension both in 2012 and 2014. Southern Province is the only province where EGRA was conducted in CiTonga and consequently there are no other geographic areas covered by this sample to which these results can be compared. Nevertheless, the letter sounds task, although not strictly comparable, should not differ greatly in difficult across languages and the consistency of this trend is compelling. Finding 2: Listening comprehension scores were relatively higher than other EGRA tasks, but still low overall. At midline, learners performed relatively better on task 1, language of instruction listening comprehension, than on other EGRA tasks, but the overall scores indicate room for improvement as shown in Exhibit 27. Provincial averages on this task ranged from a low of 42 percent of responses correct in Muchinga and Southern Provinces to a high of 64 percent in Lusaka Province. Boys significantly outperformed girls in every province except Southern Province, but while significant, the size of these differences was generally small. In all six provinces almost half (44 percent) of learners could not correctly answer a majority of questions (3 out of 5). This general pattern was replicated across all six provinces. This threshold corresponds to the minimum MESVTEE standard for grade 2 listening comprehension ability according to the Performance Level Descriptors, as indicated by the red vertical bar in Exhibit 27. Weakness in the listening comprehension task is important because understanding the language of instruction is a fundamental first step toward learning to read and, more broadly, engaging successfully in the classroom environment in general. However, the declines in Lusaka Province EGRA results noted above do stand in contrast to this because Lusaka Province also had the highest results of the six provinces on task 1 language of instruction listening comprehension. EXHIBIT 27: DISTRIBUTION OF CORRECT RESPONSES FOR LANGUAGE OF INSTRUCTION LISTENING COMPREHENSION 11.4% 12.8% 20.1% 23.8% 21.3% 10.5% 0.0% 10.0% 20.0% 0 1 2 3 4 5 Total Number Correct Responses (Out of 5) % of Learners MESVTEE standard, grade 2 Time to Learn Midline Impact Evaluation 31 Finding 3: Learners performed better in listening comprehension in the languages of instruction than in English. In all six provinces, learners performed significantly better at midline on task 1: language of instruction listening comprehension than on task 6: English listening comprehension, as presented in Exhibit 28. EXHIBIT 28: MEAN PERCENT CORRECT: LANGUAGE OF INSTRUCTION V. ENGLISH LISTENING COMPREHENSION Language of Instruction English Significance (p-value)* Central 49% 27% <.001 Copperbelt 43% 26% <.001 Eastern 61% 9% <.001 Lusaka 64% 40% <.001 Muchinga 42% 8% <.001 Southern 42% 9% <.001 *Significance for paired sample t-tests, two-tailed These results are logical given that the language of instruction is more often the language spoken at home, and in the case of community schools sampled, the language spoken at school as well. However, the two tasks were not equated and, according to the curriculum, learners are not intended to receive substantial instruction in English in grade 2. Comparable data for 2012 do not exist and 2014 data may in part reflect the effects of switching to familiar language instruction as a result of the new language policy. This finding provides compelling, but limited evidence that in the context of Zambia’s extreme linguistic diversity, the official languages of instruction are a viable medium for teaching literacy in the community school setting. In several remote regions the language of instruction is not the same as the dominant spoken language at home and in school, yet these data indicate the official language of instruction is still preferable to English. There has been concern in urban areas that the local language instruction policy may be detrimental to learners due to ethno-linguistic diversity in the home, linguistic mixing resulting in widespread usage of “town Nyanja,” and overall heightened exposure to and use of English. The data point to the exact opposite. Lusaka Province, which is the most susceptible to all three concerns, exhibits the highest task 1 language of instruction listening comprehension scores among the six provinces. The language of instruction listening comprehension task is not designed to be comparable across languages and in the only other province with the same language of instruction (CiNyanja, Eastern Province), the difference in listening comprehension scores was not significantly different. Lusaka Province had the highest English listening comprehension scores, offering support for the reasonable expectation that community school learners in Lusaka Province have a better understanding of English. Nevertheless, even in Lusaka Province learners responded significantly more accurately to listening comprehension questions in CiNyanja (p<0.001; see Exhibit 28 above). This counters the concern, at least amongst the population of community school learners, that the local language instruction policy may harm urban learners. Finding 4: Boys significantly outperform girls in most areas on most tasks, though the size of those differences remains small in absolute terms. Sex-disaggregated, provincial-level 2014 EGRA data reveal significant differences between boys’ and girls’ performance in over half of the comparisons. Exhibit 45 in Annex 1 provides a complete summary of Time to Learn Midline Impact Evaluation 32 these sex-disaggregated results at the provincial level. Exhibit 47 provides a complete summary of these sex-disaggregated results at the language of instruction level. As noted in the Sample section of Annex 2, and Exhibit 50 in particular, the sample was not designed to show sex disaggregation at provincial level so is too small to ensure accuracy at this level. Even where significant, provincial-level differences in boys’ and girls’ performance may not reflect population differences. Despite this caveat, the trend of boys outperforming girls is notably consistent. Seven tasks and six provinces yield 42 distinct comparison points between boys and girls. Boys significantly outperformed girls in 21 of these comparisons, while girls significantly outperformed boys in just 3 comparisons. Comparisons at the language of instruction level reveal similar trends of boys outperforming girls in a number of tasks. One of the instances of girls performing better is in Southern Province where the overall low results may make the sex-disaggregated differences very sensitive to outliers. In one instance in Southern Province boys outperform girls and in the other five instances differences are not significant. In Eastern Province, the differences in favor of boys are only significant on two of the seven tasks. This makes these two provinces potential exceptions in the trend of boys doing better. In contrast, in Copperbelt and Muchinga Provinces, boys significantly outperformed girls on five of seven subtasks. Central and Lusaka Provinces fall in between these two extremes in terms of the number of tasks in which boys outperformed girls. Baseline findings were inconclusive regarding differences between girls and boys, but this may have been due to the strong floor effect of the 2012 data, which resulted in means close to zero for almost all subpopulations on all tasks. The floor effect remains strong in the 2014 EGRA results and thus this finding should be interpreted in the overarching context of low overall EGRA scores. Finding 5: Most community school learners in the TTL intervention area cannot yet read at grade level, but grade 2 learners have moved closer to the standard since 2012. Exhibit 29 and Exhibit 30 present EGRA results on task 5a oral passage reading and 5b reading comprehension respectively in terms of the MESVTEE Performance Level Descriptors, which set national standards for primary grade literacy competences in Zambia. See Benchmarks for Grade 2 Reading Performance in Annex 2 for more on these standards. As shown in Exhibit 29, an estimated 1.6 percent of community school learners met the minimum MESVTEE grade 2 standard (25 words per minute) for oral passage reading, and no learners placed above the minimum standard. While this result may seem discouraging, at baseline less than 1 percent achieved the minimum standard and substantially more learners were unable to read a single word. This underscores the reality that oral reading fluency—and more foundational skills such as letter sound knowledge—started from an extremely low level and it will take time before sizeable proportions of learners achieve grade-level reading ability. Reading fluency is a trailing indicator along with reading comprehension, and it can be expected to be one of the last indicators to change, trailing even other EGRA tasks that target foundational skills (such as letter sounds and word decoding). Time to Learn Midline Impact Evaluation 33 EXHIBIT 29: ORAL PASSAGE READING RESULTS BY MESVTEE STANDARDS As shown in Exhibit 30, a similar pattern is observed in achievement of reading comprehension standards, with a heavy right skew resulting from the estimated 94.2 percent of the population (91.5- 96.1 percent confidence interval) that cannot answer any comprehension questions correctly. Due to the extent of the sample scoring zero on these two EGRA tasks, population estimates for provincial and sex disaggregated subpopulations are not presented. EXHIBIT 30: READING COMPREHENSION RESULTS BY MESVTEE STANDARDS The USAID Education Strategy Goal 1 definition refers to learners who read with comprehension. The indicator thus refers to the intersection of oral passage reading with the threshold of 80 percent reading comprehension. At midline, 0.4 percent of learners sampled were able to both obtain the minimum standard of 25 words per minute on oral passage reading and answer at least four of five comprehension questions correctly. At baseline, no learners sampled achieved this threshold. 5.2. Evaluation Question 2: How has MESVTEE community school engagement changed since baseline? The Zambia1996 Educating Our Future policy mandates the MESVTEE to provide quality education to all eligible children, including those in community schools.6 In recognition of the important role communities play in education provision in Zambia, the MESVTEE codified the Operational Guidelines for Community Schools in 2007 to operationalize this policy specifically as it pertains to community 6 There is a 2011 Education Act, but the 2007 Operational Guidelines for Community Schools were in response to the 1996 Educating Our Future policy. 92.0% 7.0% 0.7% 0.1% 0.0% 85.0% 13.5% 1.6% 0.0% 0.0% 0.0% 50.0% 100.0% Zero (0) Below Minimum (1-24) Minimum (25-39) Desirable (40-59) Outstanding (60+) % of Learners Words Per Minute 2012 2014 95.5% 1.8% 1.1% 1.6% 0.0% 94.2% 4.7% 0.2% 0.5% 0.4% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Zero Below Minimum (1-2) Minimum (3) Desirable (4) Outstanding (5) % of Learners Correct Responses 2012 2014 Time to Learn Midline Impact Evaluation 34 schools. This was done by outlining the roles and responsibilities of stakeholders. By Zambian policy, education-related documents and policies are supposed to be reviewed 5 years after implementation. As such, in 2013 TTL supported the MESVTEE to evaluate and revise these Guidelines. At the time of this report, the updated guidelines were awaiting final approval from the MESVTEE Permanent Secretary. Upon the approval, the Guidelines will be signed and replace the 2007 version. The TTL baseline and midline evaluations collected data based on the MESVTEE roles and responsibilities stipulated in the 2007 Guidelines and presented in Exhibit 31. EXHIBIT 31: MESVTEE RESPONSIBILITIES TO COMMUNITY SCHOOLS PER 2007 OPERATIONAL GUIDELINES This section presents findings based on two data sources: the community school head teacher questionnaire and the self-administered survey questionnaire to MESVTEE District Education Boards Secretary offices. Both tools focused on the following five MESVTEE responsibilities to community schools: x Financial support through grants7 x Teaching and learning materials x Free basic materials x Continuing professional development (training for teachers and head teachers) x Monitoring. 7 Grants were attributed to the MEVSTEE if they were from any Government of Zambia agency, including Constituency Development Funds. •Provide equitable financial and material resources •Ensure community schools are represented at all levels of decision making and incorporated equally into all directorates •Provide infrastructure development support at district level •Include community school teachers in any professional development activities •Provide technical support/ capacity building to improve quality of teaching and learning •Coordinate and unify all key stakeholders •Register community schools •Integrate community schools in all information management initiatives and distribute standard forms and returns •Monitor and evaluate learning and teaching Monitoring Teacher training Other Grants Time to Learn Midline Impact Evaluation 35 Head teachers were asked about two additional types of MESVTEE support their school received: x Infrastructure support (also given in terms of building materials) x Government teachers deployed to community schools. Baseline and midline percentages presented in the findings below represent the percentage of schools reported by the head teacher or MESVTEE to have received that type of support in the past year. The 2012 data from head teachers was collected at baseline. The 2012 data from the MESVTEE was collected at midline, i.e., MESVTEE district officials were asked to report on support provided to midline community schools in both 2012 and 2014. Finding 6: Data from the MESVTEE district offices and from head teachers differ greatly regarding the amount of MESVTEE support provided to community schools in both 2012 and 2014. MESVTEE district officials reported providing a high proportion of community schools sampled with support in 2012 along all five of the areas asked. By support type, responses ranged from supporting 74 percent of schools with teaching and learning materials to 90 percent with free basic materials (indicated by the dark blue bars in Exhibit 32). This high reporting left little room for “improvement” in 2014 where numbers remained in the 80 percent range, except for the provision of free basic materials, which dropped from 90 percent in 2012 to 56 percent in 2014 and grants, which decreased from 78 percent in 2012 to 64 percent in 2014 (indicated by the red bars in Exhibit 32). Conversely, head teachers reported relatively low levels of MESVTEE support in 2012 for all areas asked. By type of support, the proportion of schools receiving support ranged from 11 percent for teaching and learning materials to 43 percent for grants (indicated by the grey bars in Exhibit 32). In 2014, head teachers reported a substantial increase in support in all areas asked, except for grants (indicated by the light blue bars in Exhibit 32). EXHIBIT 32: MESVTEE SUPPORT PROVIDED IN 2012 AND 2014 AS REPORTED BY MESVTEE AND HEAD TEACHERS Although there is quite a difference between the MESVTEE and head teacher perceptions of MESVTEE support across all five areas, in general there is consistency in the amount of 2014 MESVTEE support in two areas: (1) provision of free basic materials (56 percent reported by the MESVTEE and 60 percent reported by head teachers), and (2) MESVTEE monitoring of community schools (85 percent reported by the MESVTEE and 77 percent reported by head teachers). 78% 74% 90% 86% 89% 64% 79% 56% 91% 85% 43% 11% 13% 39% 16% 30% 69% 60% 63% 77% 0% 20% 40% 60% 80% 100% Grants Teaching and Learning Materials Free Basic Materials Teacher Trainings Monitoring Type of Support MESVTEE 2012 MESVTEE 2014 Head Teacher 2012 Head Teacher 2014 Time to Learn Midline Impact Evaluation 36 Finding 7: Head teachers reported increased support from the MESVTEE since 2012 in five of the seven areas asked. Head teachers reported increases in support from the MESVTEE from 2012 to 2014 in five areas: teaching and learning materials (58 percent increase), free basic materials (47 percent increase), teacher trainings (23 percent increase), monitoring (61 percent increase), and provision of government teachers (11 percent increase) as presented in Exhibit 33. The two areas with the highest increase from 2012 to 2014, according to the MESVTEE, were teaching and learning materials and teacher trainings (both a 5 percent increase) as presented in Exhibit 33. Although head teachers indicated that training support has increased since 2012, nearly 50 percent of teachers interviewed following classroom observation said that they have not received enough training to teach reading and writing. EXHIBIT 33: MESVTEE SUPPORT PROVIDED IN 2012 AND 2014 AS REPORTED BY HEAD TEACHERS The two areas of support that decreased from 2012 to 2014 according to head teachers were grants (by 13 percent) and infrastructure (by 5 percent). While the large increases identified by head teachers in most forms of MESVTEE support are positive, head teachers’ assessment of school needs indicate the decline in grants and infrastructure support may be particularly concerning for this group. When head teachers were asked about the level of resources their school has in relation to its needs for different items, water was the only infrastructure item for which the most common answer (mode) was that the school has sufficient access to it. For all other items, the most common answer was that the material needs of the school were not being met. These items were: teaching and learning materials (85.3 percent), number of classrooms (79.4 percent), basic classroom materials (72.5 percent), male (60.8 percent) and female (58.8 percent) toilets for learners, chalk boards (52 percent), learner benches (47.1 percent), and male staff toilets (32.4 percent). Female staff toilets and desks and chairs for teachers were most commonly said to be entirely nonexistent at sampled schools. The items that were most often reported as being in “adequate supply” were chalk boards and learner benches (one third of schools). There was some, but little, variation across provinces. Finding 8: Head teachers reported receiving more types of support from the MESVTEE and more support from zonal and district offices in 2014 than in 2012. 11% 13% 39% 16% 0.5% 43% 10% 69% 60% 63% 77% 12% 30% 5% 0% 20% 40% 60% 80% 100% Teaching and Learning Materials Free Basic Materials Teacher Trainings Monitoring Government Teachers Grants Infrastructure Type of Support Head Teacher 2012 Head Teacher 2014 Time to Learn Midline Impact Evaluation 37 Head teachers reported an upward trend in support from the MESVTEE to community schools from 2012 to 2014. When head teachers were asked how many forms of support they received from the MESVTEE in 2012, 99 percent reported 0-3 forms compared to 2014 when 62 percent reported receiving four to seven forms of support (see Exhibit 34). The median number of types of MESVTEE support received increased from one at baseline to four at midline; the difference was significant (p<0.001, Mann-Whitney U=1607.5, two-tailed). In 2012, 20 percent of schools received zero types of support, whereas all schools sampled in 2014 received at least one type of support from the MESVTEE. The baseline, which used a more intensive process for analyzing content of open-ended qualitative questions, showed that 95 percent of schools received at least one type of support. EXHIBIT 34: MESVTEE SUPPORT PROVIDED OVER TIME AS REPORTED BY HEAD TEACHERS Head teachers who received MESVTEE support were asked from what level—national, provincial, district, or zonal—had their community school received this support in the previous year; multiple responses were allowed. In 2014, 90 percent reported district and zonal levels compared to 71 percent in 2012 as presented in Exhibit 35. This represents a 16 percent increase in zonal support. At the same time, head teachers reported a 26 percent decrease in national support, and a 20 percent decrease in provincial support. District support remained relatively unchanged. EXHIBIT 35: MESVTEE SUPPORT PROVIDED BY LEVEL AS REPORTED BY HEAD TEACHERS 20% 42% 26% 12% 0% 1% 0% 0% 0% 5% 16% 18% 25% 25% 10% 1% 0% 20% 40% 60% 0 1 2 3 4 5 6 7 Number of Types of Support Received (Out of 7) 2012 (n=194) 2014 (n=102) 74.20% 82.50% 30.90% 33.00% 90.29% 82.35% 10.68% 6.80% 0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00% 100.00% Zonal District Provincial National Type of Support Schools Receiving Support in 2012 Schools Receiving Support in 2014 Time to Learn Midline Impact Evaluation 38 43% 30% 78% 64% 0% 20% 40% 60% 80% 100% 2012 2014 Head Teacher MESVTEE Finding 9: The percentage of community schools receiving grants has reduced, while grant amounts have remained relatively stable. In 2014, 30 percent of head teachers reported receiving MESVTEE grants versus 42.8 percent in 2012, a 13 percent reduction. The MESVTEE reported providing a higher percentage of schools with grants in both periods (78 percent in 2012 and 64 percent in 2014), but nearly the same reduction (14 percent). This reduction is consistent with the overall reduction of funds that may have contributed to the decline. In 2014, schools that received grants reported getting an average grant of 5,300 Kwacha per year, ranging from 2,300 Kwacha per year in Muchinga Province to 13,200 Kwacha per year in Central Province. MESVTEE guidance states that district offices should consider the remoteness of community schools when distributing grants, giving priority to more rural areas. Despite this, no significant correlation was found between the distance from the MESVTEE district office and the amount of funding from the MESVTEE. The MESVTEE reported giving grants to more than twice as many schools as reported by the head teachers, but giving less than half the amounts that the schools themselves report. Finding 10: The type of MESVTEE support reported the most in 2014 by both the MESVTEE and head teachers was monitoring. Of all types of support asked of MESVTEE district officials and head teachers, monitoring was the most reported with 77 percent of head teachers stating that their school had been monitored at least once in the past year, and 85 percent of MESVTEE officials stating they provided monitoring support to these same schools in the past year (see Exhibit 37). Although the percent of schools monitored, as reported by the MESVTEE, decreased by 4 percent, monitoring support reported by head teachers increased by 61 percent (from 16 percent in 2012 to 77 percent in 2014). The divergence between the two data sources shows differing perceptions of the extent of monitoring taking place. This divergence has, however, decreased sharply from over a 70 percent discrepancy in 2012 to about 10 percent in 2014. The increase in monitoring support aligns with TTL interventions of advocacy for community schools with the MESVTEE and transport allowance provisions, as well as other direct support. In 2014 similar proportions of head teachers reported receiving monitoring support across all provinces, ranging from 67 percent of schools monitored in Lusaka Province to 94 percent in Central Province. As seen in Exhibit 37, most head teachers reported being monitored by the MESVTEE in 2014. However, EXHIBIT 36: MESVTEE GRANTS MADE AS REPORTED BY HEAD TEACHERS AND THE MESVTEE EXHIBIT 37: MONITORING SUPPORT REPORTED AS RECEIVED OR PROVIDED BY HEAD TEACHERS AND MESVTEE DISTRICT OFFICIALS, RESPECTIVELY 16% 77% 89% 85% 0% 20% 40% 60% 80% 100% 2012 2014 Head Teacher MESVTEE Time to Learn Midline Impact Evaluation 39 only 25 percent of teachers responded that they were monitored in the classroom by zonal-level MESVTEE officials and 8 percent by district MESVTEE officials. Nearly 80 percent of head teachers and teachers reported that head teachers monitor teachers; however, head teachers indicated that they conducted this monitoring more frequently than teachers indicated receiving such monitoring. Over 80 percent of head teachers say they give feedback following classroom observation, but less than half reported having received instruction on how to give feedback to teachers. According to the MESVTEE district offices, monitoring occurred most frequently in Copperbelt, Eastern, and Lusaka Provinces, and least frequently in Muchinga and Southern Provinces. The differences in monitoring support across provinces are consistent from 2012 to 2014. MESVTEE district office 2012 and 2014 data indicate that schools were most commonly monitored only once a year. 5.3. Evaluation Question 3: To what extent are male and female teachers implementing TTL-supported literacy teaching methods? The classroom observation protocol included seven domains (listed in Exhibit 38) that form the core pillars of early grade literacy lessons. According to TTL, through September 2014, the project’s head teacher and teacher trainings included content that covered all of the seven domains. Exhibit 38 presents the results of the midline classroom observations, and EXHIBIT 39 highlights the percentage of lessons that fulfilled zero criteria for each domain. Across domains, the mean and median percentages of criteria fulfilled are generally close. The proximity of mean and median (the median being the point where 50 percent of cases are above and 50 percent below) indicates that individual teachers generally perform similarly, i.e., there are no extreme outliers pulling the average in one direction as would be the case if there were exceptional lessons that fulfilled far more criteria than the norm. EXHIBIT 38: PERCENT OF CRITERIA FULFILLED BY DOMAIN OUT OF TOTAL POSSIBLE GIVEN ACTUAL LESSON LENGTH Average Median Percent fulfilling 0 criteria Rank Domain 1 (orientation to print) 13% 13% 14% 3 Domain 2 (letter sounds) 21% 17% 11% 1 Domain 3 (word decoding) 19% 19% 13% 2 Domain 4 (oral passage reading) 8% 3% 41% 5 Domain 5 (reading comprehension) 1% 0% 79% 7 Domain 6 (listening comprehension) 2% 0% 54% 6 Domain 7 (writing) 11% 10% 23% 4 Time to Learn Midline Impact Evaluation 40 EXHIBIT 39: PERCENT OF LESSONS FULFILLING ZERO CRITERIA BY DOMAIN Finding 11: Teachers were most often observed practicing letter sounds and orientation to print. Eighty-nine (89) percent of teachers sampled fulfilled at least one letter sounds criterion during the lesson observed, and as seen in Exhibit 39, this domain also had the highest percentage of criteria fulfilled on average of any of the seven domains. Letter sounds criteria are activities that involve instruction on sound-symbol correspondence. Teachers observed successfully fulfilled on average 21 percent of the criteria possible for the domain, the highest of the seven domains measured by the observation protocol. For the orientation to print domain, on average teachers fulfilled 13 percent of the criteria possible. At 14 percent, the proportion of lessons fulfilling zero criteria for this domain conversely means that 86 percent of lessons included at least one orientation to print activity. This indicates that most teachers are capable of offering instruction in this area. While fewer orientation to print criteria were fulfilled on average than the letter sounds and word decoding domains, the zero scores are still relatively low. In light of the low zero scores, the low average number of criteria fulfilled is due to the fact that many teachers observed are teaching writing, but devoting less time to this domain than others. Word decoding combines the alphabetic principle with phonemic awareness in activities related to sounding out words or building words from syllables and letters. Word decoding also had a low zero score of 13 percent and teachers observed during the lesson on average fulfilled 19 percent of the criteria possible for the word decoding domain. For the writing domain, on average teachers fulfilled 11 percent of the possible writing criteria. A zero score at 23 percent for this domain indicates that 77 percent of observed lessons did engage learners in writing activities at least once. This is substantially better than the zero scores for the comprehension and oral reading domains. It also indicates that, in comparison to those domains, the low average 14% 11% 13% 41% 79% 54% 23% 0% 20% 40% 60% 80% 100% Time to Learn Midline Impact Evaluation 41 number of criteria fulfilled in the writing domain is more due to many teachers fulfilling writing criteria, but devoting less time to these activities, than to fewer teachers fulfilling these criteria at all, as is the case with the reading fluency and comprehension domains. Finding 12: Teachers were less often observed practicing reading and listening comprehension. Instruction to purposefully build comprehension was almost entirely absent from the lessons observed, both for reading comprehension (related to material the learner has read her/himself), as well as listening comprehension (related to material the teacher has read to learners). Teachers were infrequently observed incorporating activities related to these two interrelated domains, as indicated by the highest zero scores across all 7 domains at 79 percent and 54 percent, respectively. Oral passage reading had the third highest zero score at 41 percent. The oral passage reading domain includes criteria related to both the teacher modeling fluent reading and children reading connected text. Forty-one (41) percent of teachers observed didn’t fulfill any criteria in this domain, i.e., 41 percent of observed literacy lessons demonstrated no reading beyond disconnected letters or words. During interviews teachers were asked what activities they “practiced 4-5 times a week” and responded: letter sounds, word decoding, copying words from the board, and repeating words after the teacher. When asked about activities “practiced 2-3 times a week, ” teachers said: reading to learners in class, reading comprehension, listening comprehension, having learners memorize whole words by sight, and asking learners to read or write as homework. The only activity teachers said they did once a week (median) or not at all (mode) was “giving learners texts to read that are not in their textbooks.” With the exception of learners reading-out-loud in class, these responses align closely with what was observed during literacy lessons. Generally, teachers reported a positive attitude towards reading. For example, while 53-60 percent of teachers said that learning reading, and respectively writing, is very difficult for learners, over 90 percent thought that all learners can learn how to read and write. While less than half of the teachers overall said that there are differences in how fast boys and girls learn reading and writing, female teachers were more likely to view girls as faster learners and male teachers are more likely to see boys as faster learners. Finding 13: The average length of literacy lessons has significantly increased by over 30 percent compared to baseline. Observation data show that literacy lessons at midline averaged 50 minutes, a 12-minute increase from the 38-minute average lesson time at baseline. This increase of over 30 percent is a significant improvement that represents a tangible change in classroom literacy practice in 2 years. As presented in Exhibit 40, this lengthening trend is observed in each of the six provinces. The sample is only large enough to detect significance at the aggregate level. The standard deviation and range also decreased from baseline to midline, indicating that observed lessons are more consistent in the amount of time devoted to literacy. At baseline, for example, average lesson length ranged from a minimum of 21 minutes in Copperbelt Province to a maximum of 49 minutes in Southern Province (range = 28 minutes). In contrast, that range tightened to a narrower 5 minutes at the midline, ranging from 47 minutes in Lusaka Province to 52 minutes in Southern Province. Time to Learn Midline Impact Evaluation 42 EXHIBIT 40: AVERAGE OBSERVED LITERACY LESSON LENGTH (MINUTES) Province Baseline Midline Central 36.54 48.53 Copperbelt 21.12 49.76 Eastern 35.33 50.65 Lusaka 43.53 47.00 Muchinga 38.52 49.50 Southern 49.38 52.41 Total 37.69 49.62 Finding 14: Literacy lesson successes and challenges are relatively consistent across provinces and teacher sex. Literacy lessons varied slightly, but inconsistently, from province to province. Letter sound and word decoding tasks were consistently observed as the most-practiced domains across provinces, with the exception of Southern Province. On average, teachers in Central Province were observed to practice more letter sound domain criteria than the others: 35 percent on average compared to 21 percent in the six-province aggregate. This was the largest divergence of any province on a domain. Reading and listening comprehension domain criteria were consistently fulfilled less frequently than criteria in other domains in all six provinces. Reading fluency was also observed infrequently, except in Southern Province. Overall, teachers in Central and Muchinga Provinces slightly outperformed the six-province aggregate, but as explained below, these differences cannot be tested for statistical significance. In Muchinga Province teachers fulfilled more criteria on average than nationally in three of the seven domains. In Central Province teachers fulfilled more criteria in two domains. The sum of the differences for each of the seven domains between the provincial and national average was +7 percent in Muchinga Province and +10 percent in Central Province. The latter stems primarily from the strong divergence in letter sounds practice. Eastern and Southern Provinces slightly underperformed the six-province aggregate. Copperbelt and Lusaka Provinces are close to the aggregate, slightly over-performing in some domains, but commensurately underperforming in others. These findings could be random chance because the sample was not constructed to determine statistically significant differences at the provincial level. Overall, the extent of these variations from the aggregate is slight. The overarching finding is that, while there is some variation from province to province and to a lesser extent by teacher sex, literacy lessons are generally emphasizing similar domains. Finding 15: The majority of literacy lessons are conducted in the official language of instruction, but more often in grade 1 than in grade 2. According to head teacher interviews, 96.1 percent of grade 1 and 2 literacy instruction is conducted in the official language of instruction. Head teachers in schools that do not teach in the official language of instruction said they teach in English in grade 2 in Lusaka, the iCiBemba dialect Lala in Central Province, and ciTumbuka in Eastern Province. Time to Learn Midline Impact Evaluation 43 Head teachers indicated that languages other than the official language of instruction are used at times and that the official language of instruction is used less often in grade 2 than in grade 1. The majority (78.4 percent) of head teachers reported that the official language of instruction is used “all of the time” in grade 1 literacy classes and 50 percent in grade 2. Far fewer (17.6 percent) head teachers responded that the official language of instruction is used “more than half of the time” for grade 2. The official language of instruction is used less often in grade 2 because of head teacher’s and teachers’ beliefs about instructing in English. Interview data show that 34.3 percent of head teachers believe English should be used in grade 2, while only 4.9 percent believe that English should be used in grade 1. About a quarter (27.5 percent) of head teachers interviewed responded that teachers do not use the official language of instruction in grades 1 and 2 literacy instruction all the time because they believe that English is required, and 6.9 percent said that teachers think English is more important. Other possible explanations for not teaching in the official language of instruction can largely be ruled out. Only 6.9 percent of head teachers said that the school does not have local language materials for teaching and only 3.9 percent stated that parents do not support the use of the official language of instruction in the classroom. Classroom teachers not speaking the official language of instruction (answer provided by 10.8 percent of head teachers) or classroom teachers not knowing how to teach in the official language of instruction (answer provided by 5.9 percent of head teachers) were not reported as major obstacles to familiar language literacy instruction in grades 1 and 2. The exception was in Lusaka Province where 27.8 percent and 16.7 percent of head teachers respectively responded that they did not teach more often in the language of instruction because they did not know it or did not know how to teach in it. Lusaka Province is also the only province in which parents were reported as not supporting familiar language instruction (22.2 percent). Time to Learn Midline Impact Evaluation 44 EXHIBIT 42: TTL DEVELOPMENT HYPOTHESIS EXHIBIT 41: KEY EVALUATION QUESTIONS 1. To what extent have TTL interventions improved early grade literacy achievement among boys and girls in community schools across six provinces compared to baseline? 2. How has MESVTEE community school engagement changed since baseline? 3. To what extent are male and female teachers implementing TTL-supported literacy teaching methods? 6. CONCLUSIONS AND RECOMMENDATIONS The midline evaluation findings and conclusions were validated at a data consultation meeting in Lusaka in April 2015. At this meeting, external and internal stakeholders confirmed the findings were accurate and grounded in the Zambian context. Recommendations are based on the conclusions, findings, and the data consultation meeting. The conclusions are organized by the midline evaluation questions (see Exhibit 41). Overall, the conclusions support TTL’s development hypothesis (see Exhibit 42) by indicating improvement in the two grey boxes outlined in red, which correspond to evaluation questions 2 and 3, respectively. Findings also demonstrate progress towards TTL’s goal of improved reading among community school learners indicated by the top light blue box, which corresponds to evaluation question 1. 6.1. Conclusions EVALUATION QUESTION 1: TO WHAT EXTENT HAVE TTL INTERVENTIONS IMPROVED EARLY GRADE LITERACY ACHIEVEMENT AMONG BOYS AND GIRLS IN COMMUNITY SCHOOLS ACROSS SIX PROVINCES COMPARED TO BASELINE? Since baseline, learners have improved significantly on the majority of EGRA tasks. This trend is observed by language groups and at the provincial level, although in the latter, Lusaka Province has Improved reading among learners in community schools The MESVTEE provides more support to community schools TTL advocates to the MESVTEE for increased support to community schools Community schools have skilled teachers and managers TTL supports the MESVTEE to train Head teachers and teachers PCSCs are managing and advocating effectively TTL builds PCSC capacity in school management and community mobilization Community schools have and use teaching and learning materials TTL supports the MESVTEE to develop and disseminate teaching and learning materials Time to Learn Midline Impact Evaluation 45 shown backsliding on some tasks. Learners performed relatively better in listening comprehension, orientation to print, and letter sounds, and worse in nonsense word and oral passage reading and reading comprehension. The largest improvements were in the letter sounds tasks. This aligns with classroom observation findings showing that letter sounds occupy a high proportion of instructional time, and is the first area of focus when teaching reading. Without the ability to sound letters, learners are unlikely to be able to read words. Baseline findings were inconclusive regarding differences between girls and boys, but midline findings indicate a significant trend of boys doing better than girls in a number of areas. As such, there is a risk of girls falling behind if TTL does not remain vigilant. However, this should not distract from the pressing need to improve reading for all community school learners and should instead only serve as an early warning sign of the potential risk that, as reading begins to take off in community schools, it may not equally benefit boys and girls. Findings show that in all six provinces learners performed significantly better in listening comprehension in the official language of instruction than in English. This is of crucial importance given Zambia’s current transition to familiar language instruction, both offering evidence that supports the soundness of language policy – which has been supported substantially by USAID and TTL – and indicating that TTL’s efforts to translate policy into practice at the community school level may be having an effect. The linguistic characteristics of EGRA participants bolster this conclusion. The Southern and Eastern African Consortium for Monitoring Educational Quality (SACMEQ) testing consortium showed grade 6 reading levels declining in Zambia from 1995 to 2007, the most recent year for which SACMEQ data are available (Musonda and Kaba n.d.). In this context, TTL has been working not only to improve reading, but also to reverse a negative trend. Therefore, even the small, but statistically significant, improvements in the findings are positive. In light of research by Pritchett and Beatty, the rates of improvement observed in the TTL intervention area appear strong, particularly given the historical rates of progress observed in the broader southern and eastern African region (Beatty and Pritchett 2012). Given that MESVTEE support shows strong improvement and teacher practice appears to be strengthening, using TTL’s development hypothesis, it is reasonable to expect the rate of EGRA improvement to continue and possibly even accelerate over the next 2 years. EGRA results are a trailing indicator of all the TTL intermediate results, so EGRA scores would reasonably be expected to be the last outcome to show change. EVALUATION QUESTION 2: HOW HAS MESVTEE COMMUNITY SCHOOL ENGAGEMENT CHANGED SINCE BASELINE? Although perceptions differ between the MESVTEE and head teachers on the amount of MESVTEE support provided to community schools in 2012 and 2014, head teachers report a marked increase in and more forms of MESVTEE support since 2012. This shift perceived by head teachers includes noticeable improvements in the proportions of schools receiving teaching and learning materials, free basic materials, training, and monitoring visits. Head teachers also perceive more MESVTEE support from zonal levels in 2014 than in 2012, which could be an indication of MESVTEE decentralization or closer relationships and communication between community schools and zonal MESVTEE officials. The large discrepancy in perceptions about the percentage of schools receiving grants, the average grant size, and other types of support cannot be explained through recall issues alone. Head teachers may be unclear about what support to community schools can be attributed to the MESVTEE versus other sources outside of the Government of Zambia, while the MESVTEE may be over-reporting support Time to Learn Midline Impact Evaluation 46 provided. It is possible that more communication and clearer definitions are needed to align views on the provision of financial support. Some project stakeholders felt that the upward trend in MESVTEE monitoring of community schools should have produced a larger corresponding improvement in learner performance. This could reflect poor quality in terms of depth and length (i.e., time spent at the school) of the monitoring provided. EVALUATION QUESTION 3: TO WHAT EXTENT ARE MALE AND FEMALE TEACHERS IMPLEMENTING TTL-SUPPORTED LITERACY TEACHING METHODS? In the 2 years since baseline, the average length of literacy lessons has significantly increased. This increase aligns with the Year 2 performance evaluation finding that many teachers and head teachers increased the amount of classroom time devoted to literacy as a result of the 1 hour per day curriculum guidance promoted through TTL trainings (page 24). The current literacy teaching practices among teachers are positive, but insufficient to reach minimal MESVTEE standards for learner performance. Findings show that teachers appear to be including new literacy techniques into their lessons, although there is still far to go to meet the standards. This may indicate inconsistencies in TTL training implementation or that core training content is not being fully absorbed by teachers. Overall, findings indicate that teachers are applying at least some of the knowledge and skills in which they have been trained. For example, teachers have generally shown capacity to apply phonics-based fundamentals. The TTL baseline did not systematically collect data on the presence or absence of letter sound teaching, but anecdotal data indicate that the approach to literacy was not phonics-based, in which case the extent of letter sound instruction in 2014 would point to a positive change. Findings also show a gap between classroom focus on oral reading and comprehension that build learners’ ability to read with comprehension. In order for readers to achieve fluent reading, i.e., reading with both speed and accuracy, it is fundamental that they observe this behavior modeled by successful readers and have the opportunity to read more than single words. 6.2. Recommendations 1. TTL should work with the MESVTEE and head teachers to accelerate and improve the quality of formative community school monitoring. x TTL should strengthen its efforts to increase the use of classroom observation as a formative monitoring tool that supports and reinforces other teacher development activities. Strengthening classroom observation requires building understanding of the role of monitoring as well as practical skills to observe teachers in class, provide appropriate and timely feedback, and tailor ongoing technical support to respond to areas of weakness observed. All community school stakeholders—MESVTEE, head teachers, PCSCs, parents, community leaders—should be actively engaged in monitoring classroom instruction. Monitoring, particularly that conducted by MESVTEE standards officers, should be seen as a formative process that guides schools and teachers in areas for improvement, and not just a summative exercise in data collection. TTL is well-placed to support the MESVTEE and increase understanding of the importance of monitoring teachers’ literacy instruction. x TTL should intensify its work with the MESVTEE to relieve or reduce zonal in-service coordinators’ teaching loads in order to increase their availability to monitor community schools more frequently and effectively. TTL could work with the MESVTEE to explore policy options for strengthening the Time to Learn Midline Impact Evaluation 47 office of the zonal in-service coordinator in order to improve monitoring. Additional guidance or capacity building for zonal in-service coordinators’ could harness the full potential of this monitoring as a formative opportunity for the schools. TTL could use e-EGRA Instruct to allow zonal in-service coordinators to identify literacy areas that need more support at the school level, and provide teachers with suggestions for how to address these areas. x TTL should use classroom observation findings to initiate further dialogue with the MESVTEE about the proper allocation of time across various domains during literacy lessons. The midline presents the first real attempt to quantify the extent to which literacy lessons fulfill the core criteria expected by the MESVTEE, because the midline classroom observation protocol measures not just whether criteria were fulfilled, but for what period of time they were fulfilled. This dialogue could prompt further conclusions regarding how teachers are currently allocating classroom instructional time across the various literacy domains versus the desired allocation across domains. 2. TTL and the MESVTEE should identify the barriers head teachers and teachers are facing in providing comprehensive and equitable early grade literacy lessons. x TTL should investigate what is preventing teachers from integrating more comprehension instruction and time for learners to read connected text into reading lessons. TTL should work with the MESVTEE to explore classroom-related factors that are leading to weak EGRA scores in oral passage reading and reading comprehension. TTL can use the 2015 TTL performance evaluation to explore these factors in order to focus its teacher development activities to enable teachers to spend more time on these tasks. The 2015 TTL performance evaluation could possibly consider the distinct needs of the group of teachers who are not doing any learner reading or comprehension activities, as well as the needs of teachers who are offering learners the opportunity to read but are not asking comprehension questions afterwards. Any efforts should intentionally explore what supports teacher knowledge and skill acquisition and attitude change, and consider implications of community school resource limitations. x TTL and the MESVTEE should investigate further the differences between community school boys’ and girls’ enrollment rates, drop-out rates, and literacy performance. Because the potential scope of such investigation is vast, TTL should encourage the MESVTEE to include this topic in its research agenda so that the MESVTEE can engage the broader Zambian educational research community in this work. The MESVTEE should also investigate these emerging trends and, where findings in this evaluation report align with existing data, the MESVTEE should highlight this convergence as bolstering evidence of these trends. One topic that is particularly poorly understood and in need of better understanding at present is whether the decline in girls’ enrollment as grade increases is due to dropout or other factors. TTL should also use its research capacity building work as an opportunity to explore differences in boys’ and girls’ EGRA performance, though TTL should ensure that the discourse does not lose sight of the fact that boys also have a long way to go to achieve MESVTEE reading standards. 3. TTL and the MESVTEE should strengthen teacher training on comprehension and oral passage reading where teachers are struggling. x TTL should consider ways to intensify training focus on reading and listening comprehension and oral passage reading, which are critical for learners’ ability to read fluently, understand passages, and answer comprehension questions. TTL and the MESVTEE could train teachers directly rather than cascade through head teacher trainings (budget permitting). TTL could utilize the MESVTEE zonal in￾service coordinators more as part of its strategy to intensify zonal in-service coordinators’ support to community schools and provide these coordinators with the instruments / guides they need to focus on these comprehension areas. Time to Learn Midline Impact Evaluation 48 x TTL should work with the central MESVTEE and Curriculum Development Centre on future in-service content. TTL, the MESVTEE, and the USAID Read to Succeed project can jointly work to include reading and listening comprehension in the new pre- and in-service teacher training curriculum that the Directorate of Teacher Education and Specialized Services is currently developing. These curricula could also include gender equality in order to help teachers be more aware of gender biases they may have and how these biases may impact learners’ reading acquisition. x TTL and the MESVTEE should increase their support to teachers on how to teach writing and how to encourage teachers to teach learners how to write. While the TTL EGRA did not asses writing competences, based on classroom observation findings, project stakeholders reiterated the link between writing and reading acquisition for early grade learners. Future trainings should also focus on improving teachers’ skills in teaching letter sounds to ensure that the relative focus of teacher trainings on letter sounds activities translates into reading gains. x TTL should design new—or strengthen existing—materials that accompany trainings (e.g., videos in phones) to build skills in reading and listening comprehension. Teacher trainings should also support teachers to use the full range of teaching and learning materials available, especially new TTL￾distributed materials that provide grade-level text in the official language of instruction specifically designed for literacy acquisition. Teaching and learning material use should include opportunities for learners to read for themselves, i.e., usage rates should increase among both teachers and learners. Time to Learn Midline Impact Evaluation 49 ANNEXES Annex 1. EGRA Summary Tables by Province and Language Annex 2. Detailed Methods and Limitations Annex 3. Evaluation Questions by Data Source and Indicator Annex 4. Documents Reviewed Annex 5. Evaluation Tools Time to Learn Midline Impact Evaluation 50 ANNEX 1. EGRA SUMMARY TABLES BY PROVINCE AND LANGUAGE Exhibit 43 below summarizes the midline EGRA results, presenting point estimates along all seven tasks for the population of learners enrolled in registered community schools. EXHIBIT 43: 2014 EGRA RESULTS BY PROVINCE Task 1: Language of instruction listening comprehension Task 2: Letter sounds Mean % correct Percent scoring 0 Standard Error Confidence Interval Mean letters / minute (max score = 100) Percent scoring 0 Standard Error Confidence Interval Lower Upper Lower Upper Central 49% 12% 0.045 48% 51% 9.26 27% 0.272 8.73 9.79 Copperbelt 43% 14% 0.031 41% 44% 5.05 40% 0.133 4.79 5.31 Eastern 61% 6% 0.028 60% 62% 4.90 38% 0.130 4.65 5.15 Lusaka 64% 3% 0.021 63% 65% 3.89 63% 0.147 3.60 4.18 Muchinga 42% 16% 0.049 40% 44% 6.50 32% 0.238 6.03 6.97 Southern 42% 31% 0.040 40% 43% 1.35 77% 0.077 1.20 1.50 Task 3: Decoding non-words Task 4: Orientation to print Mean words / minute (max score = 50) Percent scoring 0 Standard Error Confidence Interval Mean % correct Percent scoring 0 Standard Error Confidence Interval Lower Upper Lower Upper Central 2.96 71% 0.202 2.56 3.36 94% 3% 0.018 93% 95% Copperbelt 2.14 85% 0.153 1.84 2.44 60% 31% 0.029 58% 62% Eastern 1.82 76% 0.098 1.63 2.01 76% 19% 0.024 75% 78% Lusaka 1.12 89% 0.070 0.98 1.26 92% 5% 0.015 91% 93% Muchinga 2.07 76% 0.167 1.74 2.40 85% 9% 0.029 83% 87% Southern 0.61 92% 0.057 0.50 0.72 82% 12% 0.022 81% 83% Time to Learn Midline Impact Evaluation 51 Task 5a: Oral passage reading Task 5b: Reading comprehension Mean words / minute Percent scoring 0 Standard Error Confidence Interval Mean % correct Percent scoring 0 Standard Error Confidence Interval Lower Upper Lower Upper Central 2.47 70% 0.215 2.05 2.89 5% 88% 0.025 4% 6% Copperbelt 2.46 82% 0.180 2.11 2.81 4% 95% 0.017 3% 4% Eastern 2.51 72% 0.147 2.22 2.80 6% 83% 0.019 5% 7% Lusaka 1.09 89% 0.079 0.94 1.24 2% 94% 0.010 2% 3% Muchinga 2.69 78% 0.231 2.24 3.14 4% 91% 0.023 3% 5% Southern 0.55 94% 0.058 0.44 0.66 1% 96% 0.006 1% 1% Task 6: English listening comprehension Mean % correct Percent scoring 0 Standard Error Confidence Interval Lower Upper Central 27% 35% 0.044 25% 29% Copperbelt 26% 35% 0.028 25% 27% Eastern 9% 76% 0.019 8% 10% Lusaka 40% 45% 0.031 39% 41% Muchinga 8% 81% 0.029 6% 9% Southern 9% 75% 0.021 8% 9% Time to Learn Midline Impact Evaluation 52 EXHIBIT 44: BASELINE V. MIDLINE EGRA RESULTS BY PROVINCE Task 1*: Language of instruction listening comprehension Key Mean % correct p-value % scoring zero correct Significant at: Baseline Midline Baseline Midline 5% Central 30% 49% <0.001 24% 12% 1% Copperbelt 41% 43% 0.006 7% 13% Eastern 36% 61% <0.001 10% 5% Lusaka 32% 64% <0.001 23% 2% Muchinga 36% 42% <0.001 28% 14% Southern 55% 42% <0.001 1% 31% Task 2: Letter sounds Task 3: Decoding non-words Mean LPM (max 100) p-value % scoring zero correct Mean WPM (max 50) p-value % scoring zero correct Baseline Midline Baseline Midline Baseline Midline Baseline Midline Central 2.24 9.26 <0.001 68% 26% 0.46 2.96 <0.001 96% 68% Copperbelt 2.16 5.05 <0.001 74% 38% 0.99 2.14 <0.001 89% 82% Eastern 3.49 4.9 <0.001 58% 40% 1.54 1.82 0.014 83% 77% Lusaka 4.21 3.89 0.042 34% 63% 0.74 1.12 <0.001 84% 86% Muchinga 4.09 6.5 <0.001 54% 29% 1.97 2.07 0.635 84% 75% Southern 0.48 1.35 <0.001 90% 76% 0.31 0.61 <0.001 97% 93% Task 5a: Oral passage reading Task 5b: Reading comprehension Mean WPM p-value % scoring zero correct Mean % correct p-value % scoring zero correct Baseline Midline Baseline Midline Baseline Midline Baseline Midline Central 0.54 2.47 <0.001 97% 68% 1% 5% <0.001 98% 87% Copperbelt 1.34 1.9 <0.001 91% 81% 2% 3% <0.001 95% 94% Eastern 1.9 2.1 0.157 89% 75% 1% 6% <0.001 98% 85% Lusaka 0.58 0.91 <0.001 89% 88% 6% 2% <0.001 91% 94% Muchinga 1.96 2.07 0.673 90% 78% 3% 4% 0.231 94% 92% Southern 0.37 0.59 0.001 98% 94% 0% 1% <0.001 98% 96% *denotes non-equated measures Time to Learn Midline Impact Evaluation 53 EXHIBIT 45: 2014 EGRA RESULTS BY PROVINCE, DISAGGREGATED BY SEX* Key Significant at: 5% 1% Task 1: Language of instruction listening comprehension Task 2: Letter sounds Mean % correct p-value % scoring zero correct Mean LPM (max 100) p-value % scoring zero correct Boys Girls Boys Girls Boys Girls Boys Girls Central 51% 47% .015 6.8 16.8 9.66 8.75 .098 22.2 32.6 Copperbelt 46% 40% <.001 11.3 17.1 4.39 5.61 <.001 41.2 38.6 Eastern 64% 59% <.001 4.1 6.8 4.89 4.91 .929 33.6 41.8 Lusaka 65% 62% <.001 1.4 4.3 3.62 4.15 .069 67.4 58.7 Muchinga 45% 39% .001 11.2 20.3 6.96 6.08 .066 31.1 32.5 Southern 40% 43% .140 33 30 0.82 1.85 <.001 83.2 70.1 Task 3: Decoding non-words Task 4: Orientation to print Mean WPM (max 50) p-value % scoring zero correct Mean % correct p-value % scoring zero correct Boys Girls Boys Girls Boys Girls Boys Girls Central 3.84 1.8 <.001 67.1 75.2 57% 56% .195 1.4 4.6 Copperbelt 3.48 0.99 <.001 84.6 84.8 38% 34% .003 26.7 34.8 Eastern 1.92 1.71 .292 71.2 79.9 46% 46% .644 17.1 19.9 Lusaka 1.27 0.96 .024 89.4 88.6 54% 56% .005 5.7 3.5 Muchinga 2.68 1.54 .001 69.2 82.1 55% 48% <.001 4.7 13 Southern 0.67 0.56 .307 92.3 94 51% 47% <.001 9 13 Time to Learn Midline Impact Evaluation 54 Task 5a: Oral passage reading Task 5b: Reading comprehension Mean WPM p-value % scoring zero correct Mean % correct p-value % scoring zero correct Boys Girls Boys Girls Boys Girls Boys Girls Central 4.06 2.09 <.001 66 75.4 7% 3% <.001 84.4 92.4 Copperbelt 4.03 1.13 <.001 82.6 82.3 7% 1% <.001 92 98.1 Eastern 2.63 2.38 .405 73.1 71.2 6% 6% .703 81.5 83.6 Lusaka 1.17 1.02 .345 91.5 87.1 2% 2% .766 95 92.2 Muchinga 3.3 2.15 .014 72.6 82.9 5% 3% .041 86.9 95.1 Southern 0.53 0.64 .377 94.4 94 1% 1% .846 96 97 Task 6: English listening comprehension Mean % correct p-value % scoring zero correct Boys Girls Boys Girls Central 29% 25% .068 32.7 37.4 Copperbelt 27% 26% .322 33.3 36.1 Eastern 12% 6% <.001 69.9 81.5 Lusaka 46% 34% <.001 41.1 48.2 Muchinga 11% 5% <.001 73.8 87.8 Southern 8% 9% .258 73.4 77.3 * Significance given by p-value of t-tests for equality of independent means; p-values adjusted for comparisons violating equality of variance assumption (tested using Levene's test at 5%) Time to Learn Midline Impact Evaluation 55 EXHIBIT 46: 2014 EGRA RESULTS BY LANGUAGE Task 1: Language of instruction listening comprehension Task 2: Letter sounds Mean % correct Percent scoring 0 Standard Error Confidence Interval Mean letters / minute (max score = 100) Percent scoring 0 Standard Error Confidence Interval Lower Upper Lower Upper CiNyanja 63% 3% 0.017 62% 63% 4.33 53% 0.101 4.13 4.53 CiTonga 42% 31% 0.04 40% 43% 1.35 76% 0.077 1.20 1.50 iCiBemba 44% 13% 0.023 43% 45% 6.43 33% 0.115 6.20 6.66 Task 3: Decoding non-words Task 4: Orientation to print Mean words per minute (max score = 50) Percent scoring 0 Standard Error Confidence Interval Mean % correct Percent scoring 0 Standard Error Confidence Interval Lower Upper Lower Upper CiNyanja 1.42 82% 0.058 1.31 1.53 85% 12% 0.014 84% 86% CiTonga 0.61 93% 0.057 0.50 0.72 82% 11% 0.022 81% 83% iCiBemba 2.33 77% 0.102 2.13 2.53 74% 18% 0.019 73% 76% Task 5a: Oral passage reading Task 5b: Reading comprehension Mean words per minute Percent scoring 0 Standard Error Confidence Interval Mean % correct Percent scoring 0 Standard Error Confidence Interval Lower Upper Lower Upper CiNyanja 1.43 82% 0.066 1.30 1.56 4% 90% 0.01 6% 7% CiTonga 0.59 94% 0.062 0.47 0.71 1% 96% 0.006 1% 2% iCiBemba 2.08 77% 0.093 1.90 2.26 4% 92% 0.011 6% 7% Task 6: English listening comprehension Mean % correct Percent scoring 0 Standard Error Confidence Interval Lower Upper CiNyanja 27% 51% 0.022 26% 27% CiTonga 9% 75% 0.021 8% 9% iCiBemba 22% 43% 0.021 21% 23% Time to Learn Midline Impact Evaluation 56 EXHIBIT 47: 2014 EGRA RESULTS BY LANGUAGE, DISAGGREGATED BY SEX* Key Significant at: 5% 1% Task 1: Language of instruction listening comprehension Task 2: Letter sounds Mean % correct p-value % scoring zero correct Mean LPM (max 100) p-value % scoring zero correct Boys Girls Boys Girls Boys Girls Boys Girls CiNyanja 65% 61% <.001 2% 4% 4.18 4.48 .132 51% 54% CiTonga 40% 43% .140 33% 30% 0.82 1.85 <.001 83% 70% iCiBemba 47% 41% <.001 8% 17% 6.48 6.39 .681 33% 33% Task 3: Decoding non-words Task 4: Orientation to print Mean WPM (max 50) p-value % scoring zero correct Mean % correct p-value % scoring zero correct Boys Girls Boys Girls Boys Girls Boys Girls CiNyanja 1.55 1.28 .019 81% 84% 51% 52% .034 12% 11% CiTonga 0.67 0.56 .307 92% 94% 51% 47% <.001 9% 13% iCiBemba 3.41 1.29 <.001 74% 81% 47% 42% <.001 13% 22% Task 5a: Oral passage reading Task 5b: Reading comprehension Mean WPM p-value % scoring zero correct Mean % correct p-value % scoring zero correct Boys Girls Boys Girls Boys Girls Boys Girls CiNyanja 1.5 1.35 .234 84% 81% 4% 4% .850 91% 89% CiTonga 0.53 0.64 .377 94% 94% 1% 1% .846 96% 97% iCiBemba 2.99 1.21 <.001 73% 80% 6% 2% <.001 87% 96% Task 6: English listening comprehension Mean % correct p-value % scoring zero correct Boys Girls Boys Girls CiNyanja 31% 22% <.001 43% 59% CiTonga 8% 9% .258 73% 77% iCiBemba 24% 21% <.001 40% 47% * Significance given by p-value of t-tests for equality of independent means; p-values adjusted for comparisons violating equality of variance assumption (tested using Levene's test at 5%) Time to Learn Midline Impact Evaluation 57 EXHIBIT 48: BASELINE VERSUS MIDLINE EGRA RESULTS BY LANGUAGE Key Significant at: 5% 1% Task 1*: Language of instruction listening comprehension Baseline mean % correct Midline mean % correct p-value Baseline % scoring 0 Midline % scoring 0 CiNyanja 34% 63% <.001 19% 3% CiTonga 55% 42% <.001 1% 31% iCiBemba 36% 44% <.001 17% 13% Task 2: Letter sounds Task 3: Decoding non-words Baseline mean Midline mean p-value Baseline % scoring 0 Midline % scoring 0 Baseline mean Midline mean p-value Baseline % scoring 0 Midline % scoring 0 CiNyanja 3.96 4.33 0.001 42% 53% 1.01 1.42 <.001 84% 82% CiTonga 0.48 1.35 <.001 90% 76% 0.31 0.61 <.001 97% 93% iCiBemba 2.5 6.43 <.001 68% 33% 0.92 2.33 <.001 91% 77% Task 5a: Oral passage reading Task 5b: Reading comprehension Baseline mean Midline mean p-value Baseline % scoring 0 Midline % scoring 0 Baseline mean % correct Midline mean % correct p-value Baseline % scoring 0 Midline % scoring 0 CiNyanja 1.03 1.43 <.001 89% 82% 4% 4% 0.098 94% 90% CiTonga 0.37 0.59 0.001 98% 94% 0% 1% <.001 98% 96% iCiBemba 1.10 2.08 <.001 93% 77% 1% 4% <.001 96% 92% *denotes non-equated measures Time to Learn Midline Impact Evaluation 58 ANNEX 2. SAMPLE, METHODS, INTER-RATER RELIABILITY, AND LIMITATIONS Sample Exhibit 49 summarizes the stages of the sampling design, as described in section 3.1.1. EXHIBIT 49: SAMPLING STAGES TTL’s interventions target the population of all community schools in its six-province intervention area. Consequently, the sample drew from the population of registered community schools in these provinces, all of which should benefit from TTL; there is no separation between ‘intervention’ and ‘comparison’ schools in the sample. The MESVTEE 2013 annual school census data provided the sampling frame. After removing schools that lacked enrollment figures or school type (e.g., government, community, private, grant-aided), the frame contained 1,251 schools meeting the sampling criteria. Exhibit 50 shows the target midline sample. The statistical test for learner outcomes is an independent t￾test, one tailed, d=.3, alpha=.05. The 95.9 percent power refers to differences in sex-aggregated provincial-level means across evaluation phases. The sample provides 77.3 percent power for differences in sex-disaggregated means across evaluation phases at the provincial level. Power analysis calculations are for comparison between midline and endline. The statistical test for classroom practice outcomes is independent samples t-test, one tailed, d=.35, alpha=.05. The statistical power is only for comparison between midline and endline evaluations at the provincially-aggregated level; replacement of the classroom observation tool at midline renders comparison with baseline not applicable. If a longitudinal design is used for measuring evolution in teacher practice between midline and endline (using school as the unit of analysis), the statistical power would be 96.6 percent (paired samples t-test). Stage of Sampling Population Type of Sampling Sample Stage 1 (clusters) TTL Intervention Schools in each of 6 provinces (6 samples) Probability Proportional to size; Lusaka Province stratified by rural/urban School In schools with more than 1 section of grade 2: Intermediary stage Grade 2 classes Simple Random Class Stage 2 Learners in selected Grade 2 class Simple Random stratified by sex Learners Time to Learn Midline Impact Evaluation 59 EXHIBIT 50: TARGET MIDLINE SAMPLE Per Province Six Provinces Schools (Sample stage 1; cluster) Learners (Sample Stage 2) Total Schools Total Learners Male Female Total Midline 17 128 128 256 102 1,536 Endline 17 128 128 256 102 1,536 Power n/a 77.3% 95.9% 79.4% n/a Sample expectations were met (or surpassed) in all provinces except Muchinga Province, which only achieved a sample of 16 schools (230 learners) due to premature school closing for caterpillar collection in Mpika District. Data collection was extended in Muchinga Province, but the onset of rains constrained the ability to meet the target sample. The target sample identified alternate schools for use in the case of a sampling “refusal”, which occurs when data cannot be collected from a selected school for any reasons, e.g. the school has closed, is inaccessible, or does not consent to the procedures. Refusals were removed from the sampling frame post-data collection. Four schools (4% of the sampled schools) selected for sampling were refusals as a result of having been upgraded from community schools to government schools since the 2013 school census data were reported. These four schools were divided equally between Central and Muchinga Provinces. Anecdotal evidence as well as stakeholder feedback during the data validation meeting in Lusaka indicated that the MESVTEE often upgrades the “best” community schools. All baseline and midline EGRA data presented in this report are weighted. Sampling weights were calculated as the inverse of the probability of selection for each learner, taking into account stage 1 and, where applicable, intermediary stage sampling. Methods EGRA: A literacy assessment focusing on foundational pre-reading and reading competencies proven through research and instrument validation to be necessary foundational skills for early reading and literacy acquisition. EGRA is administered orally to one learner at a time in the language of literacy instruction. The USAID Read to Succeed project, which works with government primary schools, and the Education Data for Decision Making mechanism (EdData II) administered EGRA in parallel with the TTL midline impact evaluation. To maintain comparability across these three USAID activities, all three entities used the same EGRA. The 2014 EGRA follows the general structure of the 2012 EGRA conducted as part of the TTL baseline with modifications. These modifications were made to respond to MESVTEE and TTL project priorities and the new MESVTEE-issued performance level descriptors, with support from the USAID Read to Succeed project in order to provide concrete benchmarks for early grade literacy performance. The modifications are: 1. Inclusion of orientation to print in all provinces, which was introduced during the second round of baseline data collection in Central, Copperbelt, Muchinga, and Southern Provinces in 2012, but had not been done in the first round of baseline data collection in Eastern and Lusaka Provinces in 2012. Time to Learn Midline Impact Evaluation 60 2. Substitution of English vocabulary with English listening comprehension in order to more accurately capture learner understanding of the English language (the official language in Zambia) and allow for comparability with the local language listening comprehension task. To mitigate against potential bias that can arise through assessment leakage, tasks 5a oral passage reading and 5b reading comprehension in all three languages used text passages that were equated with the 2012 EGRA rather than the same passages. Equating was conducted by EdData II using the means ratio methods, with ratios derived from a matched pairs study in September 2014. For additional information on the equating process see Report on the Equating Pilot Study. As a result, the 2014 EGRA is comparable with the 2012 EGRA along the following four tasks: x Task 2: Letter sound knowledge x Task 3: Non-word decoding x Task 5a: Oral passage reading x Task 5b: Reading comprehension. Classroom observation protocol: At baseline, the evaluation team collected data on classroom literacy practice using an adapted version of the EDC’s Standards-based Classroom Observation Protocol for literacy (SCOPE). The MESVTEE data collectors found the SCOPE difficult to administer and, together with TTL technical specialists, felt it focused on higher-order pedagogical practices rather than the specific early grade literacy instructional methods promoted by the TTL training program. Reflecting these two weaknesses, the baseline SCOPE data exhibited very low inter-rater reliability scores and weak correlation with EGRA scores. Consequently, the evaluation team developed the TTL￾specific classroom observation protocol for the midline evaluation to capture literacy instruction practices. With this tool, the evaluation team was able to collect specific descriptive data on instructional delivery with high reliability. The tool included a post-observation teacher questionnaire that provided insight into teachers’ beliefs and background. When the teacher observed was also the community school head teacher questionnaire respondent, the post-observation teacher questionnaire component was omitted to avoid placing an undue time burden upon a single respondent. Community school head teacher questionnaire: In each sampled school, the head teacher was supposed to be interviewed using this questionnaire. If the head teacher was absent on the day of the school visit, another school representative was interviewed instead. The evaluation team made minor updates to the baseline version of the community school head teacher questionnaire to account for changes in project implementation, enable measurement of exposure to TTL interventions, and improve question clarity and ease of administration for administrators and respondents. The evaluation team retained core questions for comparison to baseline data to respond to evaluation question 2, and to help explain learner EGRA scores. This questionnaire was modeled on Snapshot of School Management Effectiveness (SSME) tools and captured key data such as school enrollment, external support types and sources, and parental engagement. Benchmarks for Grade 2 Reading Performance Evaluation finding 5 analyzes EGRA data in terms of the MESVTEE standards for grade two performance. These standards can be found in the Reading Performance Level Descriptors for Grades 1 – 4, and became MESVTEE policy in February 2014. Exhibit 51 below presents the grade 2 performance standards for oral passage reading and reading comprehension. Time to Learn Midline Impact Evaluation 61 EXHIBIT 51: ZAMBIA GRADE 2 READING PERFORMANCE STANDARDS Task Below Minimum Minimum Desirable Outstanding Oral Passage Reading Learner is able to read less than 25 words in 1 minute. Learner is able to read at least 25 words in 1 minute. Learner is able to read at least 40 words in 1 minute. Learner is able to read at least 60 words in 1 minute. Reading Comprehension Learner is able to answer 0-2 of 5 literal questions correctly. Learner is able to read a passage and answer 3 of 5 literal questions correctly. Learner is able to read a passage and answer 4 of 5 literal questions correctly. Learner is able to read a passage and answer 5 of 5 literal questions correctly. While the standards provided in the performance level descriptors reference “grade-level text”, there is no definition of the difficulty of language appropriate for different grades. There are also no grade-level texts that use MESVTEE-sanctioned guidelines for grade-level text (e.g., number of words, level of vocabulary, number of lines per page, or size of font for print). Despite these caveats, the advent of these standards was the first attempt to set national literacy standards for primary grades in Zambia, and thus represent a notable step forward that is useful in EGRA instrument adaptation. The continued specification of these standards can help further improve Zambia’s EGRA instruments in future years. Inter-rater Reliability The classroom observation protocol and EGRA are quantitative measures that require subjective assessments by data collectors. Inter-rater reliability testing determines the level of agreement between observers (data collectors) in order to ensure that different observers give consistent estimates of the same phenomenon. It is, therefore, a key component of ensuring data quality for such tools. EGRA and classroom observation protocol training included inter-rater reliability testing in order to ensure trainees could consistently implement these tools. Additionally, 12 percent of the classroom observations were sampled for inter-rater reliability testing while in the field for data collection. The results presented below calculate inter-rater reliability using the intra-class correlation coefficient with a two-way mixed model specifying absolute agreement. For training scenarios, the single measure was used because trainees were compared against a single master score determined carefully by the training facilitators in conjunction with the tool developers (a “gold standard” score). For field scenarios, the average measure was used as the “correct” scoring cannot be determined in this setting. EGRA Inter-rater Reliability Results Two formal inter-rater reliability tests were conducted during the EGRA training, the first on day 3 (in iCiBemba) and the second on day 5 (in CiNyanja). Numerous informal reliability sessions throughout the week helped trainees improve their reliability and gain an understanding of the importance of inter-rater reliability in EGRA administrations. Only one trainee failed to achieve the 90 percent reliability threshold and, as a result, was not part of the data collection team. Classroom Observation Inter-rater Reliability Results The classroom observation protocol inter-rater reliability can be calculated either by comparing observers’ scores for each of the 23 criteria or by comparing observers’ scores by the domains listed in Exhibit 11. The latter would be expected to present higher inter-rater reliability scores a priori by eliminating some of the variations in scoring of individual criteria; this approach also reflects the analysis presented in this report, which is by domain. Time to Learn Midline Impact Evaluation 62 Formal inter-rater reliability scoring was conducted on the final day of the classroom observation protocol training; all trainees scored above the 80 percent threshold set in the midline implementation plan. Scores ranged from a low of 83 percent to a high of 90 percent. For the classroom observation protocol, inter-rater reliability testing was also conducted on 12 out of the 102 lessons observed (12 percent of the sample), in order to ensure that observers remained consistent during actual tool use. For each of these 12 lessons, a second observer independently and simultaneously observed the same lesson as that team’s primary classroom observation protocol administrator. The second observers were data collection team managers; team managers were also facilitators during data collector trainings. Each of the 12 pairs of scores were compared to determine the inter-rater reliability for each pair. Exhibit 52 presents the inter-rater reliability scores for the 12 pairs. Grey shading signifies an inter-rater reliability score of 85 percent or above, blue indicates 80-85 percent, and orange indicates below 80 percent. EXHIBIT 52: INTER-RATER RELIABILITY (IRR) SCORES FOR THE CLASSROOM OBSERVATION PROTOCOL SAMPLE8 IRR by domain IRR by criteria Pair 1 0.923 0.843 Pair 2 0.806 0.766 Pair 3 0.984 0.85 Pair 4 0.787 0.895 Pair 5 0.927 0.933 Pair 6 0.86 0.766 Pair 7 0.996 0.989 Pair 8 0.865 0.943 Pair 9 0.977 0.989 Pair 10 0.936 0.849 Pair 11 0.999 0.998 Pair 12 0.468 0.643 The inter-rater reliability scores show that administrators generally remained consistent when in “real-world” data collection settings. For classroom observations, during actual data 8 Intra-class correlation coefficient: two-way mixed, absolute agreement, 95 percent confidence interval, test value=0, average measure for the pair. Time to Learn Midline Impact Evaluation 63 collection a score of above 80 percent is considered desirable. When scores were compared by domain, all but two scores were above this threshold and the majority of scores were above 90 percent, which is a very high threshold for a new tool such as the classroom observation protocol. The two scores that are below the 80 percent threshold (pairs 4 and 12) are for a different pair of observers; both of these two pairs had inter-rater reliability scoring conducted more than once (pairs 10 and 11 are for the same main observer as pair 12, and pairs 1 to 3 and 5 are all for the same main observer as pair 4), which indicates that the low scores are likely anomalies. Seven of the 12 inter-rater reliability scores decline when inter-rater reliability is calculated by criteria instead of domain. This finding confirms the planned approach of analyzing classroom observation protocol data by domain, demonstrating that it is critical to avoid placing too much importance on the score of any single criterion. It is noteworthy, however, that even when calculated by criteria, 9 out of the 12 inter-rater reliability scores remained above the 80 percent threshold. The results indicate that data collectors were acceptably reliable. Classroom Observation Protocol Internal Reliability Exhibit 53 presents Cronbach’s alphas for the classroom observation protocol. As a new tool, validation, including checking for internal reliability, is an important step in assessing the strength of the classroom observation protocol. The values at right show that domains 2, 3, and 4 (shaded in orange) each present reasonably reliable measures of their respective domains for a new tool, as measured by Cronbach’s alpha.9 Domains 1, 5, 6, and 7 perform less well, indicating that the criteria for those domains may not be measuring the same construct. The low alphas may in part be explained by the small number of items in each domain, and further indicate that the constructs the classroom observation protocol measures are complex and multi￾faceted. For a new tool, the protocol performs well, but further testing and refinement are necessary in order to improve its reliability. 9 Lance, C. et al. 2006. The sources of four commonly reported cutoff criteria: what did they really say? Organizational Research Methods. Volume 9 number 2: 202-220. Alpha N of items Single scale, 23 criteria 0.522 23 Domain 1. Orientation to Print 0.044 4 Domain 2. Letter-Sound Knowledge 0.775 3 Domain 3. Word Decoding 0.747 4 Domain 4. Oral Passage Reading 0.719 2 Domain 5. Reading Comprehension 0.516 3 Domain 6. Listening Comprehension 0.278 3 Domain 7. Writing 0.176 4 EXHIBIT 53: INTERNAL RELIABILITY OF CLASSROOM OBSERVATION PROTOCOL Time to Learn Midline Impact Evaluation 64 Limitations Design Limitations Independently pooled cross-sectional designs depend on data from more than one period of time. Throughout this report, findings reporting change or comparing 2014 to 2012 draw on baseline datasets. For a detailed review of baseline data limitations, see the baseline study report. Cross-sectional designs are adept at capturing changes in indicators over time, but cannot statistically attribute changes to a specified intervention. Through a combination of intervention exposure indices and the broader 5-year evaluation mixed-methods evaluation approach, TTL evaluations seek to build evidence for attribution, but this should not be conflated with statistical attribution. Engagement of MESVTEE officials, who are not seasoned evaluators, builds their capacity over the long term, but could affect data quality (integrity and precision) in the short term. This challenge was noted at baseline. As detailed above, inter-rater reliability results show that this challenge has been effectively mitigated in the midline. As a result of the undersampling in Muchinga Province described above, that province’s sample contains fewer than the target of 256 learners and provides slightly less than the anticipated statistical power. If confirmed, the MESVTEE practice of upgrading high performing community schools to government schools could depress the gains resulting from TTL interventions that are seen in EGRA data, as the highest performers are reclassified as government schools. Tool Limitations EGRA: No baseline data exist for task 4 orientation to print and task 6 English language listening comprehension. Task 1 language of instruction listening comprehension passages were altered for 2014, but no equating was conducted so the results cannot be accurately compared to 2014. The 2014 EGRA used was not designed to be comparable across the three languages of instruction and testing. The languages themselves differ in difficulty, which affects the speed of reading ability acquisition. Because languages differ in inherent complexity and EGRA passages are language-specific, EGRA results cannot be compared across the three languages of instruction in which TTL works. The MESVTEE Performance Level Descriptors, however, are the same for all languages, irrespective of these differences, and the MESVTEE has not yet issued language-specific guidance for assessments. EGRA passages are designed to be grade-appropriate, but because the MESVTEE Performance Level Descriptors lack a common definition of grade appropriate text, there is room for differences in interpretation in the meaning of grade level. Furthermore, the 2014 EGRA has been equated with the 2012 EGRA, which predated the Performance Level Descriptors. This increases the possibility for differences in defining what is truly grade-appropriate. Because the standards are the same for all seven languages of instruction but languages inherently differ in difficulty, there is also the risk that the standards may be inappropriately high for more challenging languages. Classroom Observation Protocol: The replacement of the baseline classroom observation tool limited comparison of teacher practice between baseline and midline, making calculation of change in teaching practice impossible. The benefit of replacing the observation tool is that TTL and stakeholders will gain a more specific insight into the current classroom instructional practice and uptake of TTL methods by community school teachers. The new tool corresponds more closely with EGRA test items and offers increased explanatory power in understanding variation in learner reading outcomes. Time to Learn Midline Impact Evaluation 65 In 27 cases (26 percent of the sample), the teacher observed was the same as the community school head teacher questionnaire respondent. For these cases, the post-observation teacher questionnaire was omitted (see Section 3.2, community school head teacher questionnaire). Consequently, teacher demographic and attitude data are missing for these 27 cases. Community School Head Teacher Questionnaire: Recall issues are especially likely for questions related to training and it is possible that head teachers do not remember every training teachers at the school attended, which might result in underreporting. Schools may have a “vested interest” in underreporting MESVTEE support received with the hope of getting more support. MESVTEE Self-administered Survey Questionnaire: Responses related to grants were attributed to the MEVSTEE if they were coming from any government agency, including Constituency Development Fund grants. The reason for this is that schools might not know or recall the actual donor of the grant and MESVTEE district offices often support schools in getting Constituency Development Fund grants, which originate through the office of the local Parliamentarian. Because this questionnaire was self-administered, there is a risk that responses were based on recall. Additionally, the MESVTEE may have a “vested interest” in reporting high levels of support. As noted in Section 3.2, the achieved response rate for this tool provides information about 79 percent of the 102 schools sampled in the midline. It is possible that divergence between the MESVTEE and head teacher-reported levels of MESVTEE support to community schools is partially a result of this response rate. Time to Learn Midline Impact Evaluation 66 ANNEX 3. EVALUATION QUESTIONS BY DATA SOURCE AND INDICATOR Key Evaluation Questions Sub-questions Data Source Indicators 1. To what extent have TTL interventions improved literacy achievement in community schools? (TTL IR 2) a. To what extent have TTL interventions increased reading skills? b. What proportion of learners in TTL-supported community schools can read and understand the meaning of grade-level text after 2 years of primary schooling? EGRA Number of male and female learners in grade 2 who exhibit reading skills gains on EGRA tasks 3-5 EGRA tasks 5a, 5b 2. How has MESVTEE community school engagement changed as a result of TTL interventions? (TTL IR 1) a. To what extent are community schools receiving more support from the MESVTEE? b. To what extent has the MESVTEE improved its monitoring of community school? Community School Head Teacher Questionnaire Forms of support:10 1. Infrastructure 2. Grants11 3. Teaching and Learning Materials 4. Free basic materials (chalk, notebooks, pens/pencils, etc.) 5. Continuing Professional Development 6. Government-deployed teachers 7. Monitoring 3. To what extent are teachers implementing TTL￾supported literacy teaching methods? (TTL IR 2) Classroom Observation Protocol 10 Operational Guideline for Community Schools, Republic of Zambia Ministry of Education, Science, Vocational Training and Early Education, (2007). Indicators 1-6 correspond to evaluation question 2.a, and number 7 informs evaluation question 2.b. 11 Grants can be used for 1) Infrastructure maintenance; 2) Teaching/learning materials; 3) OVC support; 4) School health and nutrition. The decision is to be made by the school in the request; the grants are made through the DEBS office. Evaluation Report on the Implementation of Operational Guidelines for Community Schools, Time to Learn (2013). Time to Learn Midline Impact Evaluation 67 ANNEX 4. DOCUMENTS REVIEWED The following documents were reviewed by the evaluation team as part of this midline evaluation. TTL Teacher Training Materials Continuing Professional Development for Community School Teachers (April/May 2014) Education Leadership and Management Training Manual for Community School Head Teachers (August 2013) Monthly Training Reading (2013) Monthly Training Writing (2013) Monthly Training Alphabet Sounds (2014) Quickstart Training Materials (2013): x Circles handwriting practice x Finger plays and chants x Handwriting in the Southern Australian Curriculum Appendix 2 x Handwriting positioning graphic x Things to Keep in Mind: Left-handed and right-handed writers x Manuscript writing for kids x Pre-reading and pre-writing activities x Slash handwriting practice x Vertical line handwriting practice Quickstart Training Presentations (2013): x Key Competencies Comprehension x Key Competencies Fluency x Key Competencies Phonemic Awareness x Key Competencies Phonics x Key Competencies Vocabulary and Spelling x Key Competencies Read Aloud Teachers’ Guide on School-based Assessment (2014) Time to Learn Midline Impact Evaluation 68 Other TTL Documents Midline Impact Evaluation Implementation Plan (August 2014) TTL Evaluation Plan – Revised (December 2012) TTL Baseline Study (2013) TTL Project Performance Review Monitoring Meeting Presentation (December 2014) TTL Project Year 2 Performance Review Report (December 2013) External Documents Beatty, A., and L. Pritchett. 2012. From Schooling Goals to Learning Goals: How Fast Can Student Learning Improve? CGD Policy Paper 012. Washington D.C.: Center for Global Development. Available at http://www.cgdev.org/content/publications/detail/1426531. EdData II Report on the Equating Pilot Study (n.d.) Lance, C. et al. 2006. The sources of four commonly reported cutoff criteria: what did they really say? Organizational Research Methods. Volume 9 number 2: 202-220. MESVTEE Education Policy (1996) MESVTEE Education Act (2011) MESVTEE Educational Statistical Bulletin (2013) MESVTEE Operational Guidelines for Community Schools (2007) MESVTEE Operational Guidelines for Community Schools (2014) MESVTEE Reading Performance Level Descriptors for Grades 1– 4 (2014) Musonda, B., and A. Kaba. n.d. The SACMEQ-III Project in Zambia: A Study of the Conditions of Schooling and the Quality of Education. Southern and Eastern African Consortium for Monitoring Educational Quality testing consortium USAID Education Strategy (2011). Available at http://pdf.usaid.gov/pdf_docs/PDACQ946.pdf. Time to Learn Midline Impact Evaluation 69 ANNEX 5. DATA COLLECTION TOOLS The EGRA protocols, head teacher interview, classroom observation protocol, and self-administered MESVTEE survey questionnaire used for data collection are provided under separate cover in conjunction with this report. Please refer to the supplemental document titled “Time to Learn Midline Impact Evaluation: Annex 5” for the complete set of data collection tools. May 2015 This publication was produced for the USAID Time to Learn Project. It was prepared by Zachariah Falconer-Stout, Lyn Messner, and Vera Wedekind, Time to Learn/EnCompass, LLC. TIME TO LEARN MIDLINE IMPACT EVALUATION: ANNEX 5 Time to Learn Project Time to Learn project (TTL) is funded by the United States Agency for International Development (USAID) in Zambia under contract number AID 611-C-12-00002, funded March 1, 2012. Time to Learn is implemented by Education Development Center, Inc. (EDC), in collaboration with Camfed (the Campaign for Female Education), EnCompass LLC, and FAWEZA (the Forum for African Women Educationalists in Zambia). The project assists the Ministry of Education, Science, Vocational Training and Early Education (MESVTEE) through a 5-year national program to provide an equitable standard of education service for vulnerable learners, improve reading skills, and implement practical strategies to strengthen school quality and promote community engagement in community schools. Cover photo by: Zachariah Falconer-Stout, EnCompass LLC TIME TO LEARN MIDLINE IMPACT EVALUATION: ANNEX 5 May 29, 2015 Project Number AID-611-C-12-00002 DISCLAIMER This evaluation is made possible by the generous support of the American people through the United States Agency for International Development (USAID). The contents are the responsibility of Education Development Center (EDC) and do not necessarily reflect the views of USAID or the United States Government. CONTENTS EARLY GRADE READING ASSESSMENT (EGRA)............................................................................................................ 5 THE COMMUNITY SCHOOL HEAD TEACHER QUESTIONNAIRE................................................................................... 50 CLASSROOM OBSERVATION TOOL ........................................................................................................................... 69 DISTRICT EDUCATION BOARD SECRETARY SELF- ADMINISTERED SURVEY QUESTIONNAIRE..................................... 94 Time to Learn Midline Impact Evaluation: Annex 5 4 ANNEX 5. DATA COLLECTION TOOLS The interview and focus group discussion guides used for data collection are provided here. All respondents were administered a confidentiality and informed consent statement. EARLY GRADE READING ASSESSMENT (EGRA) CHINYANJA Time to Learn Midline Evaluation Early Grade Reading Assessment (EGRA) – CiNyanja Administrator Instructions and Protocol – 2014 General instructions Establish a playful and relaxed rapport with the child through a short conversation (see example topics below). The child should perceive the assessment almost as a game to be enjoyed rather than a test. Use this time to identify in what language the child is most comfortable communicating. Read aloud slowly and clearly ONLY the sections in boxes. Mwauka bwanji. Dzina langa ndine…………..ndipo ndimakhala ku………..Ndingakonde kukuuza za moyo wanga. Good morning. My name is ____ and I live in _____. I’d like to tell you a little bit about myself. [Number and ages of children; favorite sport, radio or television program, etc.] 1. Kodi umakonda kucita ciani ngati siuli mu sukulu? What do you like to do when you are not in school? [Wait for response; if learner is reluctant, ask question 2, but if they seem comfortable continue to verbal consent]. 2. Kodi ndi masewera otani amene umakonda kusewera? What games do you like to play? Database ID: ________ Time to Learn Midline Impact Evaluation: Annex 5 5 Verbal Assent: Read the text in the box clearly to the child. • Ndifuna kukuuza cifukwa cake ndabwera kuno lero. Ndimagwira nchito mu unduna wa maphunziro mu Zambia ndipo tikufuna kumvetsetsa mmene ana amaphunzirira kuwerenga ndi kuchita masamu. Iwe wasankhidwa mwamwai. Let me tell you why I am here today. I work with the Ministry of Education and we are trying to understand how children learn to read. You were picked by chance. • Ife tifuna thandizo lako pa nkhaniyi. Koma iwe suyenera kutengamo mbali ngati sufuna. We would like your help in this, but you do not have to take part if you do not want to. • Ife tizachita masewero a kuwerenga ndi kucita masamu. Ine ndizakufunsa kuwerenga malembo, mau ndi ka nkhani kocepa mokweza mau. Ndizakufunsanso kuzindikira manambala, kuwerengera ndi kuyankha mafunso ocepa. We are going to play a reading game. I am going to ask you to read letters, words, and a short story out loud. • Mwakugwiritsa nchito koloko ili, ndizaona nthawi imene utenga kuti utsiriza nchito zoperekedwa. Using this stopwatch/device/gadget, I will see how long it takes you to read. • Zimene tizachita pano si mayeso ndipo sizidzakhudza magiredi ako pasukulu lino. This is NOT a test and it will not affect your grade at school. • Ndizakufunsanso mafunso ena onena za banja lako monga kuti ndi cilankhulo citi cimene banja lanu limagwiritsa nchito ndipo ndi zimene banja lanu liri nazo. I will also ask you other questions about your family, like what language your family uses at home and some of the things your family has. • Sindizalemba dzina lako ndipo palibe aliyense adzadziwa za mayankho ako. I will NOT write down your name so no one will know these are your answers. • Kaciwirinso, sungatengemo mbali ngati sufuna kutero. Tikayamba kufunsa mafunso, ngati sufuna kuyankha funso ungakhale cete, zilibwino cabe. Once again, you do not have to participate if you do not wish to. Once we begin, if you would rather not answer a question, that’s all right. • Kodi uli ndi mafunso alionse? Do you have any questions? • Kodi wakonzeka kuti tiyambe? Are you ready to get started? Check box if verbal assent is obtained: *ii* YES (If verbal consent is not obtained, thank the child and move on to the next child, using this same form) Time to Learn Midline Impact Evaluation: Annex 5 6 LEARNER BACKGROUND INTERVIEW Data collector to fill in items 1-10 individually or with help from respondent, as necessary. 1. EMIS: 2. Dzina la Wotenga uthenga: Data Collector Name: 3. Tsiku: Date: 4. Dzina la Sukulu: School Name: 5. Dera: Province: 6. Cigawo: District: 7. Zoni: Zone: 8. Woyankha Mafunso: Mwamuna/Mkazi: Respondent Sex: Male/Female: 9. Zaka zakubadwa: Age: (in years) a. no response/do not know 10. Class: a. Grade 2 11. Kunyumba kwanu, kodi banja lanu lili ndi: (Linganizani zonse zoyenera): At home, does your family have: (Check all that apply) a. Rediyo Radio b. Foni yam’manja (Selefoni) Mobile Phone c. Magetsi Electricity d. Firiji Fridge e. Sitovu Yophikira Cooking Stove f. Televizyoni TV g. Kompyuta Computer h. Cimbudzi cokumba Latrine i. Njinga yamoto Motorcycle j. Njinga Bicycle k. Galimoto Car/Motor Vehicle 12. Kodi ndi cilankhulo citi cimene mumagwiritsa nchito/mumalankhula ku sukulu? (Linganizani zonse zoyenera) What language do you use/speak at school? (Check all that apply) a. Chibemba Bemba b. Chikaonde Kaonde c. Chilozi Lozi d. Chilunda Lunda e. Chiluvale Luvale f. Chinyanja Nyanja g. Chitonga Tonga h. Chilozi Silozi i. Chingerezi English j. Zina Other i. Ngati ndi zina: Ndi ziti? If other: which? ___________ Time to Learn Midline Impact Evaluation: Annex 5 7 13. Kodi ndi cilankhulo citi cimene mumagwiritsa nchito/mumalankhula kunyumba? (Linganizani zonse zoyenera) What language do you use/speak at home? (Check all that apply) a. Chibemba Bemba b. Chikaonde Kaonde c. Chilozi Lozi d. Chilunda Lunda e. Chiluvale Luvale f. Chinyanja Nyanja g. Chitonga Tonga h. Chilozi Silozi i. Chingerezi English j. Zina Other i. Ngati ndi zina: Ndi ziti? If other: which? _________ 14. Kodi mumakhala ndi ndani kunyumba kwanu? (Linganizani zonse zoyenera) Who do you live with? (Check all that apply) a. Amai ndi/kapena Atate Mother and/or Father b. Agogo (Ambuya) Grandparent c. Wacibale Wamwamuna Male Relative d. Wacibale Wamkazi Female Relative e. Wosati Wacibale Non-Relative f. Abale Anga Siblings g. Ndekha (ngati ndi g., lumphirani funso 17) Alone (if g., skip to question 17) 15. Kodi ndani amene adziwa kuwerenga kunyumba kwanu? (Linganizani zonse zoyenera) Who in your house knows how to read? (Check all that apply) a. Amai ndi/kapena Atate Mother and/or Father b. Agogo (Ambuya) Grandparent c. Wacibale Wamwamuna Male Relative d. Wacibale Wamkazi Female Relative e. Wosati Wacibale Non-Relative f. Abale Anga Siblings g. Palibe amene adziwa kuwerenga (ngati ndi g., lumphirani funso 17) Nobody knows how to read (if g., skip to question 17) 16. Kodi umayeserera kuwerenga ndi munthu wina aliyense kunyumba kwanu? Izi ziphatikizapo iweyo kuwerenga nao ndiponso iwo kuwerenga mokweza kwa iwe. Do you practice reading with anybody in your house? This includes you reading with them and them reading aloud to you. a. Inde Yes b. Iai No Time to Learn Midline Impact Evaluation: Annex 5 8 17. Kodi ndi nthawi zingati pomwe umayeserera kuwerenga ndi munthu wacikulire kunja kwa sukulu? (Linganizani zonse zoyenera) How often do you practice reading with an adult outside of school? (Check all that apply) a. Masiku onse Every day b. Kosacepera pa kamodzi pamlungu At least once a week c. Kosacepera pa kamodzi pamwezi At least once a month d. Kocepera pa kamodzi pamwezi Less than once a month e. Palibe Never 18. Kodi ndi nthawi zingati zomwe umawerenga pawekha kusukulu? (Linganizani zonse zoyenera) How often do you read on your own outside of school? (Check all that apply) a. Masiku onse Every day b. Kosacepera pa kamodzi pamlungu At least once a week c. Kosacepera pa kamodzi pamwezi At least once a month d. Kocepera pa kamodzi pamwezi Less than once a month e. Palibe Never 19. Do you have reading materials at home? a. Inde Yes b. Lai No (If no, skip to Q21) 20. Kodi uli ndi ciani cymene ungawerenge kunyumba kwanu? What do you have to read in your home? a. Manyuzipepala ndi zowerenga zina Newspapers and other periodicals b. Mabuku a acikulire Books for adults c. Mabuku a ana Children’s books d. Zinthu zowerenga za ana Other reading materials for children e. Zinthu zacipembedzo, kuphatikapo Baibulo Religious materials, including the bible f. Zinthu zina zowerenga Other reading materials i. Ngati ndi zina, ndi ziti? If other, which? _________ g. Palibe Nothing 21. Kodi ndi nthawi zingati zimene umayeserera kuwerenga kusukulu? How often do you practice reading in school? a. Masiku onse Every day b. Masiku ambiri pamlungu Several days a week c. Mlungu ulionse Every week d. Kocepera pamlungu ulionse Less than every week e. Kapena: masiku 0 mpaka 5 pamlungu 0-5 days a week Time to Learn Midline Impact Evaluation: Annex 5 9 22. Kodi aphunzitsi anu amakuwerengerani nthano mokwesa mu kalasi? Does your teacher read stories aloud to you in class? a. Inde Yes b. Iai No 23. Pambali pa mabuku anu owerenga ndi mabuku anchito, kosi pali zinthu zina zowerenga kusukulu? Besides your textbooks and notebooks, are there other things for you to read at school? a. Inde Yes b. Iai No Ngati inde, kodi zimenezo n’ciani? If yes, what are they? _________________________ 24. Kodi wakhala pa sukulu lino kucokera pamene munatsegulira sukulu caka cino? Have you been in this school since the start of the school year? a. Inde Yes b. Iai No 25. Kodi ndi masiku angati pamlungu amene umapita kusukulu? (0-5) How many days in a week do you normally attend school? (0-5) ___ a. Masiku onse Every day b. Masiku 4 4 days c. Masiku 3 3 days d. Masiku 2 2 days e. Tsiku 1 1 day f. Palibe 0 days 26. Kodi aphunzitsi anu amakupatsani homuweki? Does your teacher give you homework? a. Inde Yes b. Iai No 27. (Ngati inde), kodi pali wina aliyense amene mukhala naye angakuthandize kucita homuweki? (If yes), does anyone you live with help you do your homework? a. Inde Yes b. Iai No Time to Learn Midline Impact Evaluation: Annex 5 10 Sub-test 1a. LISTENING COMPREHENSION (2014)  X  X  Ndidzakuwerengera ka nthano/nkhani mokweza KAMODZI ndipo pambuyo pake ndidzakufunsa mafunso. Conde umvetsere mosamalitsa ndipo uyankhe mafunso mmene ungakwanitsire. Ungayankhe mafunso mcilankhulo ciriconse cimene ukonda. I am going to read you a short story aloud ONCE and then ask you some questions. Please listen carefully and answer the questions as best as you can. You can answer the questions in whichever language you prefer. Ready? Let’s begin. Remove the learner stimuli booklet from the child’s view. Do not allow the child to look at the passage or the questions. If a child says “I don’t know,” mark as incorrect.  () 1 = Correct () 0 = Incorrect () . = No response. Patsiku Lolemba, Mangani anapita kusukulu. Ananyamula mabuku ndi nyama m’cola cake. Pamene anali kuyenda, anapeza galu wamkulu panjira. Anafuna kuthawira pathengo koma anagwa pansi. Yunifomu yake inada ndipo galu anatenga nyama yake. Mangani anathawira kunyumba. Pamene anafika kunyumba, m’bale wake anamubwereka yunifomu yake. Anakondwera. Ndi tsiku liti pamene Mangani anapita kusukulu? (Pa Lolemba) 1 0 . Ananyamula ciani mu cola cake? (Mabuku ndi nyama) 1 0 . N’ciani cimene anapeza panjira? (Anapeza galu wamkulu) 1 0 . Ndi cifukwa ciani Mangani anathawa galu? (Anaopa kuti galu angamulume) 1 0 . Ndi cifukwa ciani m’bale wake anamubwereka yunifomu Mangani? (Cifukwa yunifomu yake inada). 1 0 . Wacita bwino! Tiye tipitirize patsamba lotsatira Good effort! Let’s go on to the next section. Time to Learn Midline Impact Evaluation: Annex 5 11 Sub-test 2. LETTER SOUND KNOWLEDGE  Page 1  60 seconds  Pano ndili ndi tsamba limene liri ndi malembo a alifabeti ya muchingerezi. Coonde ndiuze MVEKERO za malembo a alifabeti amene udziwa. Usanene maina ake. Koma mvekero zake. Here is a page full of letters of the Chinyanja alphabet. Please tell me the SOUNDS of as many letters of the alphabet as you can. Not their names, but their sounds. [point to the letter A] Mwacitsanzo, mvekero la lembo ili ndi /a/. For example, the sound of this letter is /a/. [point to the letter p] Tiye tiyeserere: ndiuze mvekero la lembo ili: Let’s practice: Tell me the sound of this letter.  Cabwino, mvekero la lembo ili ndi /p/ Good, the sound of this letter is /p/.  Mvekero la lembo ili ndi /p/ The sound of this letter is /p/. [point to the letter L] Tsopano tiye tiyese lembo lina. Ndiuze mvekero la lembo ili. Now let us try another one. Tell me the sound of this letter.  Cabwino, mvekero la lembo ili ndi /l/. Good, the sound of this letter is /l/.  Mvekero la lembo ili ndi /l/. The sound of this letter is /l/. [point to first letter] Ndikanena kuti “tiyambe”, uyambire apa ndi kupitiriza mopingasa tsamba ili. Lata ku lembo lirilonse ndipo ndiuze mvekero la lembo limenelo mmau okweza. Ngati wafika pa lembo limene sudziwa, pitiriza kupita ku lembo lotsatira. Ika cala cako pa lembo loyamba. Wakonzeka? Yamba. When I say “Begin,” start here and go across the page. Point to each letter and tell me the sound of that letter in a loud voice. Read as quickly and carefully as you can. If you come to a letter you do not know, go on to the next letter. Put your finger on the first letter. Ready? Begin. Start the timer when the child reads the first letter.  If a child hesitates or stops on a letter for 3 SECONDS, point to the next letter and say “Go on”.  When the timer reaches 0, say “stop.”  If the child does not provide a single correct response on the first line (10 items), say “Thank you!”, discontinue this subtask, check the box at the bottom, and go on to the next subtask.  ( / ) Mark any incorrect letters with a slash ( Ø ) Circle self-corrections if you already marked the letter as incorrect ( ] ) Mark the final letter read with a bracket Examples: A p L 1 2 3 4 5 6 7 8 9 10 Time to Learn Midline Impact Evaluation: Annex 5 12 m N A J K u I k m d (10) C d U K b w o I A L (20) G n A D a g e E I s (30) l r A T A i a f W v (40) D a A M t Y L a i N (50) t u N k z O N I e i (60) u Z i M P l i N i U (70) A p T k A M a W c B (80) a w a A N a m R h E (90) n A a U o S l n T O (100)  Time remaining on stopwatch at completion (number of SECONDS)  Exercise discontinued because the child had no correct answers in the first line Wacita bwino! Tiye tipitirize patsamba lotsatira Good effort! Let’s go on to the next section. Time to Learn Midline Impact Evaluation: Annex 5 13 Sub-test 3. NON-WORD READING  Page 2  60 seconds  Pano pali mau opangidwa mcinyanja. Ndifuna kuti uwerenge mau amene ungakwanitse kuwerenga, Uwerenge mau awa osati masipelingi. Here are some made-up words in Chinyanja. I would like you to read as many as you can. Do not spell the words, but read them. [point to the word “oli”] Mwacitsanzo, liu lopangidwa ili ndi: “oli” For example, this made-up word is: “oli”. [point to the word “koki”] Tiye tiyeserere: conde werenga liu ili. Let’s practice: Please read this word.  wacita bwino. Liu ili ndi “koki” Good, this made-up word is “koki.”  Liu lopangidwa ili ndi “koki” This made-up word is “koki.” [point to the word “cota”] Tsopano tiye tiyese liu lina: conde werenga liu ili: Now let us try another one. Please read this word.  “wacita bwino, liu lopangidwa ili ndi “cota” Good, this made-up word is “cota.”  Liu lopangidwa ili ndi “cota” This made-up word is “cota.” [point to first word] Ndikanena kuti “yamba” uyambire pano ndipo uwerenge mopingasa patsamba ili. Lata liu lirilonse ndipo uliwerenge mokweza. Uwerenge mofulumira ndi mosamala mmene ungakwanitsire. Ngati wapeza liu limene sudziwa, pita ku liu lotsatira. Ika cala cako pa liu loyamba. Wakonzeka? Yamba. When I say “Begin,” start here [point to first word] and read across the page [point]. Point to each word and read it in a loud voice. Read as quickly and carefully as you can. If you come to a word you do not know, go on to the next word. Put your finger on the first word. Ready? Begin. Start the timer when the child reads the first word.  If a child hesitates or stops on a word for 3 SECONDS, point to the next word and say “Go on”.  When the timer reaches 0, say “stop.”  If the child does not provide a single correct response on the first line (5 items), say “Thank you!”, discontinue this subtask, check the box at the bottom, and go on to the next  subtask. ( / ) Mark any incorrect words with a slash ( Ø ) Circle self-corrections if you already marked the word as incorrect ( ] ) Mark the final word read with a bracket Examples: oli koki cota Time to Learn Midline Impact Evaluation: Annex 5 14 1 2 3 4 5 kelo nipe gelu atapi mdzimu (10) rizi ninane umbe wondi ninda (20) ngalo ledesi tomo fikiraku zirama (30) mukudi yu mwane ane dzimo (40) wekusera liraku anuli ia dzimoli (50) anauna cofukwa kubu udi mtisinaka (60) amoi wera diko eka kasuci (70) komi ateta nacho lia labo (80) nthua menepa ndaako ncheto balo (90) mtingi mtanyama ndokonda mtutu ko (100)  Time remaining on stopwatch at completion (number of SECONDS)  Exercise discontinued because the child had no correct answers in the first line Wacita bwino! Tiye tipitirize patsamba lotsatira Good effort! Let’s go on to the next section. Time to Learn Midline Impact Evaluation: Annex 5 15 Sub-test 4. ORIENTATION TO PRINT  Page 3  X  Show the child a story passage in the learner stimuli packet. Read the instructions in the grey boxes below, recording the child’s response before moving to the next instruction. Materials: a passage from the learner stimuli packet Sindifuna kuti uwerenge tsopano. Pa pepela iri, ungayambire kuti kuwerenga? Ndionetse ndi cala cako. I don’t want you to read this now. On this page, where would you begin to read? Show me with your finger. 1. (Child puts finger on the top row, left￾most word) O Correct O Incorrect O No Response Tsapano ndionetse mbali imene udzawerenga motsatira. Now show me in which direction you would read next. 2. (Child moves finger from left to right) O Correct O Incorrect O No Response Ukafika kotsirizira kwa mzere, udzawerenga kuti motsatira? When you get to the end of the line, where would you read next? 3. (Child moves finger to left-most word of second line) O Correct O Incorrect O No Response Total Correct /3 Time to Learn Midline Impact Evaluation: Annex 5 16 Sub-test 5a. ORAL READING PASSAGE (2014)  60 seconds Sub-test 5b: READING COMPREHENSION Show the child the sheet in the learner stimulus booklet as you read the instructions.  If a child hesitates or stops on a letter for 3 SECONDS, say “Go on”.  If the child does not provide a single correct word on the first line of text, do not ask any comprehensio n questions. If a child says “I don’t know,” mark as incorrect. After the child is finished reading, REMOVE the passage from in front of the child. Ask the child only the questions related to the text read. A child must read all the text that corresponds with a given question. If the child does not provide a response to a question after 10 seconds, mark “no response” and continue to the next question. Do not repeat the question.  Pano pali nthano yayifupi. Ndifuna kuti uwerenge mokweza, mofulumira komanso mosamala. Ukatsiriza kuwerenge, ndizakufunsa mafunso onena za nkhani imene wawerenge. Ndikanena kuti “ yamba,” uwerenge bwino kwambiri mmene ungakwanisire. Ngati wapeza liu limene sudziwa, pita ku liu lotsatira. Ika cala cako pa liu loyamba. Wakonzeka? Yamba. Here is a short story. I want you to read it out loud quickly, but carefully. When you finish, I will ask you some questions about what you have read. When I say “Begin,” read the story as best as you can. If you come to a word you do not know, go on to the next word. Put your finger on the first word. Ready? Begin.  Tsopano ndidzakufunsa mafunso ocepa onena za nthano imene wawerenga. Yesa kuyankha mafunso mmene ungakwanisire. Ungayankhe mafunso mcilankhulo ciriconse cimene ukonda. Now I am going to ask you a few questions about the story you just read. Try to answer the questions as well as you can. You can provide your answers in whichever language you prefer.  ( / ) Mark any incorrect letters with a slash ( Ø ) Circle self-corrections if you already marked the letter as incorrect ( ] ) Mark the final letter read with a bracket  () 1 = Correct () 0 = Incorrect () . = No response. Questions [Answers] Amai anapita kumsika m’masana tsiku lina. 6 Ndani anapita kumsika? (Amai) 1 0 . Anasiya mwana ndi mkulu wake Dolika. 12 Mwana anatsala ndi ndani? (Dolika) 1 0 . Time to Learn Midline Impact Evaluation: Annex 5 17 Anzake a Dolika anabwera kudzamtenga pamodzi ndi mwanayo. Dolika ndi anzake anaphunzitsa mwana kuyimba. Anamuphunzitsa nyimbo ya alifabeti. 30 Kodi mwana anaphunzitsiwa kucita ciani? (Kuyimba) 1 0 . Atabwerako kumsika amai, anapeza mwana ali kuyimba. 37 Kodi mwana anadziwa bwanji kuyimba nyimbo ya alifabeti? Dolika ndi anzake anamphunzitsa) 1 0 . Amai anakondwera kwambiri. 40 N’cifukwa ciani amai anakondwera? (Mwana anali kuyimba) 1 0 .  Time remaining on stopwatch at completion (number of SECONDS)  Exercise discontinued: the child had no correct answers in the first line Wacita bwino! Tiye tipitirize patsamba lotsatira Good effort! Let’s go on to the next section. Time to Learn Midline Impact Evaluation: Annex 5 18 Sub-test 6. LISTENING COMPREHENSION - ENGLISH  X  X  Ndidzakuwerengera ka nthano/nkhani mokweza KAMODZI ndipo pambuyo pake ndidzakufunsa mafunso. Conde umvetsere mosamalitsa ndipo uyankhe mafunso mmene ungakwanitsire. Ungayankhe mafunso mcilankhulo ciriconse cimene ukonda. I am going to read you a short story aloud ONCE in English and then ask you some questions. Please listen carefully and answer the questions as best as you can. You can answer the questions in whichever language you prefer. Ready? Let’s begin. Remove the learner stimuli booklet from the child’s view. Do not allow the child to look at the passage or the questions. If a child says “I don’t know,” mark as incorrect.  () 1 = Correct () 0 = Incorrect () . = No response. Every morning Mukata’s father walks out from the house. “Where are you going?” Mukata asked. “I am going to the river to fish,” said his father. Mukata asked to go with him. “When you grow up I will teach you how to fish,” said Mukata’s father. The next day father was surprised to see Mukata down by the river with a hook. What does Mukata’s father do every morning? / Kodi atate ake Mukata amacita ciani m’mawa muli monse? [walks out of the house] 1 0 . Where does he go? / Kodi amapita kuti? [to the river] 1 0 . What did Mukata ask his father? / Kodi Mukata anawafunsa ciani atate ake? [he asked to go with him] 1 0 . What does Mukata’s father do? / Kodi atate Mukata amacita ciani? [he is a fisherman] 1 0 . What was Mukata doing by the river? / Kodi Mukata anali kucita ciani kumtsinje? [trying to fish] 1 0 . Time to Learn Midline Impact Evaluation: Annex 5 19 CHITONGA Time to Learn Midline Evaluation Early Grade Reading Assessment (EGRA) – CiTonga Administrator Instructions and Protocol – 2014 General instructions Establish a playful and relaxed rapport with the child through a short conversation (see example topics below). The child should perceive the assessment almost as a game to be enjoyed rather than a test. Use this time to identify in what language the child is most comfortable communicating. Read aloud slowly and clearly ONLY the sections in boxes. Wabuka buti? Izyina lyangu ndime______, ndikkala_____. Ndiyanda kulipandulula asyoonto kutegwa undizyibe. Good morning. My name is ____ and I live in _____. I’d like to tell you a little bit about myself. [Number and ages of children; favorite sport, radio or television program, etc.] 1. Ncinzi ncoyandisya kucita ciindi notali kucikolo? What do you like to do when you are not in school? [Wait for response; if learner is reluctant, ask question 2, but if they seem comfortable continue to verbal consent]. 2. Nzyisobano nzi nzyoyandisya kusobana? What games do you like to play? Verbal Assent: Read the text in the box clearly to the child. • Ndiyanda kukwaambila nceli waano sunu. Ndibeleka mumutabi wamfwulumende uulanganya lwiiyo mucisi ca Zambia alimwi tusola kuzyiba mbuli bana mbobayiya kubala akulemba nambala. Yebo nduwe wakasalwa mucoolwe. Let me tell you why I am here today. I work with the Ministry of Education and we are trying to understand how children learn to read. You were picked by chance. • Tuyanda kuti yebo mbookasalwa utugwasye. Pele kuti naa kotayandi kutola lubazu inga wakaka. We would like your help in this, but you do not have to take part if you do not want to. • Tuyakusobana cisobano cakubala mabala alimwi anambala. Ndila kwaambila kuti ubale tubala, mabala alimwi akaano kafwaafwi cakwaambisya. Alimwi ndilakwaambila kuti unditondezye nambala, uzyibeleke akujana bwiinguzi kumanambala aya. We are going to play a reading game. I am going to ask you to read letters, words, and a short story out loud. • Kwiinda mukubelesya nkoloko eeyi, ndiyakubona ciindi ncotitole kubala. Using this Database ID: ________ Time to Learn Midline Impact Evaluation: Annex 5 20 stopwatch/device/gadget, I will see how long it takes you to read. • Ooyu TAWULI musunko pe alimwi taakwe mbowukonzya kunyonganya lwiiyo lwako lwamucikolo. This is NOT a test and it will not affect your grade at school. • Ndilakubuzya iimbi mibuzyo iijatikizya mukwasyi wako mbuli mulaka ngomwaambaula mumukwasyi kung’anda yanu alimwi azyintu nzyomujisi mumukwasyi wenu. I will also ask you other questions about your family, like what language your family uses at home and some of the things your family has. • TANDIKWE kulilemba zyina lyako kutegwa kubule uuzyiba kuti nduwe wandipa bwiinguzi oobu. I will NOT write down your name so no one will know these are your answers. • Alimwi ulaangulukide kutatola lubazu kuti kotayandi. Twatalika, naa kuli mubuzyo uutikukakile kuwiingula, cili biyo kabotu inga wauleka. Once again, you do not have to participate if you do not wish to. Once we begin, if you would rather not answer a question, that’s all right. • Sena ulijisi mibuzyo yakubuzya? Do you have any questions? • Sena walibambila kuti inga twatalika? Are you ready to get started? Check box if verbal assent is obtained: *ii*YES (If verbal consent is not obtained, thank the child and move on to the next child, using this same form) Time to Learn Midline Impact Evaluation: Annex 5 21 LEARNER BACKGROUND INTERVIEW Data collector to fill in items 1-10 individually or with help from respondent, as necessary. 1. EMIS 2. Data Collector Name: 3. Date (dd/mmm/yyyy): 4. Province: 5. District: 6. Zone: 7. School Name: 8. Sena sikubuzya: Respondent Sex: a. Musankwa Male b. Musimbi Female 9. Myaka: Age (in years): a. no response/ do now know 10. Class a. Grade 2 11. Ku ŋanda, hena mukwasyi wako ulijisi: At home, does your family have: (Check all that apply) a. Sikapepele Radio b. Fooni yakumaanza Mobile phone c. Magesi Electricity d. Fuliji Fridge e. Cokufwu cakujikila Cooking Stove f. Namacaaca TV g. Kkompyuta Computer h. Cimbuzi Latrine i. Kandukunduku/Mndududu Motorcycle j. Ncinga Bicycle k. Mootokala Car/motor vehicle 12. Ino kucikolo mubelesye/mukanana musyobo nzi? What language do you use/speak at school? (Check all that apply) a. Chibemba Bemba b. Chikaonde Kaonde c. Chilozi Lozi d. Chilunda Lunda e. Chiluvale Luvale f. Chinyanja Nyanja g. Chitonga Tonga h. Chilozi Silozi i. Chikuwa English j. Azyimwi Other i. Kuti naa wavwiila kuti zyimwi? If other: Which? __________ Time to Learn Midline Impact Evaluation: Annex 5 22 13. Musyobo nzi ngomubelesye/kukanana? What language do you use/speak at home? (Check all that apply) a. Chibemba Bemba b. Chikaonde Kaonde c. Chilozi Lozi d. Chilunda Lunda e. Chiluvale Luvale f. Chinyanja Nyanja g. Chitonga Tonga h. Chilozi Silozi i. Chikuwa English j. Azyimwi Other i. Kuti naa wavwiila kuti zyimwi? If other: Which? __________ 14. Ino ukkala aba ni? Who do you live with? (Check all that apply) a. Bamaama/Bataata Mother and/or Father b. Bakaapa Grandparent c. Basazima baalumi Male Relative d. Basazima bakaintu Female Relative e. Muntu buyo utali musazima Non-relative f. Bana bokwesu Siblings g. Endikke Alone (if g, skip to Q17) 15. Nguni muŋanda yanu ucizyi kubala? Who in your house knows how to read? (Check all that apply) a. Bamaama/Bataata Mother and/or Father b. Bakaapa Grandparent c. Basazima baalumi Male Relative d. Basazima bakaintu Female Relative e. Muntu buyo utali musazima Non-relative f. Bana bokwesu Siblings g. Taku ucizyi kubala Nobody knows how to read (if g., skip to question 17) 16. Hena kuli ngomuyiisyanya kubala muŋanda yanu? Eeci caamba iwe kubala ambabo abo kubala cakupozya kuli ndiwe. Do you practice reading with anybody in your house? This includes you reading with them and them reading aloud to you. a. Inzya Yes b. Peepe No 17. Ino zyiindi zyongaye zyomuyiisyanya amuntu mupati citali ciindi cacikolo? How often do you practice reading with an adult outside of school? (Check all that apply) f. Buzuba abuzuba Every day g. Ciindi comwe amvwiki At least once a week h. Ciindi comwe amwezi At least once a month i. Kutaindilila ciindi comwe amwezi Less than once a month j. Taakwe Never Time to Learn Midline Impact Evaluation: Annex 5 23 18. Ino zyiindi zyongaye zyobala olikke citali ciindi cacikolo? How often do you read on your own outside of school? (Check all that apply) a. Buzuba abuzuba Every day b. Ciindi comwe amvwiki At least once a week c. Ciindi comwe amwezi At least once a month d. Kutaindilila ciindi comwe amwezi Less than once a month e. Taakwe Never 19. Do you have reading materials at home? a. Inzya Yes b. Peepe No (If no, skip to Q21) 20. Ino inga ubala nzi kuŋanda kwenu? What do you have to read in your home? a. Miteende azyimwi buyo zyakubala Newspapers and other periodicals b. Mabbuku abapati Books for adults c. Mabbuku aabana Children’s books d. Zyimwi buyo zyakubala zyabana Other reading materials for children e. Zyakubala zyabukombi kubikkilizya Bbaibele Religious materials, including the Bible f. Zyakubala buyo zyimwi Other reading materials i. Kuti naa wasala zyimwi, Ino zyinzi? If other, which? __________________ g. Taakwe Nothing 21. Ino zyiindi zyongaye zyobala mucikolo? How often do you practice reading in school? a. Abuzuba Every day b. 4 days per week c. 3 days per week d. 2 days per week e. 1 day per week f. 0 days per week g. Don’t know/no response 22. Hena bayi benu balamubalila cakupozya twaano mukkilasi? Does your teacher read stories aloud to you in class? a. Inzya Yes b. Peepe No 23. Kunze kwamabbuku akubala akulembela, hena kuti zyimwi zyomukonzya kubala kucikolo? Besides your textbooks and notebooks, are there other things for you to read at school? a. Inzya Yes b. Peepe No i. Kuti naa Inzya, ino zyinzi? If yes, what are they? ______________________________ Time to Learn Midline Impact Evaluation: Annex 5 24 24. Hena wali mucikolo oomu kuzwa kumatalikilo amwaka? Have you been in this school since the start of the school year? a. Inzya Yes b. Peepe No 25. Ino mazuba ongaye acikolo ngowunka kucikolo a mvwiki? How many days a week do you normally attend school? (0-5) a. Abuzuba Every day b. Bomwe 4 days c. Obilo 3 days d. Otatwe 2 days e. One 1 day f. Taakwe 0 days 26. Hena bayi benu balamupa mulimo wakucita kuŋanda? Does your teacher give you homework? a. Inzya Yes b. Peepe No 27. (Kuti naa Inzya), hena kuli ngomukkala limwi ukugwasya kucita mulimo ooyu? (If yes), does anyone you live with help you do your homework? a. Inzya Yes b. Peepe No Time to Learn Midline Impact Evaluation: Annex 5 25 Sub-test 1. LISTENING COMPREHENSION (2014) X  X Ndilakubalila caano cifwaafwi munzila yakwaambisya CIINDI COMWE biyo, mpoona ndamana kubala, ndakubuzya mibuzyo imwi. Ndalomba uswiilisyisye kabotu ndabala, kutegwa upe bwiinguzi bubotu kumibuzyo. Ulakonzya kupa bwiinguzi kumibuzyo mumulaka ngoyanda omwini. Sena walibambila? Atutalike. I am going to read you a short story aloud ONCE and then ask you some questions. Please listen carefully and answer the questions as best as you can. You can answer the questions in whichever language you prefer. Ready? Let’s begin. Remove the learner stimuli booklet from the child’s view. Do not allow the child to look at the passage or the questions. If a child says “I don’t know,” mark as incorrect.  () 1 = Correct () 0 = Incorrect () . = No response. Banji ukkala mudolopo. Ukkala a banyina abawisi. Ulabeleka canguzu ang’anda. Ulagwasya bazyali bakwe kujika mapopwe akusanzya zyimbaya-mbaya. Bazyali bakwe balikkomene ku milimo njacita. Bazyali bakwe ba kamuulila ncinga ku mulumba. Banji uliyanda ncinga yakwe, alimwi njabelesya kuya kucikolo. Banji ukkala kuli? (Kudolopo) 1 0 . Ncinzi Banji ncabagwasya bazyali kucita? (Kujika mapopwe akusanzya zyimbaya-mbaya) 1 0 . Nkaambo nzi cobakkomene bazyali bakwe anguwe? (Nkaambo kakubeleka canguzu/Nkaambo ka milimo yakwe) 1 0 . Nkaambo nzi bazyali bakwe ncobakamuulila ncinga? (Kumulumba kumilimo yakwe) 1 0 . Ncinzi Banji ncayiyandila ncinga yakwe? (Nkaambo njabelesya kuunka ku cikolo). 1 0 . Wacita kabotu! Atuunke kucibeela citobela. Good effort! Let’s go on to the next section. Time to Learn Midline Impact Evaluation: Annex 5 26 Sub-test 2. LETTER SOUND KNOWLEDGE Page 1 60 seconds  Ndeeli peeji lijisi tubala tubelesegwa kubamba mabala aacitonga. Ndalombesya undaambile tubeela twamabala MBOTUMVWIKA mbuli mbokonzya. Here is a page full of letters of the Chitonga alphabet. Please tell me the SOUNDS of as many letters of the alphabet as you can. Not their names, but their sounds. [point to the letter T] Mukonzyanyo ngooyu, mbokamvwika kabeela kakabala aaka ngu /t/. For example, the sound of this letter is /t/. [point to the letter M] Atutalike kubala: Ndaambile mbokamvwika kabeela kakabala aaka? Let’s practice: Tell me the sound of this letter.  Cabota, mbombuboobu mbokamvwika kabeela kakabala aaka, ngu /m/. Good, the sound of this letter is /m/.  Aaka kabeela kakabala kamvwika boobu /m/. The sound of this letter is /m/. [point to the letter S] Ono atusoleke kubala kambi kabeela kakabala. Ndaambile mbokamvwika kabeela kakabala aaka. Now let us try another one. Tell me the sound of this letter.  Cabota, mbombuboobu mbokamvwika kabeela kakabala, ngu /s/. Good, the sound of this letter is /s/.  Aaka kabeela kakabala kamvwika boobu/s/. The sound of this letter is /s/. [point to first letter] Ndaamba kuti “talika”, utalike mpoona aawa a peeji kuunka kumbela. Kotondeka kabeela kakabala komwe-komwe akundaambila mbokamvwika cakwaambisya. Kofwambaanisya kubala alimwi mukabotu￾kabotu mbuli mbokonzya. Kuti naa wajana kabeela kakabala nkotazyi, kasotoke wiinke kukabeela kakabala katobela. Kobikka kanwe kako akabeela katanguuna kakabala. Sena walibambila? Talika. When I say “Begin,” start here and go across the page. Point to each letter and tell me the sound of that letter in a loud voice. Read as quickly and carefully as you can. If you come to a letter you do not know, go on to the next letter. Put your finger on the first letter. Ready? Begin. Start the timer when the child reads the first letter. If a child hesitates or stops on a letter for 3 SECONDS, point to the next letter and say “Go on”. When the timer reaches 0, say “stop.” If the child does not provide a single correct response on the first line (10 items), say “Thank you!”, discontinue this subtask, check the box at the bottom, and go on to the next subtask.  ( / ) Mark any incorrect letters with a slash ( Ø ) Circle self-corrections if you already marked the letter as incorrect ( ] ) Mark the final letter read with a bracket Examples: t m s 1 2 3 4 5 6 7 8 9 10 T u V A m c A y s E (10) Time to Learn Midline Impact Evaluation: Annex 5 27 f b a J A I b u B N (20) O u i O l a k S e L (30) i o p E k D U e A W (40) U m ŋ B t b a l k S (50) z I n A w h I N m a (60) U Y U N u M Z K w A (70) A g I B n c M E N W (80) t H y N i K o a L c (90) L G Y O E K d i K B (100)  Time remaining on stopwatch at completion (number of SECONDS)  Exercise discontinued because the child had no correct answers in the first line Wacita kabotu! Atuunke kucibeela citobela. Good effort! Let’s go on to the next section. Time to Learn Midline Impact Evaluation: Annex 5 28 Sub-test 3. NON-WORD READING Page 2 60 seconds  Ngaaya mabala aabambidwe mucitonga. Ndiyanda kuti ubale mabala manji mbuli mbokonzya. Utaambi tubeela twamabala pele ubale mabala oonse mbuli mbwaabede. Here are some made-up words in Chitonga. I would like you to read as many as you can. Do not spell the words, but read them. [point to the word “tico”] Mukonzyanyo ngooyu, eeli bbala lyapangwa ngu: “tico”. For example, this made-up word is: “tico”. [point to the word “bino”] Atutalike kubala: Akaka kobala bbala eeli. Let’s practice: Please read this word.  Cabota. Eeli bbala lyabambwa ngu “bino.” Good, this made-up word is “bino.”  Eeli bbala lyabambwa ngu “bino.” This made-up word is “bino.” [point to the word “maba”] Lino atusoleke kubala limbi bbala: Ndalomba ubale bbala eeli. Now let us try another one. Please read this word.  Cabota, eeli bbala lyabambwa ngu “maba.” Good, this made-up word is “maba.”  Eeli bbala lyabambwa ngu “maba.” This made-up word is “maba.” [point to first word] Ndaamba kuti “talika”, utalikile waawa, ubale peeji yoonse. Kotondeka ibbala lyomwe- lyomwe, akulibala cakwaambisya. Ubale cakufwambaana alimwi mukabotu-kabotu mbuli mbokonzya. Kuti naa wajana ibbala ndyotakozyi kubala, utaleki, ubale ibbala litobela. Kobikka kanwe kako abbala lyakutaanguna.Sena walibambila? Talika. When I say “Begin,” start here [point to first word] and read across the page [point]. Point to each word and read it in a loud voice. Read as quickly and carefully as you can. If you come to a word you do not know, go on to the next word. Put your finger on the first word. Ready? Begin. Start the timer when the child reads the first word. If a child hesitates or stops on a word for 3 SECONDS, point to the next word and say “Go on”. When the timer reaches 0, say “stop.” If the child does not provide a single correct response on the first line (5 items), say “Thank you!”, discontinue this subtask, check the box at the bottom, and go on to the next subtask.  ( / ) Mark any incorrect words with a slash ( Ø ) Circle self-corrections if you already marked the word as incorrect ( ] ) Mark the final word read with a bracket Examples: tico bino maba Time to Learn Midline Impact Evaluation: Annex 5 29 1 2 3 4 5 ndewa kila asi juna kulusyika (10) bwede lanza kulu aambo bbeki (20) musumbi mbaasi nkati nsodo monka (30) fakulya bulona kisya ukwese bwazu (40) anji fwuza nusikila gena ndongu (50) mungila meya mbanto syuubwa wupwe (60) puma mudulu fulilo waga anteza (70) jikuwa mevwulu maku milebi misyubo (80) kulenwa paambe taama mbilule emba (90) duwe weta myami masufo zanji (100)  Time remaining on stopwatch at completion (number of SECONDS)  Exercise discontinued because the child had no correct answers in the first line Wacita kabotu! Atuunke kucibeela citobela. Good effort! Let’s go on to the next section. Time to Learn Midline Impact Evaluation: Annex 5 30 Sub-test 4. ORIENTATION TO PRINT  3  X  Show the child a story passage in the learner stimuli packet. Read the instructions in the grey boxes below, recording the child’s response before moving to the next instruction. Materials: a passage from the learner stimuli packet [Tandyandi kuti eeci ucibale ono pe. Apeeji eeli, ino mpaali mpoyelede kutalikila kubala? Konditondezya amunwe wako] I don’t want you to read this now. On this page, where would you begin to read? Show me with your finger. 1. (Child puts finger on the top row, left￾most word) O Correct O Incorrect O No Response Ono konditondezya lubazu ndoyelede kutobela kkuccilizya kubala. Now show me in which direction you would read next. 2. (Child moves finger from left to right) O Correct O Incorrect O No Response Kuti naa wasika kumamanino aamulaini, mpaali mpoyelede kuccililizya kubala? When you get to the end of the line, where would you read next? 3. (Child moves finger to left-most word of second line) O Correct O Incorrect O No Response Total Correct /3 Time to Learn Midline Impact Evaluation: Annex 5 31 Sub-test 5a. ORAL READING PASSAGE (2014) 60 seconds Sub-test 5b: READING COMPREHENSION Show the child the sheet in the learner stimulus booklet as you read the instructions.  If a child hesitates or stops on a letter for 3 SECONDS, say “Go on”. If the child does not provide a single correct word on the first line of text, do not ask any comprehensio n questions. If a child says “I don’t know,” mark as incorrect. After the child is finished reading, REMOVE the passage from in front of the child. Ask the child only the questions related to the text read. A child must read all the text that corresponds with a given question. If the child does not provide a response to a question after 10 seconds, mark “no response” and continue to the next question. Do not repeat the question. Nceeci caano cifwaafwi nceyanda kuti ubale cakwaambisya alimwi mukabotu-kabotu cakufwaambaana. Mbotimanine buyo, ndilakubuzya mibuzyo imwi iizwa mucaano ncicona eeci ncoobala. Ndaamba kuti “talika”, ucibale kabotu caano mbuli mbokonzya.Naa kuti ujane ibbala ndyotakonzyi kubala, utaimi pele uunke kumbele akubala bbala litobela ndokonzya. Kobikka kanwe kako abbala lyakutaanguna. Sena walibambila? Talika. Here is a short story. I want you to read it out loud quickly, but carefully. When you finish, I will ask you some questions about what you have read. When I say “Begin,” read the story as best as you can. If you come to a word you do not know, go on to the next word. Put your finger on the first word. Ready? Begin. Lino ndiyanda kukubuzya mibuzyo misyoonto iizwa mucaano ncoobala. Soleka kupa bwiinguzi bubotu mbuli mbokonzya. Ulakonzya kupa bwiinguzi kumibuzyo mumulaka ngoyanda omwini. Now I am going to ask you a few questions about the story you just read. Try to answer the questions as well as you can. You can provide your answers in whichever language you prefer.  ( / ) Mark any incorrect words with a slash ( Ø ) Circle self-corrections if you already marked the letter as incorrect ( ] ) Mark the final letter read with a bracket  () 1 = Correct () 0 = Incorrect () . = No response. Questions [Answers] Ciindi coonse naluntaambwe wakali kunjila mung’anda. 6 Ncinzi cakali kunjila munganda ciindi coonse? (Naluntaambwe) 1 0 . Bumwi buzuba banyina Choolwe bakali kujika. Naluntaambwe wakabainda akuulu. Baama 17 Ncinzi ncobakali kucita baama n bakabona naluntaambwe? (Bali 1 0 . Time to Learn Midline Impact Evaluation: Annex 5 bakayoowa. kujika) Baama bakamwiita Choolwe. Choolwe wakasinka cipulo cakali kubwaanda aakali kunjilila naluntaambwe. 28 Ino Choolwe waka cita nzi? (Wakasinka cipulo) 1 0 . Buzuba butobela, naluntaambwe wakasoleka kuyanda kunjila pele wakajana kwiina cipulo. 38 Hena naluntaambwe wakanjila mung’anda alimwi? (Pepe) 1 0 . Eeci cakabakkomanisya kapati baama. 43 Ncinzi cakapa kuti baama bakkomane? (Nkaambo naluntaambwe tacinjili munganda.) 1 0 .  Time remaining on stopwatch at completion (number of SECONDS)  Exercise discontinued: the child had no correct answers in the first line Wacita kabotu! Atuunke kucibeela citobela. Good effort! Let’s go on to the next section. Time to Learn Midline Impact Evaluation: Annex 5 Sub-test 6. LISTENING COMPREHENSION - ENGLISH X  X Ndilakubalila caano cifwaafwi, ciindi comwe mucikuwa, mpoonya ndamana kubala, ndakubuzya mibuzyo imwi. Ndakulomba uswiilisyisye kabotu-kabotu, kutegwa upe bwiinguzi bubotu kumibuzyo mbuli mbocikonzya. Ulakonzya kupa bwiinguzi kumibuzyo mumulaka ngoyanda omwini. Sena walibambila? Atutalike. I am going to read you a short story aloud ONCE in English and then ask you some questions. Please listen carefully and answer the questions as best as you can. You can answer the questions in whichever language you prefer. Ready? Let’s begin. Remove the learner stimuli booklet from the child’s view. Do not allow the child to look at the passage or the questions. If a child says “I don’t know,” mark as incorrect.  () 1 = Correct () 0 = Incorrect () . = No response. Every morning Mukata’s father walks out from the house. “Where are you going?” Mukata asked. “I am going to the river to fish,” said his father. Mukata asked to go with him. “When you grow up I will teach you how to fish,” said Mukata’s father. The next day father was surprised to see Mukata down by the river with a hook. What does Mukata’s father do every morning? / Ino usyi Mukata ucitaanzi lyoonse mafwumina? [walks out of the house] 1 0 . Where does he go? / Ino nkukuli nkwaunka? [to the river] 1 0 . What did Mukata ask his father? / Ino Mukata wakamulomba nzi usyi? [he asked to go with him] 1 0 . What does Mukata’s father do? / Ino usyi Mukata ucita mulimo nzi? [he is a fisherman] 1 0 . What was Mukata doing by the river? / Ino Mukata ncinzi ncaakali kucita kumulonga? [trying to fish] 1 0 . Time to Learn Midline Impact Evaluation: Annex 5 34 ICIBEMBA Time to Learn Midline Evaluation Early Grade Reading Assessment (EGRA) – iCiBemba Administrator Instructions and Protocol – 2014 General instructions Establish a playful and relaxed rapport with the child through a short conversation (see example topics below). The child should perceive the assessment almost as a game to be enjoyed rather than a test. Use this time to identify in what language the child is most comfortable communicating. Read aloud slowly and clearly ONLY the sections in boxes. Uli shani. Ishina lyandi nine______njikala _____. Nomba ndefwaya ukuilondololako panoono. Good morning. My name is ____ and I live in _____. I’d like to tell you a little bit about myself. [Number and ages of children; favorite sport, radio or television program, etc.] 1. Finshi watemwa ukucita ilyo ushili ku sukulu? What do you like to do when you are not in school? [Wait for response; if learner is reluctant, ask question 2, but if they seem comfortable continue to verbal consent]. 2. Fyangalonshi watemwa ukwangala ilyo uli pa mushi? What games do you like to play? Time to Learn Midline Impact Evaluation: Annex 5 35 Verbal Assent: Read the text in the box clearly to the child. • Leka nomba nkwebe ico njishile kuno leelo. Ine momba ku ciputulwa ca masambililo muno calo umo tuleyesha ukwishiba ifyo abasambi basambilila ukubelenga na ukupenda. Iwe naukwata ishuko lyaku kusala pa kusendamo ulubali. Let me tell you why I am here today. I work with the Ministry of Education and we are trying to understand how children learn to read. You were picked by chance. • Tulefwaya utwafweko mulifi. Nomba ngataulefwaya ukuti usendemo ulubali, tatulekupatikisha kuti wakaana nga ulefwaya. We would like your help in this, but you do not have to take part if you do not want to. • Nomba twalaangala ubwangalo bwa kubelenga. Nalakwipushako ukubelenga ifilembo, amashiwi na kalyashi kanoono mu kwikatisha ishiwi. We are going to play a reading game. I am going to ask you to read letters, words, and a short story out loud. • Ukubomfya inkoloko iyi, twalamona inshita calatusendela ukuti upwishe uyu mulimo. Using this stopwatch/device/gadget, I will see how long it takes you to read. • Aya temashindano iyoo, kabili tayakapilibule imibombele yoobe ku sukulu. This is NOT a test and it will not affect your grade at school. • Kabili nalakwipushako ameepusho yambi ayapa lupwa loobe ifili nga ululimi ulupwa loobe lulanda pa ng’anda nafimbi ifyo ulupwa lwakwata pang’anda. I will also ask you other questions about your family, like what language your family uses at home and some of the things your family has. • Nshalembe ishina lyobe, kabili takwabe uukeshiba amasuko yoobe. I will NOT write down your name so no one will know these are your answers. • Nakabili, tapabe uwalakupatikisha ukusendamo ulubali ngataulefwaya. Ngatwatampa, nga ulefwaya teti wasuke ilipusho nangu limo ninshi cilifye bwino. Once again, you do not have to participate if you do not wish to. Once we begin, if you would rather not answer a question, that’s all right. • Naukwata ameepusho? Do you have any questions? • Nauipekanya ukuti tutampeko? Are you ready to get started? Check box if verbal assent is obtained: *ii* YES (If verbal consent is not obtained, thank the child and move on to the next child, using this same form) Time to Learn Midline Impact Evaluation: Annex 5 36 LEARNER BACKGROUND INTERVIEW Data collector to fill in items 1-10 individually or with help from respondent, as necessary. 1. EMIS: 2. Ishina lyabalececta: Data Collector Name: 3. Ubushiku: Date: 4. Ishina lye sukulu: School Name: 5. Polofinshi: Province: 6. Disitilikit: District: 7. Zone: Zone: 8. Abalececetwa: Baume/Banakashi: Respondent Sex: Male/Female: 9. Imyaka yakufyala : a. no response/ do not know 10. Class a. Grade 2 11. Bushe ulupwa pa ng'anda lwalikwata ifi: (Moneni ifilingile): At home, does your family have: (Check all that apply) a. Icilimba Radio b. Kamusange Mobile phone c. Amalaiti Electricity d. Fuliji Fridge e. Shitofu Cooking Stove f. TV TV g. Kompyuta Computer h. Icimbusu Latrine i. Icitukutuku Motorcycle j. Incinga Bicycle k. Kayala (car/motoka) Car/Motor Vehicle 12. Bushe lulimi nshi mulanda pa sukulu? (Moneni fyonse ifilungeme) What language do you use/speak at school? (Check all that apply) a. IciBemba Bemba b. Kiikaonde Kaonde c. Silozi Lozi d. Lunda Lunda e. Luvale Luvale f. Cinyanja Nyanja g. Chitonga Tonga h. Silozi Silozi i. Icisungu English j. Fimbi Other i. Ngakuli fimbir: Lumbuleni?If other: which? __________ Time to Learn Midline Impact Evaluation: Annex 5 37 13. Bushe lulimi nshi mulanda pa ng'anda? (Moneni fyonse ifilungeme) What language do you use/speak at home? (Check all that apply) a. IciBemba Bemba b. Kiikaonde Kaonde c. Silozi Lozi d. Lunda Lunda e. Luvale Luvale f. Cinyanja Nyanja g. Chitonga Tonga h. Silozi Silozi i. Icisungu English j. Fimbi Other i. Ngakuli fimbir: Lumbuleni? If other: which? __________ 14. Wikala nabani pa ng'anda? (Mona amasuko ayalungeme) Who do you live with? (Check all that apply) a. Bamayo elyo na/bataata Mother and/or Father b. Bashikulu naba maama Grandparent c. Ulupwa lwaume Male Relative d. Ulupwa lwanakashi Female Relative e. Ku bantu abashili ba lupwa Non-Relative f. Siblings Siblings g. Neeka (nga apeela ubu bwasuko, kabiyeni ku 17) Alone (if g., skip to question 17) 15. Nibaani abaishiba ukubelenga pa ng'nda? (Moneni amasuko yalingile) Who in your house known how to read? (Check all that apply) a. Bamayo elyo na/bataata Mother and/or Father b. Bashikulu naba maama Grandparent c. Ulupwa lwaume Male Relative d. Ulupwa lwanakashi Female Relative e. Ku bantu abashili ba lupwa Non-Relative f. Siblings Siblings g. Tapaba uwaishiba ukubelenga(nga apeela ubu bwasuko, kabiyeni ku 17) Nobody know how to read (if g., skip to question 17) 16. Bushe ulesha ukubelenga na umuntu uuli onse pa ng'anda? Ici kubabelengela elyo nabeena balekubeengela. Do you practice Reading with anybody in your house? This includes you reading with them and them reading aloud to you. a. Ee Yes b. Iyoo No Time to Learn Midline Impact Evaluation: Annex 5 38 17. Miku inga wesha ukubelenga ukupitila mukukwafwilisha ku mukalamba kunse yakusukulu? (Moneni yonse ayalungeme) How often do you practice reading with an adult outside of school? (Check all that apply) a. Lyonse Every day b. Umuku umo mu mulungu At least once a week c. Umuku umo mu mweshi At least once a month d. Tamuba mu mweshi Less than once a month e. Takuba Never 18. Miku inga ubelenga palobe kunse ya isukulu? (Moneni yonse ayalingile) How often do you read on your own outside of school? (Check all that apply) a. Lyonse Every day b. Umuku umo mu mulungu At least once a week c. Umuku umo mu mweshi At least once a month d. Tamuba mu mweshi Less than once a month e. Takuba Never 19. Do you have reading materials at home? a. Ee b. lyoo (If no, skip to Q21) 20. Finshi ukwata ifykubelenga pa ng'anda? What do you have to read in your home? a. Inyunshipepala na ma magasini Newspapers and other periodicals b. Ifitabo fyabakalamba Books for adults c. Ifitabo fya baice Children’s books d. Ifyakubelenga fimbi ifya baice Other reading materials for children e. Ifya kwa Lesa ukubikapo na Baibele Religious materials, including the bible f. Ifyakubelenga fimbi Other reading materials i. Nga epo fili,lumbula?If other, which? __________________ g. Nothing 21. Miku inga wesha ukubelenga pa sukulu? How often do you practice reading in school? a. Cila bushiku Every day b. Inshiku ishingi cila mulungu Several days a week c. Cila mulungu Every week d. Tecila mulungu Less than every week e. 0-5 inshiku/mulungu 0-5 days a week 22. Bushe bakafundisha balamubelengela utumilumbe mu kalashi? Does your teacher read stories aloud to you in class? a. Ee Yes b. Iyoo No Time to Learn Midline Impact Evaluation: Annex 5 39 23. kufumyako ifitabo fyamu kalashi, bushe kwaliba ifintu fimbi ifyo mubomfya mukubelenga pa sukulu? Besides your textbooks and notebooks, are there other things for you to read at school? a. Ee Yes b. Iyoo No Nga watila, Ee fintunshi ifi, filumbule? If yes, what are they?_________________________________ 24. Bushe waba pali lino isukulu ukutampafye umwaka baiswile? Have you been in this school since the start of the school year? a. Ee Yes b. Iyoo No 25. Ninshiku shinga uyakusukulu muli cila mulungu (0-5) How many days a week do you normally attend school? (0-5) a. Cila bushiku Every day b. Inshiku shine (4) 4 days c. Inshiku shitatu (3) 3 days d. Inshiku shibili (2) 2 days e. Ubushiku bumo 1 day f. 0 0 days 26.Bushe bakafundisha obe balakupeela umulimo wakuyabombela ku ng'anda? Does your teacher give you homework? a. Ee Yes b. Iyoo No 27. (Ngawasumina), bushe kwaliba abo mwaikala nabo abamwafwilishako ukulemba fiyakusukulu? (if yes), does anyone you live with help you do your homework? a. Ee Yes b. Iyoo No Time to Learn Midline Impact Evaluation: Annex 5 40 Sub-test 1. LISTENING COMPREHENSION (2014)  X  X  Nalakubelengela ilyashi iliipi kabili mukwikatisha ishiwi UMUKU UMO elyo nkwipushepo ameepusho. Ufwikishe elyo wasuka amepusho nalakwipusha bwino-bwino. Kuti wayasuka ameepusho mu lulimi ulo wingafwaya ukubomfya. Nauipekanya? Katutampe. I am going to read you a short story aloud ONCE and then ask you some questions. Please listen carefully and answer the questions as best as you can. You can answer the questions in whichever language you prefer. Ready? Let’s begin. Remove the learner stimuli booklet from the child’s view. Do not allow the child to look at the passage or the questions. If a child says “I don’t know,” mark as incorrect.  () 1 = Correct () 0 = Incorrect () . = No response. Mulenga alitemwa umupila waku makasa. Alitemwa ukutamba ibumba lyakwe ngalileteya. Mulenga alatemwa ibumba lyakwe nga lyawina. Alapundisha ukukoselesha ibumba lyakwe. Mukuteya kumo ukwaliko, aleefwaya ukupona pa mupando ku mulandu wakusansamuka. Banyina balimukenye ukulapundisha pantu kuti akalifya abeena mupalamano. Finshi Mulenga atemwa? (umupila waku makasa) 1 0 . Libumbanshi Mulenga atemwa ukutamba? (ibumba lyakwe ngalileleya) 1 0 . Ni munshilanshi Mulenga akoseleseshamo ibumba lyakwe? (alapunda saana) 1 0 . Mulandunshi aalefwaila ukupona pa mupando? (Aaliisansamwike) 1 0 . Mulandunshi banyina Mulenga bamuleseshe ukulapundisha? (Kuti akalifya abeena mupalamano) 1 0 . Eya cawama waesha! Katuleya ku cipande ca konkapo. Good effort! Let’s go on to the next section. Time to Learn Midline Impact Evaluation: Annex 5 41 Sub-test 2. LETTER SOUND IDENTIFICATION  Page 1  60 seconds  Ili ipepala nalikwata ifilembo ifili mu alufabeti wa Cibemba. Nomba njebako ifiunda fya ifi filembo, ulande fyonse ifyo wiishibe. Ibukisha ukuti temashina yalefwaikwa iyoo, leelo fiunda. Here is a page full of letters of the Cibemba alphabet. Please tell me the SOUNDS of as many letters of the alphabet as you can. Not their names, but their sounds. [point to the letter T] Icilangililo, iciunda ca cilembo ici t, ni /t/ For example, the sound of this letter is /t/. [point to the letter M] Natweshe ukucita ifi: Njebako iciunda ca cilembo ici: Let’s practice: Tell me the sound of this letter.  Eya cawama, iciunda ca cilembo ici ni /m/. Good, the sound of this letter is /m/.  Iciunda ca cilembo ici ni /m/. The sound of this letter is /m/. [point to the letter S] Nomba natwesha icilembo cimbi: Njebako iciunda ca cilembo ici. Now let us try another one. Tell me the sound of this letter.  Eya cisuma, iciunda ca cilembo ici ni /s/. Good, the sound of this letter is /s/.  Iciunda ca icilembo ici ni /s/. The sound of this letter is /s/. [point to first letter] Nganati “tampa”, utampe mpaka upwishe ipepala lyonse. uleesonta pali cila cilembo na ukunjeba iciunda ca cilembo mu kwikatisha ishiwi. Ubelenge mukwangufyanya kabili busaka-busaka. Ngawasanga icilembo ushishibe, wikokolapo konkanyapo ukwabula ukupoosa inshita kabiye pa cilembo cakonkapo. Biika umunwe pa cilembo ca kubalilapo. Nauipekanya? Tampako. When I say “Begin,” start here and go across the page. Point to each letter and tell me the sound of that letter in a loud voice. Read as quickly and carefully as you can. If you come to a letter you do not know, go on to the next letter. Put your finger on the first letter. Ready? Begin. Start the timer when the child reads the first letter.  If a child hesitates or stops on a letter for 3 SECONDS, point to the next letter and say “Go on”.  When the timer reaches 0, say “stop.”  If the child does not provide a single correct response on the first line (10 items), say “Thank you!”, discontinue this subtask, check the box at the bottom, and go on to the next subtask.  ( / ) Mark any incorrect letters with a slash ( Ø ) Circle self-corrections if you already marked the letter as incorrect ( ] ) Mark the final letter read with a bracket Examples: t m s 1 2 3 4 5 6 7 8 9 10 e F u t W a p b L a (10) U a e s o i B k E A (20) Time to Learn Midline Impact Evaluation: Annex 5 42 N F P Y c a M I u L (30) i A K η a L i a s M (40) u t U K m o u n i A (50) b a n a E a O u s E (60) A n a S M L m η b T (70) u t i w I u B c N I (80) a I w a i N k m a L (90) y P M A U O A n a A (100)  Time remaining on stopwatch at completion (number of SECONDS)  Exercise discontinued because the child had no correct answers in the first line Eya cawama waesha! Katuleya ku cipande cakonkapo. Good effort! Let’s go on to the next section. Time to Learn Midline Impact Evaluation: Annex 5 43 Sub-test 3. NON-WORD READING  Page 2  60 seconds  Apa pali amashiwi aya kupangafye ayashilepilibula nangu cimo mu Cibemba. Ndefwaya ukuti ubelenge aya mashiwi yonse ayo wingabelenga. Wilalumbula ifilembo cimo-cimo iyoo kanofye ukubelenga ishiwi lyonse. Here are some made￾up words in Icibemba. I would like you to read as many as you can. Do not spell the words, but read them. [point to the word “ut”] Icilangililo: Ili shiwi lyapangwa ilyakuti: “ut”. For example, this made-up word is: “ut”. [point to the word “dif”] Natweshe nomba: belenga ili shiwi. Let’s practice: Please read this word.  Eya cawama, ilishiwi ni “dif”. Good, this made-up word is “dif.”  Ilishiwi lyakupangafye “dif” talipilibula nangu cimo. This made-up word is “dif.” [point to the word “mab”] Nomba esha nalimbi: Belenga nalimbi ishiwi ili. Now let us try another one. Please read this word.  Ciisuma, ilishiwi lyaku pangafye ni “mab”. Good, this made-up word is “mab.”  Ili shiwi lyaku pangafye ni “mab”. This made-up word is “mab.” [point to first word] Ilyo ndetila “Tampa” utampile apa no kubelenga yonse ayali pepepala lyonse. Uleesonta pali cila ishiwi na ukubelenga ukwikatisha ishiwi. Belenga mukwangufyanya kabili mu mutekatima. Ngawasanga ishiwi ushishibe wikokolapo uye palikonkelepo. Sonta peeshiwi lyaku balilapo. waipekanya? Tampako. When I say “Begin,” start here [point to first word] and read across the page [point]. Point to each word and read it in a loud voice. Read as quickly and carefully as you can. If you come to a word you do not know, go on to the next word. Put your finger on the first word. Ready? Begin. Start the timer when the child reads the first word.  If a child hesitates or stops on a word for 3 SECONDS, point to the next word and say “Go on”.  When the timer reaches 0, say “stop.”  If the child does not provide a single correct response on the first line (5 items), say “Thank you!”, discontinue this subtask, check the box at the bottom, and go on to the next subtask.  ( / ) Mark any incorrect words with a slash ( Ø ) Circle self-corrections if you already marked the word as incorrect ( ] ) Mark the final word read with a bracket Examples: opa toti maba 1 2 3 4 5 lebi ndite luti oya lusi (10) Time to Learn Midline Impact Evaluation: Annex 5 44 mibu kibe shuti tobe njolo (20) angi shipe nomi sani opu (30) nepa wipi tupu naye koi (40) tate shuma telu shingu yoba (50) seni nupa etu kika onu (60) sale pafu tawe ebi ewa (70) ipa ombi kendi ngopa ndika (80) afu yema mawe tebi folo (90) fimu yapo tibu bife lefu (100)  Time remaining on stopwatch at completion (number of SECONDS)  Exercise discontinued because the child had no correct answers in the first line Eya cawama waesha! Katuleya kucipande cakonkapo. Good effort! Let’s go on to the next section. Time to Learn Midline Impact Evaluation: Annex 5 45 Sub-test 4. ORIENTATION TO PRINT  Page 3  X  Show the child a story passage in the learner stimuli packet. Read the instructions in the grey boxes below, recording the child’s response before moving to the next instruction. Materials: a passage from the learner stimuli packet Nshilefwaya ukuti ubelenge nomba. Pali ili pepala, kuti wayambila peesa ukubelenga ? Sontapo na umunwe obe. I don’t want you to read this now. On this page, where would you begin to read? Show me with your finger. a. (Child puts finger on the top row, left￾most word) O Correct O Incorrect O No Response Nomba ndanga uko wingalalola ilyo ulebelenga. Now show me in which direction you would read next. b. (Child moves finger from left to right) O Correct O Incorrect O No Response Nga wafika kumpela ya mulaini, ni peesa wingatendekela ukubelenga nakabili? When you get to the end of the line, where would you read next? c. (Child moves finger to left-most word of second line) O Correct O Incorrect O No Response Total Correct /3 Time to Learn Midline Impact Evaluation: Annex 5 46 Sub-test 5a. ORAL READING PASSAGE (2014)  60 seconds Sub-test 5b: READING COMPREHENSION Show the child the sheet in the learner stimulus booklet as you read the instructions.  If a child hesitates or stops on a letter for 3 SECONDS, say “Go on”.  If the child does not provide a single correct word on the first line of text, do not ask any comprehensio n questions. If a child says “I don’t know,” mark as incorrect. After the child is finished reading, REMOVE the passage from in front of the child. Ask the child only the questions related to the text read. A child must read all the text that corresponds with a given question. If the child does not provide a response to a question after 10 seconds, mark “no response” and continue to the next question. Do not repeat the question.  Aka akalyashi keepi. Ndefwaya ukuti ubelenge ukwikatisha ishiwi mu mutima nteka kabili bwangu bwangu. Nga wapwisha ukubelenga nalakwipusha ameepusho pa lyashi walabelenga. Nganati “Tampa”, walabelenga akalyashi busaka-busaka. Nga wasanga ishiwi ushishibe wikokolapo kabiye peeshiwi limbi. Sonta peeshiwi lyaku balilapo. Nauipekanya? Tampako. Here is a short story. I want you to read it out loud quickly, but carefully. When you finish, I will ask you some questions about what you have read. When I say “Begin,” read the story as best as you can. If you come to a word you do not know, go on to the next word. Put your finger on the first word. Ready? Begin.  Nomba nalakwipusha ameepusho ayanoono peelyashi wabelenga. Weshe ukwasuka ameepusho ayo wingeshiba bwino-bwino. Kuti wayasuka ameepusho mu lulimi wingafwaya ukubomfya. Now I am going to ask you a few questions about the story you just read. Try to answer the questions as well as you can. You can provide your answers in whichever language you prefer.  ( / ) Mark any incorrect letters with a slash ( Ø ) Circle self-corrections if you already marked the letter as incorrect ( ] ) Mark the final letter read with a bracket  () 1 = Correct () 0 = Incorrect () . = No response. Questions [Answers] Mulenga aya ku sukulu ilyaba ukutali na ku ηanda. 9 Isukulu lyakwa Mulenga lyaba kwisa? (ukutali na ku ηanda) 1 0 . Mu mainsa alakwata ubwafya bwakuya ku sukulu. 16 Ni nshitanshi Mulenga akwata ubwafya bwa kuya ku sukulu? (mu mainsa) 1 0 . Time to Learn Midline Impact Evaluation: Annex 5 Akamana kaba pakati ka isukulu na pa mushi kalesula. 25 Bwafyanshi Mulenga asanga pa kuya ku sukulu? (akamana kalesula) 1 0 . Abafyashi bakwe balamwafwilisha pa kwabuka akamana. 32 Bushe Mulenga abuka shani akamana? (Abafyashi bakwe balamwafwilisha) 1 0 . Abafyashi bakwe ngatabalipo abakumwabusha pa kamana limo-limo taya ku sukulu. Tatemwa ukulofwa ku sukulu. 47 Bushe Mulenga ayashani ku sukulu mu mainsa abafyashi bakwe ngatabalipo? (tayako ku sukulu) 1 0 .  Time remaining on stopwatch at completion (number of SECONDS)  Exercise discontinued: the child had no correct answers in the first line Eya cawama waesha! Katuleya ku cipande cakonkapo. Good effort! Let’s go on to the next section. Time to Learn Midline Impact Evaluation: Annex 5 Sub-test 6. LISTENING COMPREHENSION - ENGLISH  X  X  Nalakubelengela akalyashi akeepi akamucisungu mu kwikatisha ishiwi UMUKU UMO elyo nkwipushe ameepusho. Umfwikishe bwino, na ukwasuka ameepusho nalakwipusha mu mutima nteka. Kuti wayasuka ameepusho mu cilimi ico wingafwaya ukubomfya. Nauipekanya? Katutampe. I am going to read you a short story aloud ONCE in English and then ask you some questions. Please listen carefully and answer the questions as best as you can. You can answer the questions in whichever language you prefer. Ready? Let’s begin. Remove the learner stimuli booklet from the child’s view. Do not allow the child to look at the passage or the questions. If a child says “I don’t know,” mark as incorrect.  () 1 = Correct () 0 = Incorrect () . = No response. Every morning Mukata’s father walks out from the house. “Where are you going?” Mukata asked. “I am going to the river to fish,” said his father. Mukata asked to go with him. “When you grow up I will teach you how to fish,” said Mukata’s father. The next day father was surprised to see Mukata down by the river with a hook. What does Mukata’s father do every morning? / Finshi bawishi Mukata bacita cila lucelo? [walks out of the house] 1 0 . Where does he go? /Bushe bayakwi? [to the river] 1 0 . What did Mukata ask his father? / Finshi Mukata aipwishe bawishi? [he asked to go with him] 1 0 . What does Mukata’s father do? / Mulimonshi bawishi Mukata babomba? [he is a fisherman] 1 0 . What was Mukata doing by the river? /Finshi Mukata aaleecita ku mumana? [trying to fish] 1 0 . Time to Learn Midline Impact Evaluation: Annex 5 49 THE COMMUNITY SCHOOL HEAD TEACHER QUESTIONNAIRE1 Time to Learn Midline Evaluation Community School Head Teacher Questionnaire Notes to interviewer are in italics. PREPARATION This questionnaire is intended for the Head teacher or, if the Head teacher is not present, the individual who is the most able to provide information about the school. Prior to starting data collection, the Head Teacher, PCSC Chair, evaluation team manager, and data collectors should meet for introductions and the informed consent procedure. 1 Note that this questionnaire was given in all three languages. INTRODUCE YOURSELF AND PROJECT: Thank you very much for setting time aside for us today. My name is ________, and I am part of the USAID-funded Time to Learn project (TTL), which collaborates with the MESVTEE to improve reading in community schools and increase equitable access to education for orphans and other vulnerable children. INTRODUCE EVALUATION: This interview is part of the TTL Midline Evaluation, which will assess how TTL activities are fostering learner literacy so we can improve project activities and increase learner literacy. Yours is one of 100 schools sampled for this evaluation in 6 provinces (Central, Copperbelt, Eastern, Lusaka, Muchinga, and Southern). We are interested in learning more about your school. This is not an evaluation of your school, your performance, or the MESVTEE, and we will not be reporting on this school to anyone. I also want to tell you that I am not directly involved in any funding decisions for TTL; I am here simply to gather information about the effectiveness of TTL. EXPLAIN CONFIDENTIALITY AND INFORMED CONSENT: Before we begin, we want to let you know that none of the information we gather during today’s visit will be attributed to a specific school or person. All information used in the evaluation report will only be attributed to a general stakeholder group (e.g., teacher, Head teacher, boy/girl learners in Province X, MESVTEE officials) and this school will not be identified in any reports. Your participation is completely voluntary and you are free to stop the interview at any time or not to answer any questions, and this will not affect your relationship with TTL or the MESVTEE. CONFIRM TIMEFRAME AND ASK PERMISSION TO RECORD: This interview will take about 60 minutes and if you don’t mind, I would like to take notes today on this tablet. ANY QUESTIONS? Before we begin, do you have any questions about the interview? ASK PERMISSION: Are you willing to participate? YES informed consent provided (if not provided, discontinue) Time to Learn Midline Impact Evaluation: Annex 5 50 BACKGROUND Data collector to fill in items a-k individually or with help from respondent, as necessary. a. Data Collector’s Name: b. Respondent’s Name: c. Respondent’s Sex: Male/Female: d. Respondent’s Mobile Phone Number: e. EMIS: f. Date: g. Province: h. School Name: i. Zone: j. District: k. Distance to DEBS: (in kilometers) Confirm that you are speaking to the Head teacher 1. What is your position at this school? (Select all that apply) a. Head Teacher/Teacher in Charge b. School Manager/Supervisor c. Deputy Head teacher d. Senior teacher e. Teacher f. PCSC chair g. Other a. Please specify: SCHOOL PROFILE I am now going to ask some information about the number of classes, learners, and teachers at your school. It would be helpful if you could have a school register with you to help answer these questions. 2. In what year was the school founded? ____ 3. How many classes/streams are there in the school this year? ____ 4. What is the highest grade being taught in the school this year? Do not read response choices. c. Grade 1 d. Grade 2 e. Grade 3 f. Grade 4 g. Grade 5 h. Grade 6 Time to Learn Midline Impact Evaluation: Annex 5 51 i. Grade 7 j. Higher than Grade 7 5. Are there any combined classes in the school this year? k. Yes (Go to 6) l. No (Skip to 7) 6. Which classes are combined? (Select all that apply) a. Grades 1 and 2 b. Grades 2 and 3 c. Grades 3 and 4 d. Grades 4 and 5 e. Grades 5 and 6 f. Grades 6 and 7 g. Other (Please specify) f. None Based on responses above, ask the Head teacher about the number of classes, learners, and teachers for each grade that is taught in the school. The tablet automatically calculates the number of total learners and teachers. # # 2014 Classes Total Learners # # Male Learners # # Female Learners Total Learners # # Male Teachers # # Female Teachers Total Teachers 7. Grade 1 8. Grade 2 9. Grade 3 10. Grade 4 11. Grade 5 12. Grade 6 13. Grade 7 a. Male b. Female Time to Learn Midline Impact Evaluation: Annex 5 52 14. What is the total number of teachers for all grades in your school? 15. How many teachers at the school are volunteers and do not get paid by anyone (not expecting a stipend or salary)? 16. How many teachers at the school are retired from government schools? 17. How many teachers at the school are employed by the MESVTEE, but seconded to this school? 18. How many teachers at the school are trained and waiting for postings from the Ministry? HEAD TEACHER BACKGROUND This section is only for respondents who indicated that they were the Head teacher/ Teacher in Charge. I would like to ask you a few questions about your background and role as a Head teacher. 19. What is the highest level of academic education you have completed? Do not read response choices. a. Grade 7 (exam passed) b. Grade 8 c. Grade 9 (exam passed) d. Grade 10 e. Grade 11 f. Grade 12 (exam passed) g. BA/BS h. MA/MS i. Other i. Please specify: 20. Have you received any pre-service training? a. Yes (Go to 21) b. No (Skip to 22) 21. What is the highest level of pre-service training you have received? a. Teaching certificate/diploma: Primary Education b. Teaching certificate/diploma: Early Childhood Education c. Diploma: Secondary Education d. Degree/Bachelors of Primary Education/Bachelors of Education e. Other ii. Please specify: f. None Time to Learn Midline Impact Evaluation: Annex 5 53 22. How many years have you been in the teaching service? ___ 23. How many years have you been a Head teacher at this school? ___ 24. Were you a teacher before becoming a Head teacher? a. Yes (Go to 25) b. No (Skip to 26) 25. For how many years were you a teacher before becoming a Head teacher? __ 26. Were you a Head teacher at another school before becoming a head teacher at this school? a. Yes (Go to 27) b. No (Skip to 28) 27. For how many years were you a Head teacher at another school? ___ 28. Do you currently teach any classes? a. Yes (Go to 29) b. No (Skip to 30) 29. If yes, which grade do you currently teach? (Select all that apply). a. Grade 1 b. Grade 2 c. Grade 3 d. Grade 4 e. Grade 5 f. Grade 6 g. Grade 7 h. Combined grades 1 and 2 i. Combined grades 2 and 3 j. Combined grades 3 and 4 k. Combined grades 4 and 5 l. Combined grades 5 and 6 m. Combined grades 6 and 7 n. Other iii. Please specify: 30. Who is your employer? / Who hired you? (Select all that apply). a. Government b. Parent Community School Committee c. Church d. Private Employer e. Head Teacher is a volunteer (not expecting a salary or stipend) f. Other iv. Please specify: Time to Learn Midline Impact Evaluation: Annex 5 54 31. What are your most important roles as a head teacher? (Choose three) Ask as an open-end question, then select appropriate responses. Do not read response choices to the Head teacher unless the respondent is unable to answer the question). a. Information and records management (i.e., maintaining attendance records, school log book, staff files, etc.) b. Financial management (i.e., maintaining financial records, creating budgets, etc.) c. Supervising teachers and classrooms d. Monitoring curriculum and instruction e. Monitoring learners and teachers performance f. Fundraising g. Implementing MESVTEE policy h. Working with the community (i.e., PCSC, NGOs, etc.) i. Providing psychosocial support to learners and teachers j. Other v. Please specify: 32. Have you received any in-service training on how to teach early grade reading or on leadership skills since 2012? Probe if respondent is not sure or answers the question with no: In-service training is training received since starting service as a teacher, and does not include pre-service training. This training could be called in-service training, Grace Meetings, Teacher Learner Circles (TLC), Continuing Professional Development (CPD), workshops, Inset, or go by another name. We are not interested in the name of the training, only in the content. a. Yes (Go to 33) b. No (Skip to 38) 33. Who conducted the training? Check all that apply. 34. What was the name or focus of the training? 35. In what year was the training? c. vi. vii. viii. ix. x. a. Time to Learn Check all that apply. a. Quickstart b. Reading/module reading c. Writing/module writing d. Alphabet/module alphabet e. Assessment training, also known as CPD literacy training/CPD for community school teachers Time to Learn Midline Impact Evaluation: Annex 5 55 f. Operational Guidelines for Community School (OGCS) orientation g. PCSC training in supporting reading (at the household and community level) (could also be known as Parent Community Literacy Mobilization) h. Educational Leadership and Management i. Don’t know j. Other (Specify) k. MESVTEE xi. xii. l. Other government institution Please specify xiii. xiv. m. Private institution Please specify: xv. xvi. n. Nongovernmental organization Please specify: xvii. xviii. o. Other Please specify: xix. xx. If the respondent indicated attendance at a TTL training, proceed to question 36. If they did not report attendance at a TTL training, skip to question 37. 36. Please specify: Did you receive any teaching and learning materials at the TTL training to bring back to the school? a. Yes (Go to 37) b. No (Skip to 37) 37. Please name all the TTL teaching and learning materials you ever received at a TTL training or through the Education Ministry’s zones or districts. (Select all that apply) Ask the respondent about each item in turn, i.e., “Did you receive flash cards?” a. Flash cards b. Story cards c. CASAS graded readers (story books) d. Maiden Health Books (in local languages) e. Assessment sheets (to record results) f. Assessment booklets/guides (instructional) g. Teachers Guide on School-based Assessment h. School register i. Enrollment forms Time to Learn Midline Impact Evaluation: Annex 5 56 j. Training Module Materials a. Materials received in quickstart training b. Materials received in reading/module reading training c. Materials received in writing/module writing training d. Materials received in alphabet/module alphabet training k. Zambia Basic Education Syllabi (Grade 1-7) l. Radios m. Operational Guidelines for Community Schools (PCSC) n. Community Literacy Mobilization Manual (for PCSC) o. Education Leadership and Management Training Manual for Community School Head Teachers p. Other i. Please specify: FUNDING AND RESOURCES I am now going to ask a few questions about the school’s budget and then questions about and non-monetary resources. Please answer these questions using re-based Kwacha/the new currency. 38. Does your school have an annual operating budget? a. Yes (Go to 39) b. No (Skip to 40) 39. What is the annual operating budget for your school? _____ Kwacha 40. Please list the current funding sources and amounts: a. Source 1_____________ i. Amount __________ b. Source 2 ____________ ii. Amount___________ c. Source 3 ____________ iii. Amount ___________ d. Source 4 ____________ iv. Amount ___________ TOTAL 41. Has the school had major changes in funding over the last 2 years? a. Yes b. No 42. Does the school currently have a feeding program? a. Yes (Go to 43) b. No (Skip to 44) Time to Learn Midline Impact Evaluation: Annex 5 57 43. Who funds the feeding program? a. MESVTEE b. Other government institution i. Please specify: c. PCSC d. Nongovernmental organization i. Please specify: e. Private institution i. Please specify: f. Other i. Please specify: 44. What other, non-monetary support has the school received from the MESVTEE in the previous year? Check all that apply. Zonal i. Basic classroom materials/free basic materials (chalk, notebooks/exercise books, etc.) ii. TTL Teaching and learning materials iii. Teaching and learning materials iv. Teacher trainings/continuing professional development v. School visits/monitoring vi. Guidance through ZICs vii. Building materials (cement/wood) viii. Infrastructure (e.g., toilets, classrooms, teacher’s house) ix. Furniture (e.g., desks/tables/chairs) x. Seconded teachers xi. None xii. Other Please specify: District i. Basic classroom materials/free basic materials (chalk, notebooks/exercise books, etc.) ii. TTL Teaching and learning materials iii. Other teaching and learning materials iv. TTL Teacher trainings/continuing professional development v. Non-TTL teacher training/continuing professional development vi. School visits/Monitoring vii. Building materials (e.g., cement/wood) viii. Infrastructure (e.g., toilets, classrooms, teacher’s house) ix. Furniture (e.g., desks/tables/chairs) x. None Time to Learn Midline Impact Evaluation: Annex 5 58 xi. Other Please specify: Provincial i. Basic classroom materials (chalk, notebooks, etc.) ii. Teaching and learning materials iii. Building materials (e.g., cement/wood) iv. Infrastructure (e.g., toilets, classrooms, teacher’s house) v. Furniture (e.g., desks/tables/chairs) vi. None vii. Other Please specify: National i. Basic classroom materials (chalk, notebooks, etc.) ii. Teaching and learning materials iii. Building materials (e.g., cement/wood) iv. Infrastructure (e.g., toilets, classrooms, teacher’s house) v. Furniture (e.g., desks/tables/chairs) vi. None vii. Other Please specify: 45. What other, non-monetary support has the school received from other partners in the previous year? Check all that apply. i. Basic classroom materials/free basic materials (chalk, notebooks/exercise books, etc.) ii. Teaching and learning materials iii. Teacher training/Continuing professional development iv. School visits/Monitoring v. Building materials (e.g., cement/wood) vi. Infrastructure (e.g., toilets, classrooms, teacher’s house) vii. Furniture (e.g., desks/tables/chairs) viii. None ix. Other Please specify: 46. Please indicate the level of each resource the school has in relation to its needs: a. More than Adequate b. Adequate c. Inadequate d. None Classrooms Learner desks and chairs Teacher desks and chairs Time to Learn Midline Impact Evaluation: Annex 5 59 Class boards/chalk boards Basic classroom materials (chalk, notebooks) Teaching and learning materials Water supply Male staff toilets Female staff toilets Male learner toilets Female learner toilets INSTRUCTIONAL PRACTICES I will now ask some questions on reading instruction and assessment in grades 1 and 2. 47. What is the main language that teachers use to teach reading in Grades 1 and 2? a. Bemba b. Chewa/ Nyanja c. Town Nyanja d. Tonga e. Lozi f. English g. Other i. Please specify: 48. According to the curriculum, in which grade should schools start to teach reading/literacy in English? a. Grade 1 b. Grade 2 c. Grade 3 d. Grade 4 e. Grade 5 f. Grade 6 g. Grade 7 h. Other i. Please specify: 49. How much is local language used in Grade 1 literacy classes? a. All of the time b. More than half of the time c. About half of the time Time to Learn Midline Impact Evaluation: Annex 5 60 d. Less than half of the time e. None of the time f. Don’t know 50. How much is local language used in Grade 2 literacy classes? a. All of the time b. More than half of the time c. About half of the time d. Less than half of the time e. None of the time f. Don’t know 51. What are the factors that prevent teachers from using the local language more often to teach reading in grades 1 and 2 in this school? (Select all that apply) a. Teachers do not speak the local language b. Teachers do not know how to teach reading in the local language c. The school does not have local language materials d. Teachers think English is required e. Teachers think English is more important f. Parents do not support use of the local language to teach reading g. All grade 1 and 2 reading lessons are in local language h. None i. Other: __________________________ 52. How many minutes per day do you expect teachers to spend teaching reading and writing/literacy classes to their learners? a. Grade 1: ____ b. Grade 2: ____ 53. What methods do teachers in grades 1 and 2 use the most often to teach reading skills? (Choose top three) Ask as an open-ended question, then select appropriate responses. Do not read response choices to the Head teacher unless the respondent is unable to answer the question. a. Learners match pictures to words b. Learners memorize words c. Learners recite words d. Learners sound out letters e. Learners decode words/make words with syllables f. Learners read g. Teacher reads to learners h. Other ii. Please specify: Time to Learn Midline Impact Evaluation: Annex 5 61 i. Don’t know 54. How often do teachers prepare lesson plans at your school? a. For every lesson b. For most lessons c. For some lessons d. For a few lessons e. Never f. Don’t know 55. Does the school have reading materials that learners are allowed to take home? a. Yes b. No 56. Have you ever received a TOTAL library box/trunk through Time to Learn? Probe: They could be called library books, mobile library books, or book boxes and are different from the other Teaching and Learning Materials you have received because they are for the use of the community outside of school, and parents and learners can borrow these books and take them home. a. Yes b. No c. Don’t know 57. How often are learners in grades 1 and 2 assessed on their reading ability? a. Weekly b. Monthly c. Termly d. Yearly e. Never f. Don’t know 58. What formal methods do you use the most often to assess reading acquisition in Grade 1 and 2 learners? (Choose top three) a. 5th weekly or monthly assessment b. 10th weekly or term assessment c. Use reading Continuous Assessment tool d. Give quiz or test e. Red level tracker f. Performance level descriptors g. Other iii. Please specify: h. No formal learner assessment i. Don’t know 59. What informal methods do you use the most often to assess reading ability in Grade 1 and 2 learners? Time to Learn Midline Impact Evaluation: Annex 5 62 Ask as an open-ended question, then select appropriate responses. Do not read response choices to the Head teacher unless the respondent is unable to answer the question. a. Ask learners questions about lesson b. Monitor learners as they work c. Observe learners in group activities d. Listen to individual learners read aloud e. Ask learner to tell about what they have just read f. Check learner’s exercise book or homework g. Other i. Please specify: h. No informal learner assessment i. Don’t know EDUCATIONAL LEADERSHIP AND MANAGEMENT I will now ask some questions on the types of support that are available for teachers at this school. 60. Do you ever observe literacy lessons being taught? a. Yes (Go to 64) b. No (Skip to 68) 61. How often do you observe literacy lessons being taught? a. Weekly b. Monthly c. Termly d. Yearly 62. How often do you observe each teacher? a. Weekly b. Monthly c. Termly d. Yearly 63. Following an observation, how often do you give feedback to the teacher? a. Every time b. More than 50 percent of the time c. About 50 percent of the time d. Less than 50 percent of the time e. Never (Skip to 68) 64. How do you give feedback to the teacher? Probe: Ask teacher to describe how feedback is given, and check all that apply, as they speak. Do not read response categories. If teachers do not speak to these categories, prompt them for type (written/verbal) and setting (one-on-one/group) of feedback. Time to Learn Midline Impact Evaluation: Annex 5 63 a. Written form b. Verbally, one-on-one c. Verbally, in a group d. Other: ________ 65. Have you ever organized a meeting with teachers to specifically address teaching quality? a. Yes (Go to 69) b. No (Skip to 70) 66. How often do you organize these meetings? a. Monthly b. Termly c. Yearly d. Never 67. Have you received any training or sensitization on how to provide feedback to teachers? a. Yes b. No 68. Have the teachers in your school received any in-service or in-house training since 2012? a. Yes (Go to 72) b. No (Skip to 76) 69. Who conducted the training? Check all that apply. 70. What was the name or focus of the training? 71. In what year was the training? 72. How many teachers attended the training? i. ii. iii. iv. v. vi. vii. viii. ix. a. Time to Learn Check all that apply. a. Quickstart b. Reading/ module reading c. Writing/ module writing d. Alphabet/ module alphabet e. Assessment training, also known as CPD literacy training/ CPD for community school teachers i. ii. Male ____ iii. Female ____ Time to Learn Midline Impact Evaluation: Annex 5 64 f. Operational Guidelines for Community School (OGCS) orientation g. PCSC training in supporting reading (at the household and community level) (could also be known as Parent Community Literacy Mobilization) h. Educational Leadership and Management i. Other (Specify) j. Don’t know b. Head teacher i. ii. Male ___ Female ___ c. Zonal MESVTEE official i. ii. Male ___ Female ___ d. Other government institution Please specify: i. ii. Male ___ Female ___ e. Private institution Please specify: ii. iii. iv. Male ____ v. Female ____ f. Nongovernmental organization Please specify: i. ii. iii. Male ____ iv. Female ____ g. Other Please specify: ii. iii. iv. Male ____ v. Female ____ PARENT COMMUNITY SCHOOL COMMITTEE I am now going to ask a few questions about the role of parents at this school. 73. Does this school have a Parent Community School Committee (PCSC)? a. Yes (Go to 77) b. No (Skip to 86) 74. How often does it meet? a. Monthly or more b. Termly Time to Learn Midline Impact Evaluation: Annex 5 65 c. Yearly d. No regular meetings 75. In what ways is the PCSC involved in school operations? (Select all that apply) Ask as an open-end question, and then select the appropriate responses. Do not read responses choices to the Head teacher unless the respondent is unable to answer the question. a. Monitoring attendance of teachers b. Monitoring attendance of learners c. Hiring and firing teachers, including Head teacher d. Curriculum design e. Build and maintain school facilities f. Purchase teaching and learning materials or classroom supplies, and monitor their use g. Help with teaching/learning activities h. Help with sports activities for learners i. Provide social support to learners j. Observe classes k. Monitor educational outcomes l. Liaise with MESVTEE m. Engage with/sensitize parents n. Collect teachers’ salaries o. Fundraising p. Other i. Please specify: 76. How is your relationship with the PCSC? a. Excellent b. Good c. Fair d. Poor e. Very poor 77. How are your teachers’ relationships with the PCSC? a. Excellent b. Good c. Fair d. Poor e. Very poor 78. Are academic reports provided to the PCSC? a. Yes b. No Time to Learn Midline Impact Evaluation: Annex 5 66 79. Did the PCSC receive any training since 2012? a. Yes (Go to 83) b. No (Skip to 86) 80. Who conducted the training? Check all that apply. 81. What was the name or focus of the training? 82. In what year was the training? ii. iii. iv. v. vi. a. Time to Learn Check all that apply. vii. Operational Guidelines for Community School (OGCS) orientation viii. PCSC training in supporting reading (at the household and community level) (could also be known as Parent Community Literacy Mobilization) ix. Other (Specify) x. Don’t know xi. c. MESVTEE xii. xiii. d. Other government institution Please specify: xiv. xv. e. Private institution Please specify: xvi. xvii. f. Nongovernmental organization Please specify: xviii. xix. g. Other Please specify: xx. xxi. ABSENTEEISM AND ATTRITION I am now going to ask a few questions about learner and teacher absenteeism and drop-out. 83. How many days has your school closed in the last year outside of the normal school calendar? If 0, skip to 88. 84. Why did the school close? (Select all that apply) Ask as an open-ended question, then select the appropriate responses. Do not read response choices to the Head teacher unless the respondent is unable to answer the question. h. Teachers left the school Time to Learn Midline Impact Evaluation: Annex 5 67 i. School structure collapsed j. Local festivals k. Teachers were ill l. Teachers were attending trainings m. Rainy season/difficult to get to school n. Teacher and/or learners participating in seasonal activities (farming, etc.) o. Conflicts p. Funerals q. Other xxii. Please specify: r. Don’t know 85. On any day, how many of your teachers are late to class or do not show up? s. Male teachers ___ t. Female teachers ___ 86. On any day, how many of your learners are late to class or do not show up? u. Male learners ____ v. Female learners ____ 87. How many of your learners dropped out of school in the last year? w. Male learners ____ x. Female learners ____ Thank you for your participation! Do you have any additional questions about the interview? Time to Learn Midline Impact Evaluation: Annex 5 68 CLASSROOM OBSERVATION TOOL2 Time to Learn Midline Evaluation EARLY GRADE READING CLASSROOM OBSERVATION PROTOCOL AND TEACHER INTERVIEW PREPARATION Selection of teacher and class This observation protocol is intended for a Grade 2 literacy lesson. In schools where there is more than one Grade 2 classroom or there are mixed grades, the evaluation team manager will select a classroom at random. The Head teacher should inform the teacher of the classroom being observed as soon as the decision is made and, if possible, the teacher should join the Head teacher, PCSC Chair, and evaluation team manager for introductions and the informed consent conversation prior to the beginning of data collection at the school. You should observe a class called ‘Literacy’ or ‘Literacy hour’. If this class does not exist in the syllabus or is not taught on the day you visit the school, observe the following classes in this order of priority: a. Reading b. Writing c. Any class that is about a local language (e.g., “Bemba”, “Tonga”, “Nyanja”) d. English Frequently, teachers do not teach the class that is listed in the syllabus/timetable for that hour. Thus, you need to ask the teacher before the class which class the teacher will be teaching. Do not rely on the syllabus. Meet with the teacher at least 5-10 minutes before the lesson begins. If the teacher who is being observed participated in the informed consent conversation with the Head teacher and PCSC Chair, then skip to “Discuss the following with the teacher”. If the teacher was not present for the informed consent conversation, begin with the following: INTRODUCE YOURSELF AND THE PROJECT: Thank you very much for setting time aside for us today. My name is ________, and I am part of the USAID-funded Time to Learn project (TTL), which collaborates with the MESVTEE to improve reading in community schools and increase equitable access to education for orphans and other vulnerable children. INTRODUCE EVALUATION: This observation is part of the TTL Midline Evaluation, which will assess how TTL activities are fostering learner literacy so we can improve project activities and increase learner literacy. Yours is one of 100 schools sampled for this evaluation in 6 provinces (Central, Copperbelt, Eastern, Lusaka, Muchinga, and Southern). We are interested in learning how you are teaching the Primary Literacy Program for the purpose of understanding learner learning. This is not an evaluation of your school, your performance, or the MESVTEE, and we will not be reporting on this school to anyone. I also want to tell you that I am not directly involved in any funding decisions for TTL; I am here simply to gather information about the effectiveness of TTL. EXPLAIN CONFIDENTIALITY AND INFORMED CONSENT: Before we begin, we want to let you know that none of the information we gather during today’s visit will be attributed to a specific school or person. All the information used in the evaluation report will only be attributed to a general stakeholder group (e.g., teacher, Head teacher, boy/girl learners in Province X, MESVTEE officials) and this school will not be 2 Note that translations into three languages were provided. Time to Learn Midline Impact Evaluation: Annex 5 69 identified in any reports. Your participation is completely voluntary and you are free to stop the observation at any time or not to answer any questions, and this will not affect your relationship with TTL or the MESVTEE. CONFIRM TIMEFRAME AND ASK PERMISSION TO RECORD: This observation will take about 60 minutes and if you don’t mind, I would like to take notes today on this tablet. After I observe your class I would like to ask you a series of questions, which will take approximately 60 minutes, to understand your reflections on the Primary Literacy Program. ANY QUESTIONS? Before we begin, do you have any questions about the interview? ASK PERMISSION: Are you willing to participate? *ii* YES informed consent provided (if not provided, discontinue) Discuss the following with the teacher after having received informed consent: • We would also like to take pictures for the final publication; you will not be identified in any photos. Is that alright? INDICATE YES ____ NO ____ (If “no”, take no pictures.) • I would like you to explain our presence to your learners when we enter your classroom. I suggest that you tell them, in a friendly and comforting manner, that: • We have some guests in our classroom today who want to understand what we do together in our classroom. • Everyone should do their best to ignore the observers; pretend they are not there. NOTE: It may help allay learner curiosity for the observers to briefly introduce themselves. Ask the teacher if s/he has any questions about the observation or evaluation. ENTER ANY QUESTIONS HERE: __________________________________________________________________________________________ OVERVIEW OF THE CLASSROOM OBSERVATION PROTOCOL AND INTERVIEW PROTOCOL OBJECTIVES 1. To document data collector’s observation of Grade 2 literacy lessons to capture how teachers are implementing key actions they are exposed to during TTL teacher trainings 2. To identify potential barriers in teachers’ knowledge, skills, and attitudes to further change. These data will be used to improve TTL activities. HOW TO USE THIS PROTOCOL This observation protocol is divided into 3 parts: Part 1: Background information: Data collectors fill in items 1-12 individually. For section 1.A ‘School information’ the help of the respondent may be asked. Time to Learn Midline Impact Evaluation: Annex 5 70 Part 2: Classroom observation: This part consists of seven domains. Each domain represents an area covered by the TTL and MESVTEE training curricula. Domains are subdivided into specific criteria representing MESVTEE standards for early grade reading instruction. You will observe what is occurring during the lesson in 3-minute intervals. During each interval, check the boxes for the criteria being fulfilled (satisfied) during that interval. The tablet keeps track of the time and automatically moves on to the next column after each 3-minute interval. You do not need to pay attention to time during the lesson. Unchecked boxes are automatically recorded as unfulfilled criteria for that interval. You may check as many criteria as satisfied during any one interval and any single criterion may be checked for more than one interval. Do not give the teacher feedback on the lesson. If the teacher requests feedback, explain that you are not assessing the teacher; you are simply trying to understand how literacy is taught. Additional comments/observations should be written clearly in the last two columns to explain, add to, or elaborate on the yes/no responses. Part 2.B.: Post-class observations on classroom environment and instruction: Data collectors fill in items 13-15 individually. Data collectors must make observations regarding questions 14 and 15 during the classroom observation in order to be able to answer the questions afterwards. Part 3: Teacher questionnaire: The teacher questionnaire has seven sections with 152 questions. It is essential to ask every single question. Before starting a new section, the data collector reads an introductory sentence that explains the new section to the teacher. Time to Learn Midline Impact Evaluation: Annex 5 71 PART I: BACKGROUND INFORMATION Data collectors fill in items 1-12 individually. For section 1.A ‘School information’ the help of the respondent may be asked. 1. A SCHOOL INFORMATION 1. Data collector’s name 2. EMIS___________ 3. __________________________ 4. Province__________________________________________________________________________ 5. ___________________________________________________________________________ 6. Zone _ 7. District____________________________________________________________________________ 8. School name___________________________________________________________________________ 9. Subject a. Reading b. Writing c. Reading and Writing/Literacy d. Zambian languages (iCiBemba/CiTonga/CiNyanja) e. English 10. Grade level of class observed: a. Grade 1 b. Grade 2 c. Multi-grade d. Other___ 1. B AUTO FIELDS 11. Date of Observation ______ /______ /______ dd / mm / yyyy 12. Start time __:___:___ 13. End time __:___:___ hh:mm:ss . Time to Learn Midline Impact Evaluation: Annex 5 72 PART 2: CLASSROOM OBSERVATION This part consists of seven domains. Each domain represents an area covered by the TTL and MESVTEE training curricula. Domains are subdivided into specific criteria representing MESVTEE standards for early grade reading instruction. You will observe what is occurring during the lesson in 3-minute intervals. During each interval, check the boxes for the criteria being fulfilled (satisfied) during that interval. The tablet keeps track of the time and automatically moves on to the next column after each 3-minute interval. You do not need to pay attention to time during the lesson. Unchecked boxes are automatically recorded as unfulfilled criteria for that interval. You may check as many criteria as satisfied during any one interval and any single criterion may be checked for more than one interval. Do not give the teacher feedback on the lesson. If the teacher requests feedback, explain that you are not assessing the teacher; you are simply trying to understand how literacy is taught. Additional comments/observations should be written clearly in the last two columns to explain, add to, or elaborate on the yes/no responses. No. Criteria P1 P2 P3 P4 P5 P6 P7 P8 P9 P 10 P 11 P 12 P 13 P 14 P 15 P 16 P 17 P 18 P 19 P 20 Learner response Comments on teacher practice I. ORIENTATION TO PRINT (top to bottom, left to right, return) 1 The teacher uses a finger or pointer to show which direction to read in and how to move to the next line. 2 The teacher watches learners’ fingers as they track across the page. 3 The teacher uses terms such as “top of the page,” “bottom of the page,” “next line.” 4 The teacher shows learners what these terms mean by using a pointer or fingers. II. LETTER-SOUND KNOWLEDGE (letter sounds production and recognition) Time to Learn Midline Impact Evaluation: Annex 5 73 No. Criteria P1 P2 P3 P4 P5 P6 P7 P8 P9 P 10 P 11 P 12 P 13 P 14 P 15 P 16 P 17 P 18 P 19 P 20 Learner response Comments on teacher practice 5 The teacher teaches or reviews letter sounds. 6 The teacher demonstrates phonemic awareness (letter sounds) to build and decode (take apart) words and syllables. 7. The teacher requires learners to demonstrate phonemic awareness (letter sounds) to build and decode (take apart) words and syllables. III. WORD DECODING (syllables, coding and decoding) 7 The teacher teaches or reviews syllables. 8 The teacher teaches coding and decoding skills (building words from syllables and taking apart words using syllables). 9 The teacher sounds out the letters in words. 10 The teacher asks learners to sound out the letters in words. IV. ORAL PASSAGE READING (fluency) 11 The teacher offers learners an opportunity to read connected text aloud or silently. 12 The teacher models fluent reading by reading aloud to the class. V. READING COMPREHENSION (comprehension) Time to Learn Midline Impact Evaluation: Annex 5 74 No. Criteria P1 P2 P3 P4 P5 P6 P7 P8 P9 P 10 P 11 P 12 P 13 P 14 P 15 P 16 P 17 P 18 P 19 P 20 Learner response Comments on teacher practice 13 The teacher asks questions about material that the learners have read themselves. 14 The teacher asks learners to predict what will happen as they are reading (reading individually or together as a class). 15 The teacher introduces and explains new vocabulary found in classroom reading material. VI. LISTENING COMPREHENSION (comprehension) 16 The teacher checks learners’ comprehension, including vocabulary, while s/he reads aloud. 17 The teacher asks prediction questions as s/he reads to the class. 18 The teacher encourages dialogue (participation, discussion, conversation, not just questions and answers) between learners and/or between teacher and learner as a means of teaching listening skills. VII. WRITING (handwriting/letter formation, drawing, original writing, copying) 19 The teacher practices strokes with the learners, such as: /// ||| \\\ OOO 20 The teacher explicitly teaches learners how to form letters by having learners write the curves and lines of a letter in their notebooks, on the board, or in Time to Learn Midline Impact Evaluation: Annex 5 75 No. Criteria P1 P2 P3 P4 P5 P6 P7 P8 P9 P 10 P 11 P 12 P 13 P 14 P 15 P 16 P 17 P 18 P 19 P 20 Learner response Comments on teacher practice the air. 21 The teacher asks learners to copy words and letters from the board. 22 The teacher gives learners the opportunity to draw and/or write without telling them what to draw or write. *ii* Check if lesson continued beyond the 60-minute time frame. P = 3-minute period (interval). PART 2B: POST CLASS OBSERVATIONS ON CLASSROOM ENVIRONMENT AND INSTRUCTION Data collectors fill in items 14-16 individually. Data collectors must make observations regarding questions 14 to16 during the classroom observation in order to be able to answer the questions afterwards. 14. What learning aids are observed in the classroom? (Check all that apply): a. Chalkboard b. Posters/talking walls c. Textbooks d. Supplementary reading materials e. Flashcards f. Exercise books or slates g. TTL materials h. Other_________ i. None 15. What learning aids are observed in use during the lesson? (Check all that apply): a. Chalkboard Time to Learn Midline Impact Evaluation: Annex 5 76 b. Posters/talking walls c. Textbooks d. Supplementary reading materials e. Flashcards f. Exercise books or slates g. TTL materials h. Other_________ i. None 16. In which language did the lesson take place? j. Bemba k. Nyanja l. Tonga m. English n. Other i. If other, specify Time to Learn Midline Impact Evaluation: Annex 5 77 PART 3: TEACHER INTERVIEW 3. A TEACHER BACKGROUND The teacher questionnaire has seven sections with 152 questions. It is essential to ask every single question. Before starting a new section, the data collector reads an introductory sentence that explains the new section to the teacher: I would like to ask you a few questions about your background.” 1. This teacher was also interviewed using the Community School and Head Teacher Questionnaire. a. Yes b. No 2. Teacher’s last name 3. Teacher’s first name 4. Sex a. Male b. Female 5. Age (in years) 6. Years teaching (active) 7. Years teaching at present school 8. Designation a. Teacher b. Senior Teacher c. Deputy Head teacher d. Head teacher/Teacher in Charge e. Acting teacher f. Acting senior teacher g. Acting Head teacher h. Other i. Specify 9. Employment status a. Government teacher (seconded) b. Trained teacher, not deployed c. Volunteer teacher (community members) 10. Highest academic qualification completed: a. Grade 7 (exam passed) b. Grade 8 c. Grade 9 (exam passed) d. Grade 10 e. Grade 11 Time to Learn Midline Impact Evaluation: Annex 5 78 f. Grade 12 (exam passed) g. BA/BS h. MA/MS i. Other i. Specify 11. Professional qualification (select the highest achieved) a. Teaching certificate/Early Childhood certificate/diploma: Primary Education b. Diploma: Secondary Education c. Degree/Bachelors of Primary Education/Bachelors of Education d. Other 12. Do you read for pleasure/enjoyment outside of school? 13. Is there a place in your area/community/zone where you can borrow books? 14. Have you ever borrowed books from this place? 15. Do you have a copy of the official syllabi? 16. Who has observed you teach since the beginning of this school year? Check all that apply. How many times were you observed? a. My Head teacher/Teacher in charge i. b. A Zonal official of the Education Ministry i. c. A District official of the Education Ministry i. d. A Provincial official of the Education Ministry i. e. A Central official of the Education Ministry i. f. Other i. Please specify: ii. 17. How many times this year have you communicated in writing for official school business? 18. How many times this year have you communicated in writing with a relative or friend? 3. B FREQUENCY OF ACTIVITIES IN CLASS I will now ask some questions on reading instruction in grades 1 and 2. How often during 1 week do you do the following during literacy lessons? Orientation 19. Explain to learners the use and purpose of capital letters, small letters, and punctuation marks. a. 6 or more times per week b. 4-5 times per week c. 2-3 times per week Time to Learn Midline Impact Evaluation: Annex 5 79 d. 1 time per week e. Never f. Never because it is inappropriate for my grade g. Never because I haven’t had time/gotten to it in the curriculum 20. Ask learners to point out full stops (periods), commas, exclamation or question marks. a. 6 or more times per week b. 4-5 times per week c. 2-3 times per week d. 1 time per week e. Never f. Never because it is inappropriate for my grade g. Never because I haven’t had time/gotten to it in the curriculum Phonics/Phonemic Awareness 21. Demonstrate how to break a word into syllables. a. 6 or more times per week b. 4-5 times per week c. 2-3 times per week d. 1 time per week e. Never f. Never because it is inappropriate for my grade g. Never because I haven’t had time/gotten to it in the curriculum 22. Help learners identify the sound each letter (or combinations of letters) in the alphabet produces. a. 6 or more times per week b. 4-5 times per week c. 2-3 times per week d. 1 time per week e. Never f. Never because it is inappropriate for my grade g. Never because I haven’t had time/gotten to it in the curriculum 23. Help learners use their knowledge of sounds and letters to read new words. a. 6 or more times per week b. 4-5 times per week c. 2-3 times per week d. 1 time per week e. Never f. Never because it is inappropriate for my grade g. Never because I haven’t had time / gotten to it in the curriculum Time to Learn Midline Impact Evaluation: Annex 5 80 24. Ask learners to find a letter in a word. a. 6 or more times per week b. 4-5 times per week c. 2-3 times per week d. 1 time per week e. Never f. Never because it is inappropriate for my grade g. Never because I haven’t had time/gotten to it in the curriculum Writing 25. Demonstrate how to form letters. a. 6 or more times per week b. 4-5 times per week c. 2-3 times per week d. 1 time per week e. Never f. Never because it is inappropriate for my grade g. Never because I haven’t had time/gotten to it in the curriculum 26. Ask learners to copy text from the board. a. 6 or more times per week b. 4-5 times per week c. 2-3 times per week d. 1 time per week e. Never f. Never because it is inappropriate for my grade g. Never because I haven’t had time/gotten to it in the curriculum 27. Ask learners to complete writing assignments at home (as homework). a. 6 or more times per week b. 4-5 times per week c. 2-3 times per week d. 1 time per week e. Never f. Never because it is inappropriate for my grade g. Never because I haven’t had time/gotten to it in the curriculum Reading Fluency 28. Ask learners to read out loud for you or for classmates. a. 6 or more times per week Time to Learn Midline Impact Evaluation: Annex 5 81 b. 4-5 times per week c. 2-3 times per week d. 1 time per week e. Never f. Never because it is inappropriate for my grade g. Never because I haven’t had time/gotten to it in the curriculum 29. Ask learners to read at home (as homework). a. 6 or more times per week b. 4-5 times per week c. 2-3 times per week d. 1 time per week e. Never f. Never because it is inappropriate for my grade g. Never because I haven’t had time/gotten to it in the curriculum 30. Invite learners to read texts or stories that are NOT in their textbook. a. 6 or more times per week b. 4-5 times per week c. 2-3 times per week d. 1 time per week e. Never f. Never because it is inappropriate for my grade g. Never because I haven’t had time/gotten to it in the curriculum 31. Help learners memorize whole words by sight, without having to sound them out. a. 6 or more times per week b. 4-5 times per week c. 2-3 times per week d. 1 time per week e. Never f. Never because it is inappropriate for my grade g. Never because I haven’t had time/gotten to it in the curriculum Reading/Listening Comprehension 32. Ask learners to try to figure out the meaning of a new word by examining how it is used in a text or a sentence. a. 6 or more times per week b. 4-5 times per week c. 2-3 times per week d. 1 time per week Time to Learn Midline Impact Evaluation: Annex 5 82 e. Never f. Never because it is inappropriate for my grade g. Never because I haven’t had time/gotten to it in the curriculum 33. Ask learners to find a word in a sentence. a. 6 or more times per week b. 4-5 times per week c. 2-3 times per week d. 1 time per week e. Never f. Never because it is inappropriate for my grade g. Never because I haven’t had time/gotten to it in the curriculum Comprehension (listening or reading) 34. Ask learners to predict the content of a story by examining the title or have learners discuss with classmates what they know about the theme or subject of a text before reading it (pre-reading). a. 6 or more times per week b. 4-5 times per week c. 2-3 times per week d. 1 time per week e. Never f. Never because it is inappropriate for my grade g. Never because I haven’t had time/gotten to it in the curriculum 35. Ask learners to tell you what happened in the beginning, middle, or end of a story or text they have read. a. 6 or more times per week b. 4-5 times per week c. 2-3 times per week d. 1 time per week e. Never f. Never because it is inappropriate for my grade g. Never because I haven’t had time/gotten to it in the curriculum 36. Ask learners to predict what will happen next in a story. a. 6 or more times per week b. 4-5 times per week c. 2-3 times per week d. 1 time per week e. Never f. Never because it is inappropriate for my grade Time to Learn Midline Impact Evaluation: Annex 5 83 g. Never because I haven’t had time/gotten to it in the curriculum 37. Ask learners to identify what they liked about a story or a text. a. 6 or more times per week b. 4-5 times per week c. 2-3 times per week d. 1 time per week e. Never f. Never because it is inappropriate for my grade g. Never because I haven’t had time/gotten to it in the curriculum 38. Ask learners to identify whether there are any similarities between the events in a story and their own life experiences. a. 6 or more times per week b. 4-5 times per week c. 2-3 times per week d. 1 time per week e. Never f. Never because it is inappropriate for my grade g. Never because I haven’t had time/gotten to it in the curriculum 39. Read or tell stories to learners. a. 6 or more times per week b. 4-5 times per week c. 2-3 times per week d. 1 time per week e. Never f. Never because it is inappropriate for my grade g. Never because I haven’t had time/gotten to it in the curriculum 40. Have learners repeat after you the sentences of a text. a. 6 or more times per week b. 4-5 times per week c. 2-3 times per week d. 1 time per week e. Never f. Never because it is inappropriate for my grade g. Never because I haven’t had time/gotten to it in the curriculum 41. Repeat or practice the spelling of high frequency words. a. 6 or more times per week b. 4-5 times per week c. 2-3 times per week Time to Learn Midline Impact Evaluation: Annex 5 84 d. 1 time per week e. Never f. Never because it is inappropriate for my grade g. Never because I haven’t had time/gotten to it in the curriculum 3. C TEACHER PRACTICE The next set of questions is no longer about frequency. I will now ask some questions on attendance, lesson planning, and assessment in grades 1 and 2. 42. Do you have an attendance register for the class I observed? a. Yes (If YES, ask: May I see your class register?) b. No (If NO, skip to Q. 46) 43. Teacher can produce attendance register. a. Yes b. No 44. Did you take attendance for the class I observed today? a. Yes (If YES, skip to Q. 46) b. No 45. Why not? a. Forgot b. No book c. Don’t know d. Was distracted by your presence e. Other 46. When was the last time you took class attendance? a. This week b. Last week c. This month d. Last month e. Don’t remember 47. What do you do when a learner in your class is absent for several days? (mark all that apply) a. Notify the Head teacher, Deputy Head teacher or other school administrator b. Notify the PCSC Chair or other PCSC member c. Ask classmates about the learner d. Ask other teachers about the learner e. Send a note to the learner’s parents/guardian f. Visit the learner at their home g. Other 48. Do you have a scheduled time for teaching reading? Time to Learn Midline Impact Evaluation: Annex 5 85 a. Yes b. No (If NO, proceed to Q.54) ASK: Can you show me the timetable? Data collectors do not ask questions 49 to 51, but answer the questions individually. 49. Teacher can show timetable a. Yes b. No 50. Number of reading lessons scheduled per week: ____ 51. Amount of time scheduled per week for reading lessons in hours and minutes: ____:___. 52. How long does the reading lesson actually take? ____:___ (hrs.) 53. How often do you actually teach reading per week? _____ 54. Do you develop daily plans for reading lessons? a. Yes b. No (If NO, proceed to Q. 56) ASK: May I see the plans? Data collectors do not ask question 55, but answer the question individually. 55. Teacher can produce lesson plan or script corresponding to the lesson observed. a. Yes (If YES, proceed to Q.57) b. No 56. What is the main reason that you don’t have any lesson plan or script for reading for today’s lesson? (Check all that apply, then proceed to Q.58) a. Don’t need it/it is not useful b. Was not given or lost lesson plan or scripts c. Don’t know how to prepare a lesson plan for reading d. Takes too much time to prepare e. Other i. If other, specify: 57. What is the main reason you have a lesson plan or script for today’s reading lesson? a. I learned to write lessons at a training and found it useful b. Because a pre-scripted lesson was provided to me c. Because I was being observed d. I have lesson plans or scripts for all my lessons e. It’s a requirement of the school f. Other i. If other, specify: 58. What materials did you use to plan this lesson? (Indicate lesson plan teacher produced, which corresponds to the lesson observed.) a. Teacher’s guide Time to Learn Midline Impact Evaluation: Annex 5 86 b. National Literacy Framework c. Learners’ book d. Syllabus e. TTL training materials f. None g. Scripted lesson h. Other i. If other, specify: 59. What teaching approach do you use most often in your classroom to teach reading? a. New Break Through to Literacy b. Primary Literacy Program c. Read on Express d. Step into English e. Read on f. Other i. If other, specify: 60. Do you keep track of your learners’ progress in learning to read? a. Yes b. No (If NO, skip to Q. 65) 61. What formal methods do you use most often to assess your learners’ reading acquisition? (Mark all that apply) a. Use reading Continuous Assessment tool b. Give quiz or test c. 5th weekly/month(ly) assessment d. 10th weekly/term(ly) assessment e. No formal learner assessment 62. What informal methods do you use most often to assess your learners’ reading acquisition? (Mark all that apply) a. Ask learners questions about lesson b. Monitor learners as they work c. Observe learners in group activities d. Listen to individual learners read aloud e. Ask learners to tell about what they have just read f. Check learner’s exercise book or homework g. No informal learner assessment 63. How often do you track individual learner’s progress? a. Daily Time to Learn Midline Impact Evaluation: Annex 5 87 b. Weekly c. Monthly d. Termly/quarterly e. Yearly f. Never (If never, skip to 65) ASK: May I see your learner progress or learner record book for reading? Data collectors do not ask questions 64 to 66, but answer the questions individually. 64. The teacher… (Check all that apply) a. Keeps records of structured assessments (tests, exams) b. Keeps notes of informal learner c. Can produce learner progress or record book 65. (Only show if 63 is “never or 60 is answered with no”) What are the main reasons for not recording or tracking learner progress? (Check all that apply, then proceed to Q.68) a. No record book b. Don’t know how c. Takes too much time d. Not necessary e. Other____________________ 66. (Only show if “Can produce learner progress or record book” is selected in 64) The date of last entry for the learner progress or record book is: a. This week b. This month c. Last month d. More than 2 months ago 67. (Only show if “Never” is not the answer for 63) Where do you obtain the criteria you use to assess learner progress? (Mark all that apply) a. MESVTEE curriculum b. Specific reading program materials c. District or Zone education authorities d. School Head teacher e. Other teachers f. Develop them myself g. Other_____ 68. What teaching-learning materials do you have access to? a. Chalkboard b. Posters/Talking walls c. Textbooks d. Supplementary reading materials e. Flashcards Time to Learn Midline Impact Evaluation: Annex 5 88 f. Exercise books or slates g. Other_________ 69. Have you ever made any of your own teaching-learning materials? a. Yes b. No ASK: May I see the teaching material you have made yourself? Data collectors do not ask question 70, but answer it individually. 70. Teacher can produce self-made teaching-learning material. a. Yes b. No 3. D TEACHER ATTITUDES I am now going to read a few statements to you and ask you whether you agree or disagree with them. If you are not sure if you agree or disagree, please tell me that you ‘do not have an opinion’ on this statement. 3. E.1) Do you agree with the statements below? (Put an X in the appropriate column. Mark one column only.) Agree Disagree No opinion 71. All learners can learn to read. 72. All learners can learn to write. 73. Learners have a lot of difficulty learning to write. 74. It is very difficult for learners to learn to read. 75. Girls learn to read or write faster than boys. 76. Boys learn to read or write faster than girls. 77. If I had sufficient reading material in my classroom, I would give learners time each day to read materials of their own choice. 78. If a learner makes an error spelling a word for the first time, it’s not a major concern. 79. It is better to teach reading and writing as two separate subjects, so as not to confuse them. 80. It is better to teach reading and writing together because learning to write strengthens learners’ reading skills and vice versa. 81. A learner must learn to read before they can learn to write. Time to Learn Midline Impact Evaluation: Annex 5 89 3. E.1) Do you agree with the statements below? (Put an X in the appropriate column. Mark one column only.) Agree Disagree No opinion 82. Girls like to read. 83. Boys like to write. 84. It is important to give learners time each day to write freely on topics of their own choice. 85. It is important to correct all the errors in learners’ sentences. 86. Before having learners read a text for the first time, it is important to discuss what they know about the subject in the text. 87. Reading stories to learners helps develop their reading skills. 88. Learners must memorize a text before they can understand it. 89. Telling learners stories helps create interest in reading and learning to read. 90. Silent reading should be avoided because the teacher can’t check if learners are actually reading or reading correctly. 91. A learner who writes “well” is a learner who does not make any grammatical or spelling errors. 92. I have received adequate training on how to teach reading/writing. 93. I often have opportunities to talk to colleagues about how to teach reading/writing. 3. E TTL EXPOSURE I will now ask some questions on the types of support that are available for teachers at this school. 94. Have you received any in-service training on how to teach early grade reading or on leadership skills since 2012? Probe, if the respondent is not sure or answers question with no: In-service training is training received since starting service as a teacher and does not include pre-service training. This training could be called in￾service training, Grace Meetings, Teacher Learner Circles (TLC), Continuing Professional Development (CPD), workshops, Inset, or go by another name. We are not interested in the name of the training, only in the content. m. Yes (Go to 95) n. No (Skip to 98) Time to Learn Midline Impact Evaluation: Annex 5 90 95. Who conducted the training? Check all that apply. 96. What was the name or focus of the training? 97. In what year was the training? a. MESVTEE i. ii. b. Other government institution i. Please specify: ii. iii. c. Time to Learn Check all that apply. a. Quickstart At which level was the TTL training held? a. School____ b. Zone____ c. District____ d. Province____ e. National____ f. Do not remember b. Reading/module reading At which level was the TTL training held? a. School____ b. Zone____ c. District____ d. Province____ e. National____ f. Do not remember c. Writing/module writing At which level was the TTL training held? a. School____ b. Zone____ c. District____ d. Province____ e. National____ f. Do not remember d. Alphabet/module alphabet At which level was the TTL training held? Time to Learn Midline Impact Evaluation: Annex 5 91 a. School____ b. Zone____ c. District____ d. Province____ e. National____ f. Do not remember e. Assessment training, also known as CPD literacy training/CPD for community school teachers a. School____ b. Zone____ c. District____ d. Province____ e. National____ f. Do not remember f. Operational Guidelines for Community School (OGCS) orientation a. School____ b. Zone____ c. District____ d. Province____ e. National____ f. Do not remember g. PCSC training in supporting reading (at the household and community level) (could also be known as Parent Community Literacy Mobilization) a. School____ b. Zone____ c. District____ d. Province____ e. National____ f. Do not remember h. Educational Leadership and Management a. School____ b. Zone____ c. District____ d. Province____ Time to Learn Midline Impact Evaluation: Annex 5 92 e. National____ f. Do not remember i. Private institution i. Please specify: ii. iii. j. Nongovernmental organization iv. Please specify: v. vi. k. Other vii. Please specify: viii. ix. 3. F TEACHER CAPABILITIES (SKILLS) We are now finished with the interview. We would like you to understand what exercise we are doing with the learners during the EGRA so I will give you some examples. I am going to show you some letters and ask you how to pronounce them. 98. ASK: Please state the name of this letter. Show letter card #1: E e a. Teacher can state the name of the letter on letter card #1. a. Yes b. No 99. ASK: Please sound the letter. Show letter card #1: E e b. Teacher can pronounce the sound of the letter on letter card #1. a. Yes b. No 100. ASK: Please state the name of this letter. Show letter card #2: W w c. Teacher can state the name of the letter on letter card #2. a. Yes b. No 101. ASK: Please sound the letter. Show letter card #2: W w d. Teacher can pronounce the sound of the letter on letter card #2. a. Yes b. No 102. ASK: Please sound the two combined letters. Show letter card #3: sh e. Teacher can pronounce the sound of the digraph on letter card #3. a. Yes b. No 103. ASK: Please sound the two combined letters. Show letter card #4: mb f. Teacher can pronounce the sound of the two-consonant blend on letter card #4. a. Yes b. No Thank you for your participation! Do you have any additional questions about the interview? Time to Learn Midline Impact Evaluation: Annex 5 93 DISTRICT EDUCATION BOARD SECRETARY SELF￾ADMINISTERED SURVEY QUESTIONNAIRE3 Time to Learn Midline Evaluation MESVTEE Self-administered Survey Questionnaire INTRODUCTION This is a self-administered survey questionnaire for the office of the District Education Board Secretary (DEBS). The DEBS is provided with this survey during the initial courtesy call by the provincial TTL Midline Evaluation data collection team after all protocols are observed (including the informed consent procedure for the community schools to be sampled in that district). The DEBS is responsible for ensuring that his/her office completes the questionnaire and consults different officers for different questions. The data collection team will pick up the survey when leaving the district (typically 1-3 days after dropping the survey off). 3 This was tailored to list only those schools sampled in the relevant district. It is intended to be used as a template. Time to Learn (TTL) is a 5-year USAID-funded project working in collaboration with the Ministry of Education, Science, Vocational Training, and Early Education (MESVTEE) to improve reading in community schools in six provinces: Central, Copperbelt, Eastern, Lusaka, Muchinga, and Southern. This survey is part of the TTL midline evaluation to improve project activities and increase learner literacy by assessing how TTL activities are fostering learner literacy. This DEBS Office is selected for participation as community schools in this district are part of the midline evaluation sample. TTL is interested in learning how the MESVTEE is supporting community schools and what additional forms of support are most needed. This is an evaluation of the TTL project, not the DEBS Office. TTL will not be reporting on this office to anyone. Information collected through this survey will be used in a general way by aggregating data from all districts, and this office will not be identified in any reports. Your office’s participation is completely voluntary and you are free to stop the survey at any time or not to answer any questions. Doing so will not affect your relationship with TTL or the MESVTEE. If you choose to complete the survey the information will be very helpful for TTL to get a more complete picture of the support each community school receives, so it is our hope that your office will be able to assist the project in this regard. Instructions for use: Please answer every question in this survey in the next 3 days. The data collection team will collect it from your office upon the completion of data collection in this district. Time to Learn Midline Impact Evaluation: Annex 5 94 BACKGROUND Enter responses in Column B. The following questions are about each of the community schools listed under question 12. Please answer the questions about each school. SCHOOL NAME: 13. How many times has this DEBS Office visited this school in the current school year? Check the correct response. a. At least once since the beginning of the school year b. At least once per term c. At least once per month d. More often than once per month Column A Column B 1. Date 28. 2. Province 29. 3. District 30. 4. Name of DEBS officer 31. 5. Sex of DEBS officer 32. 6. Mobile phone number of DEBS officer: 33. 7. How many schools does this DEBS office support? 34. 8. How many government schools does this DEBS office support? 35. 9. How many community schools does this DEBS office support? 36. 10. How many private schools does this DEBS office support? 37. 11. How many grant-aided schools does this DEBS office support? 38. 12. Which of these community schools does your office support? Answer for all community schools in Column B. 39. Time to Learn Midline Impact Evaluation: Annex 5 95 14. For each type of support listed below, write the specific support provided to this school by this DEBS Office for the relevant year(s)? If no support was provided for any given year, write “none.” Type of Support 2012 2013 2014 a) Financial support/ grants to schools (please list types/amount of support under the relevant years) b) Trainings for Head teachers (please list trainings under the relevant years) c) Trainings for teachers (please list trainings under the relevant years) d) Trainings for Parent Community School Committees (please list trainings under the relevant years) e) Free basic materials (e.g., chalk, exercise books, pencils) (please list materials under the relevant years) f) Teaching and learning materials (please list materials under the relevant years) g) Monitoring visits (please list number of visits under the relevant years) h) Guidance or assistance on how to implement MESVTEE policy that affects community schools (please list guidance or assistance given under the relevant years) i) Other Support SCHOOL NAME: Time to Learn Midline Impact Evaluation: Annex 5 96 Time to Learn Project Plot Number 203B, Off Kudu Road Kabulonga, Lusaka Zambia