Impact Evaluation of USAID/South Africa Story Powered School Program - Baseline August 2018 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Dr. Alicia Menendez and Dr. Cally Ardington. i | Page USAID/SOUTH AFRICA STORY POWERED SCHOOL PROGRAM IMPACT EVALUATION: August 2018 Prepared for the United States Agency for International Development Under GS-10F-0033M/AID-OAA-M-13-00013 Cover photo: NORC Baseline Pilot. Photo credit: Alicia Menendez DISCLAIMER This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Dr. Alicia Menendez and Dr. Cally Ardington. ii | Page TABLE OF CONTENTS EXECUTIVE SUMMARY..................................................................................................................................................1 Project Background ......................................................................................................................................................1 Evaluation Methodology Overview...........................................................................................................................1 Findings............................................................................................................................................................................2 INTRODUCTION ............................................................................................................................................................4 Evaluation Purpose and Audience .............................................................................................................................7 EVALUATION QUESTIONS AND METHODOLOGY OVERVIEW..................................................................8 Evaluation Questions....................................................................................................................................................8 Methodology Overview...............................................................................................................................................9 Sample Selection and Experimental Design.......................................................................................... 9 Sample Size ........................................................................................................................................... 9 Baseline Instruments .......................................................................................................................... 10 BASELINE DATA COLLECTION...............................................................................................................................12 FINDINGS.........................................................................................................................................................................13 Sample Description.....................................................................................................................................................13 Language ........................................................................................................................................................................14 EGRA Scores................................................................................................................................................................14 Reading Attitudes and Behaviors.............................................................................................................................21 Home Environment ....................................................................................................................................................23 School Environment ...................................................................................................................................................25 Gender and Reading Skills, Attitudes And Behaviors.........................................................................................30 Teaching activities and reading fluency ..................................................................................................................32 Reading Resources and Reading Fluency...............................................................................................................33 Teachers’ Ability to Assess Learner Reading Skills.............................................................................................34 Characteristics of Learners Who Cannot Read One Word............................................................................35 Characteristics of Learners Who Read at least 25 CWPM..............................................................................36 Variation in reading Outcomes Across Schools..................................................................................................36 Determinants of EGRA Scores................................................................................................................................38 Determinants of English Reading Fluency..............................................................................................................39 Determinants of Reading Attitudes and Behaviors .............................................................................................39 iii | Page Balance ...........................................................................................................................................................................40 CONCLUSIONS .............................................................................................................................................................41 Limitations .....................................................................................................................................................................42 REFERENCES....................................................................................................................................................................43 ANNEXES Annex I: Evaluation Statement of Work ....................................................................................................................45 Annex II: Evaluation Methods and Limitations..........................................................................................................46 Annex III: Data Collection Instruments .....................................................................................................................54 Annex IV: Addition Analyses...................................................................................................................................... 134 Annex V: Disclosure of Any Conflicts of Interest................................................................................................ 163 iv | Page TABLE OF TABLES Table 1: Theory of change ........................................................................................................................................6 Table 2: Evaluation questions...................................................................................................................................8 Table 3: Baseline sample size by province – schools and learners................................................................10 Table 4: Teacher sample size by province and grade(s) taught......................................................................10 Table 5: Skills and corresponding subtasks included in baseline EGRA tool ..............................................11 Table 6: Data collection activities .........................................................................................................................12 Table 7: Age of learner sample by gender, grade and province.....................................................................13 Table 8: Home language of learners and teachers by province .....................................................................14 Table 9: Languages used by teacher in class, as reported by learners and teachers................................. 14 Table 10: Summary of EGRA scores by province and grade – home language ............................................15 Table 11: Percentage of learners attempting at least one comprehension question and percentage correct of comprehension questions attempted correct by province and grade .....................18 Table 12: Percentage of learners reaching ORF and constructed benchmarks by province and grade . 19 Table 13: Summary of English EGRA scores by province and grade ..............................................................20 Table 14: Percentage of Grade 4 learners reaching English benchmarks by province................................21 Table 15: Average learner characteristics by province ......................................................................................23 Table 16: Teacher characteristics by province.....................................................................................................26 Table 17: Access to reading materials at school..................................................................................................29 Table 18: Absenteeism by province ........................................................................................................................30 Table 19: Characteristics of learners scoring zero compared to those with ORF between 1 and 24, and those with ORF >=25 ......................................................................................................................36 v | Page TABLE OF FIGURES Figure 1: Percentage of learners at correct grade for age by grade, gender and province ......................13 Figure 2: Percentage of learners scoring zero on EGRA subtasks by province and grade.......................16 Figure 3: Distribution of correct words per minute for familiar word reading subtask by province and grade.............................................................................................................................................................17 Figure 4: Attitudes toward reading and stories...................................................................................................22 Figure 5: Attitude toward reading by grade.........................................................................................................22 Figure 6: Frequency of reading at home by grade ..............................................................................................23 Figure 7: Frequency of reading with someone at home....................................................................................25 Figure 8: Books other than schoolbooks to read at home ..............................................................................25 Figure 9: Department of Basic Education school quintile.................................................................................26 Figure 10: Frequency of teachers reading outside work requirements...........................................................27 Figure 11: What do teachers read outside work requirements?......................................................................27 Figure 12: Teacher’s practices...................................................................................................................................28 Figure 13: Use of readers in classrooms with readers........................................................................................29 Figure 14: Gender and reading skills........................................................................................................................30 Figure 15: Gender and reading attitudes – responses to the question “Do you like to read?” ................31 Figure 16: Gender and frequency of reading on your own at home................................................................31 Figure 17: Gender and frequency of reading with someone at home .............................................................32 Figure 18: Association between teaching practices and oral reading fluency ................................................33 Figure 19: Association between reading resources and oral reading fluency ................................................34 Figure 20 : Proportion of learners achieving CWPM thresholds by teachers’ report of the percentage of learners reading with fluency and comprehension............................................................................35 Figure 21: Distribution of percentage of learners in each school who are non-readers by province .....37 Figure 22: Distribution of percentage of learners in each school who can read at least 25 CWPM by province.......................................................................................................................................................37 Figure 23: Data selection process.............................................................................................................................47 vi | Page ACRONYMS DBE Department of Basic Education EC Eastern Cape EDR Evaluation Design Report EGRA Early Grade Reading Assessment ICC Intra-Cluster Correlation IE Impact Evaluation KZN KwaZulu-Natal LOLT Language of learning and teaching MDES Minimum Detectable Effect Size ORF Oral Reading Fluency RCT Randomized Control Trial RFE Reading for Enjoyment SPS Story Powered Schools: A South African Reading Revolution 1 | Page EXECUTIVE SUMMARY NORC at the University of Chicago, through the USAID Reading and Access Evaluation Contract, has been charged with conducting the external impact evaluation (IE) of Story Powered Schools: A South African Reading Revolution (SPS). The main purpose of the evaluation will be to assess the causal impact of SPS on reading outcomes of primary school children. This report focuses on the baseline findings of the evaluation. PROJECT BACKGROUND SPS aims to develop and sustain a culture of reading for enjoyment in 720 primary schools in the Eastern Cape (EC) and KwaZulu-Natal (KZN) provinces of South Africa. The program focuses on nurturing a love of reading, in mother tongue and English, to spark children’s potential and unlock their capacity to learn. The project is run by South African non-profit organization, Nal’ibali, in partnership with the United States Agency for International Development (USAID), the DG Murry Trust, and national, provincial and local Departments of Education. The following activities take place at each SPS school: • “Big 5” training – 5 teachers trained in reading for enjoyment (2 days). • Community training for 10 community volunteers in reading for enjoyment (1 day). • Providing five hanging libraries – bags with pockets to display the reading materials – and 15 editions per year of the bilingual Nal’ibali reading newspaper supplement. • Supporting schools to register reading clubs. • Weekly visits from Story Sparkers – Nal’ibali mentors. These are high school graduates from the community that receive training and support in the field by SPS Literacy Mentors and Provincial Coordinators. On their weekly visit, Story Sparkers assist schools in creating a culture of reading for enjoyment with activities scheduled before, during, and after the school day. • Holiday programs in June/July that include multiple reading activities for learners. • Events, campaigns, and competitions, which engage the broader school community. • Reading club showcases – competitions where reading clubs are adjudicated according to established criteria and where children from reading clubs share and perform their achievements. These SPS activities aim to increase: 1) awareness about the importance of reading for enjoyment to contribute to children’s literacy development, and learning; 2) motivation to read, write, and share stories; 3) positive behaviors and habits that support reading development; 4) access to reading materials; 5) children’s skills and confidence relating to reading, writing, and storytelling; and 6) parents’ and communities’ involvement in promoting reading. EVALUATION METHODOLOGY OVERVIEW The methodology for this evaluation is a randomized-controlled trial (RCT), with communities of schools randomly assigned to treatment and control groups. This data will be complemented with qualitative data from classroom observations in a sub-sample of treatment and control schools and with in-depth case studies in four treatment schools. A total of 360 schools were included in the baseline data collection that was conducted over two years – 2017 and 2018 – due to programmatic reasons. In each school selected for data collection, NORC randomly sampled one teacher from each of grades 2, 3, and 4. Learners were randomly selected from the classes of the sampled teachers. In total, NORC randomly selected 30 learners from each school (10 each from grades 2, 3, and 4, comprised of five girls and five boys whenever possible). 2 | Page The evaluation measures reading outcomes using subtasks of the Early Grade Reading Assessment (EGRA), a widely used tool to measure various aspects of reading proficiency. The EGRA assessments were conducted in mother tongue (isiXhosa or isiZulu) for all learners. English EGRA assessments were administered only to Grade 3 and 4 learners. NORC used an existing version of EGRA and added six receptive vocabulary items where the learner was asked to identify the correct picture to match a particular word. In addition to the EGRA tool, the baseline data collection included: (1) a short student background questionnaire administered immediately following the EGRA to every student sampled; (2) a teacher questionnaire administered to the teacher of each sampled class; and (3) a school inventory completed by the enumerators. FINDINGS Reading skills: Average scores in the EGRA subtasks are very low in both provinces. One in four learners are non-readers in mother tongue – they cannot read a single word from a short, grade￾level paragraph. There is clear progress from grade to grade, but by Grade 4, 13% of learners are still unable to read one word and the average oral reading fluency is 23 words per minute. In the table below, we show the proportion of learners that are not readers in mother tongue and average oral reading fluency for each grade, separately by province Oral Reading Fluency (ORF) Eastern Cape KwaZulu-Natal Grade 2 Grade 3 Grade 4 Grade 2 Grade 3 Grade 4 ORF = 0 41.9% 28.4% 15.1% 32.1% 19.5% 10.6% ORF average number of words 5.90 13.07 19.57 8.66 19.55 25.29 Excluding non-readers, the average oral reading fluency of the rest of the learners is, on average, 20 words per minute and 24% of them cannot correctly answer one comprehension question about the passage read. The very poor foundation in mother tongue literacy at the beginning of Grade 4 is particularly concerning, as learners transition to English as the language of learning and teaching (LOLT) in this grade. English EGRA results are similar those for mother tongue, with 32% of Grade 4 learners unable to answer one English comprehension question. Many factors are associated with learners’ reading performance: • On average, girls tend to out–perform boys. • Better socio-economic status, as proxied by household possessions and child height, is associated with better reading outcomes. • Learners who are behind for their age and those repeating the grade also have poorer reading skills. Attitudes and behavior: 90% of learners report they like to read. However, 20% of learners never read at home on their own and only one in four learners reads at home daily. Absenteeism is a major challenge for both learners and teachers. SPS activities are scheduled to take place in school and absenteeism could reduce the potential impact of the intervention. Learners’ reading resources: Access to reading resources is clearly an issue. Half of learners have no books other than schoolbooks to read at home. Only 15% of learners can take a library book home. 3 | Page While the vast majority of schools have class readers, learners typically have to share them, cannot choose their own reader, and are not allowed to take readers home. Teachers’ reading behavior: 60% read outside work requirements often or very often, but less than half are reading books (fiction or non-fiction). Only 47% of teachers have more than ten books in their homes. Classroom instruction practices: Teachers report using a range of activities in literacy lessons. Rote learning and chanting remains a dominant pedagogy with 47% of teachers saying that their learners repeat sentences they have said first on a daily basis. Playing games related to the literacy lesson is much less common with only 8% of teachers reporting this as a daily activity. There is clearly room to incorporate more activities focused on reading for enjoyment. Teachers’ assessment of their learners: On average, teachers report that 41-50% of their learners are reading with fluency and comprehension in mother tongue. Matching this to learners’ EGRA outcomes would imply a fluency of 19 words per minute, far from the levels needed to be able to comprehend the text read. Teachers are not only overly optimistic, their assessment of their own learners’ reading levels does not correlate highly with the measured EGRA outcomes. 4 | Page INTRODUCTION CONTEXT AND PROJECT BACKGROUND In 2016, almost 80 percent of South African Grade 4 learners fell below the lowest international level of reading proficiency, which means they could not locate explicit information or reproduce information from text. This effectively means they cannot read for meaning. Grade 4 learners living in remote rural areas or townships are at a particular disadvantage; they have lower reading scores than learners in other locations.1 The ability to read for meaning by Grade 4 is considered crucial by education experts, since students are expected to utilize their reading skills to learn other subjects from Grade 4 onward. To tackle this challenge, the SPS project aims to develop and sustain a culture of reading for enjoyment in 720 primary schools in the Eastern Cape and KwaZulu-Natal provinces of South Africa. The SPS program is part of the broader Nal'ibali (isiXhosa for “here’s the story”) campaign, which is a national reading-for-enjoyment campaign to spark children’s potential by making reading and storytelling part of their daily lives. The SPS project focuses on nurturing a love of reading, in mother tongue and English, to spark children’s potential and unlock their capacity to learn. The project is run by South African non-profit organization Nal’ibali in partnership with USAID, the DG Murry Trust, and national, provincial, and local Departments of Education. The campaign is based on research on the link between reading for pleasure and improved education outcomes, and identifies the necessary conditions to support children’s literacy development.2 The degree to which children acquire language skills and become motivated, habitual readers, is a strong predictor of future academic success, educational attainment, employment, and income.3 To support children’s development as readers, the SPS project seeks to create the following conditions: • Increase awareness that reading for enjoyment is important and powerful, and knowledge of how to do it; • Create more opportunities for children to read, write, and share stories – alone and with others; • Inspire, equip, and encourage adults to be reading role models who share stories with children, encourage them to read, and model that reading matters; and • Increase access to quality reading material, especially in children’s mother tongues. The SPS project seeks to give children new opportunities to experience books and stories; to create the conditions where children’s literacy development can flourish; and to equip, inspire, and nurture teachers on how motivation, confidence, and linking reading and writing to children’s lived experiences can support curriculum objectives, accelerate children’s literacy development, and enable school success. Ultimately, it wants schools to embrace and take ownership of a strong culture of reading for enjoyment. 1 PIRLS Literacy 2016: South African Highlights Report http://www.up.ac.za/media/shared/164/ZP_Files/pirls￾literacy-2016-hl-report-3.zp136320.pdf 2 For example, Krashen, S. (2004). The power of reading: insights from the research. Libraries Unlimited: Westport; Evans, M.D.R. et al. (2010) Family scholarly culture and educational success: books and schooling in 27 nations. Research in Social Stratification and Mobility 28: 171-197; United Kingdom Department of Education (2012). Research evidence for reading for pleasure (available from www.gov.uk). 3 Needu (2013). NEEDU reading study 2013: the state of reading in Grade 5 in selected rural primary schools. National Education Evaluation and Development Unit. Available at https://nicspaull.files.wordpress.com/2014/12/needu-reading-study-2013_28oct14.pdf 5 | Page The following activities take place at each SPS school: • “Big 5” training – 5 teachers4 trained in reading for enjoyment Nal’ibali approach. • Community training for 10 community volunteers in reading for enjoyment Nal’ibali approach. • Providing five hanging libraries – large hanging bags with book pockets to display the reading materials and 15 editions per year of the bilingual Nal’ibali reading supplement, a bilingual newspaper supplement with stories, literacy activities, featuring people working with Nal’ibali, reading tips and support to inspire and guide caregivers, teachers and others to make reading and storytelling meaningful, enjoyable and accessible. • Supporting schools to register reading clubs. • Weekly school visits from Story Sparkers – Nal’ibali mentors. Story Sparkers are young high school graduates from the community in which the school is located5. They are recruited and employed by Nal’ibali. They receive training and are supported in the field by SPS Literacy Mentors and Provincial Coordinators. On their weekly visit, Story Sparkers assist schools in creating a culture of reading for enjoyment with activities scheduled before, during and after the school day. Typically, their weekly activities include assisting in setting up and running reading clubs, running reading for enjoyment activities in school assembly, demonstrating creative use of reading resources, and demonstrating how to effectively use the mandated “Drop Everything and Read” (DEAR)6 periods. • Holiday programs in June/July that include multiple reading activities for learners while they are on school holidays. • Coordinating and supporting events, campaigns, and competitions which engage the broader school community. These began with World Read Aloud Day in February and continue with various drives throughout the year. • Reading club showcases – a competition endorsed by the Department of Basic Education (DBE) where reading clubs are adjudicated according to established criteria and where children from reading clubs share and perform their achievements – taking place at multiple levels (school, circuit, and district). Through these activities, Nal’ibali expects each school to: • Support Story Sparkers by identifying appropriate space for literacy activities – library, empty classroom, etc. – and enabling Story Sparkers to carry out their daily plans rather than asking them to substitute teach or fill in for missing teachers. • Establish at least 3 reading clubs, which may be run Story Sparkers, teachers and/or community volunteers. • Actively implement what has been learned in training – for example, by consistently observing the mandatory DEAR reading period, and using classroom time to give children meaningful, personally satisfying opportunities to read, write, and hear stories. • Organize and participate in competitions, including an annual reading club showcase, to motivate children and teachers. • Participate in national Nal’ibali flagship events, including World Read Aloud Day and the Story Bosso storytelling competition. 4 In 2017, Nal’ibali trained 3 teachers and 2 community volunteers per school at the “Big 5” training. It was found that a critical mass of trained teachers is needed, so in 2018 they shifted to train 5 teachers per school. 5 Because story sparkers work with a number of schools which may cut across different communities, they may be community member of one school but not the others. 6 The DEAR period is part of the DBE’s Read to Lead campaign. All schools should observe at least 30 minutes per week of mandatory reading. 6 | Page • Engage parents and the community around the importance of reading, by sharing information about reading for enjoyment at parents’ meetings, sending reading materials home with children, inviting parents and community members to Nal’ibali events, and involving them in Nal’ibali reading clubs and activities as volunteers. Ultimately, the changes the SPS project expects to see at schools include: • Increased awareness that reading for enjoyment (RFE) is important and contributes to children’s literacy development, and knowledge/understanding of how RFE helps children learn; • Shifts in attitudes: children who are more excited/motivated to read, write and share stories; teachers who view reading for enjoyment as central to learning; parents who are more invested in reading and education; • Positive behaviors and habits that support reading development, such as children attending reading clubs, reading voluntarily in their free time, choosing what they want to read, taking books and supplements home, and reading a wide variety of books and stories; • Improved access to reading materials, with reading materials accessible to children in schools, and children taking books and supplements home; • Improvements in children’s skills and confidence relating to reading, writing and storytelling; • Parents and communities more involved in promoting reading. The program’s theory of change is as follows: Table 1: Theory of change INPUTS Training of teachers, officials, community volunteers Multilingual reading materials Parents and community engagement around the importance of reading At least 3 reading clubs per school, run by Story Sparkers, teacher or volunteers and supported by Nal’ibali staff Weekly visits to schools by Story Sparkers Events, campaigns and competitions Holiday programs OUTPUTS Schools invited to sign up for SPS Reading clubs operating at the school Reading materials available OUTCOMES Learners more motivated to read More time devoted reading for enjoyment Increased support of teachers, parents, and community members around reading Use of reading materials Learners attending reading clubs IMPACT Reading skills improved Reading comprehension improved Vocabulary improved 7 | Page EVALUATION PURPOSE AND AUDIENCE NORC at the University of Chicago, through the USAID Reading and Access Evaluation Contract, has been charged with conducting the IE of SPS. The main purpose of the evaluation will be to assess the causal impact of SPS on reading outcomes of primary school children. The evaluation will measure reading outcomes using subtasks of the EGRA, a widely used tool to measure various aspects of reading proficiency. Additionally, the IE will assess SPS’s impact on other intermediate outcomes such as participation of children in reading activities at school and in the community/home, time spent reading independently, children’s attitude toward reading, and teacher attitudes and behavior toward RFE and storytelling. To measure these outcomes, the evaluation will use survey instruments to collect data at the level of the learner, teacher, and school. The IE will be implemented in the provinces of Eastern Cape and Kwazulu￾Natal, and focus on isiXhosa and isiZulu languages, as well as English. It will assess the impact of SPS over a period of two years. The findings, conclusions and recommendations generated by this evaluation and research study will contribute toward understanding how activities devoted to RFE contribute to create a culture of reading and to improve reading, comprehension, and vocabulary. The audiences for this evaluation and research study comprise USAID Operating Units, notably, USAID/South Africa, the Africa Bureau, and the E3/Education Office. Other important audiences are the Government of South Africa, primarily, the Department of Basic Education, donors committed to improving learners reading skills around the world, and the implementing partner, Nal’ibali. 8 | Page EVALUATION QUESTIONS AND METHODOLOGY OVERVIEW EVALUATION QUESTIONS In April 2016, representatives from USAID/South Africa, USAID/E3/Education Office, Nal’ibali SPS team, and NORC met in South Africa to jointly discuss the SPS program and the evaluation objectives. The evaluation approach and evaluation questions in Table 2 below are an outcome of those discussions Table 2: Evaluation questions # Outcomes Questions 1 Student reading skills Q1. What is the impact of the SPS project on learners’: A. reading and writing skills in their mother tongue (including letter recognition, word recognition, fluency, comprehension, vocabulary, phonemic awareness, orientation to print)? B. reading and writing skills in English? 2 Student Motivation and Behavior Q2. What is the impact of the SPS intervention on: A. the time or frequency that students spend reading independently? B. students’ motivation to seek out, listen to, read, write, and tell stories? 3 Teachers Attitude and Behavior Q3. What is the impact of the SPS intervention on: A. teachers’ attitudes towards a. reading for enjoyment in the classroom? b. story telling in the classroom? c. reading clubs? B. teachers’ language and literacy instruction practices in the classroom (both those trained by Nal’ibali and those not trained by Nal’ibali but teaching in SPS schools)? 4 Student reading skills Q4. What is the impact of attending a Nal’ibali reading club (in a SPS) on learners’: A. reading and writing skills in their mother tongue? B. reading and writing skills in English? 5 Student Motivation and Behavior Q5. What is the impact of attending a Nal’ibali reading club (in a SPS) on: A. the time or frequency that students spend reading independently? B. Students’ motivation to seek out, listen to, read, write, and tell stories? 6 Student reading skills Q6. What is the association between teacher practices and learners’: A. reading and writing skills in their mother tongue? B. reading and writing skills in English? 7 Student Motivation and Behavior Q7. What is the association between teacher practices and: A. the time or frequency that students spend reading independently? B. students’ motivation to seek out, listen to, read, write, and tell stories? 8 Program take￾up (participation in reading activities and use of reading materials/ trainings) Q8. What is take-up of and adherence to the SPS intervention? This will include a range of indicators. For example: A. number and characteristics of invited schools signing up for SPS B. number and characteristics of teachers trained C. number of community members trained D. implementation of reading for enjoyment sessions before, during, and after school day E. number of reading clubs operating at the school, number of children in the reading clubs F. attendance at reading clubs G. other events (competitions, etc.) H. use of mini libraries 9 | Page METHODOLOGY OVERVIEW The methodology for this evaluation is a randomized-controlled trial, with communities of schools randomly assigned to treatment and control groups. This data will be complemented with qualitative data from classroom observations in a sub-sample of treatment and control schools and with in-depth case studies in four treatment schools. Data will be collected in two rounds: a baseline prior to program implementation, and an endline two to three years later. Ideally, we will collect student-level panel data by administering assessments to the same individual students at both baseline and endline. The main advantage of a student-level panel is that it would allow for student-level fixed effects, which can improve the power and precision of the analysis if baseline outcomes are highly correlated with endline outcomes. However, absenteeism and attrition may make it difficult to re-interview the same students at endline. While we will continue to explore this possibility, the alternative would be a repeated cross-section at the school level, in which one random sample of students would be drawn from each school at baseline and a second random sample at endline. Estimating the effect of attending Nal’ibali reading clubs in the SPS schools is somewhat challenging. Participating in the clubs is not a random event, but rather a choice made by each learner. Learners that choose to attend reading clubs and learners that do not cannot be directly compared as they might differ, not only in terms of their club participation but also in taste for reading, reading skills, family support, etc. Therefore, the effect of attending a Nal’ibali reading club in a SPS school will be estimated using statistical techniques (propensity score matching) to construct a comparison group of learners from control schools that is very similar to the group of learners who attend reading clubs. More details on data sources and data analysis methods can be found in Annex II. Sample Selection and Experimental Design Community randomization and school selection: While SPS is a school based intervention, the Nal’ibali approach involves the community around the school in a number of ways. Randomization was, therefore, done at the community level, with all schools in a community assigned to either treatment or control. The design considerations and process of creating appropriate communities of schools is described in detail in Annex II. Within each community, between two and four schools were randomly selected to be part of the evaluation. Teacher selection: In each school selected for data collection, we randomly sampled one teacher each from grades 2, 3 and 4. Where the school was a multi-grade schools with learners from different grades taught by the same teacher, we selected one teacher from each grade combination in the Grade 2 to 4 range. Learner selection: Learners were randomly selected from the classes of the sampled teachers. We randomly selected 30 learners from each school (10 each from grades 2, 3, and 4, comprising of five girls and five boys whenever possible). Learners were only selected if they were present at the school on the day that the field team arrived to complete the assessments7. Sample Size Due to programmatic considerations, the baseline was conducted over two years with 192 schools included in each of cohort I (2017) and cohort II (2018). See Annex II for more details. The planned and achieved sample sizes for the baseline are summarized in Table 3. 7 The absenteeism rate is approximately 11% in all grades, and it is the same in treatment and control schools. 10 | Page Table 3: Baseline sample size by province – schools and learners Eastern Cape KwaZulu-Natal Planned Completed Planned Completed Schools 192 172 192 188 Learners: Grade 2 learners 1920 1676 1920 1780 Grade 3 learners 1920 1671 1920 1784 Grade 4 learners 1920 1685 1920 1808 The number of schools in the sample is slightly lower than planned, particularly in the Eastern Cape. This is largely due to language issues as several selected schools turned out to have Sesotho rather than isiXhosa as the medium of instruction in the foundation phase. Table 1 in Annex II provides full details of schools that were not included in the final sample. The number of learners in the sample is also slightly lower than planned, partly due to the schools that were not included in the study, but also due to very low enrollment in some schools. There were also a handful of schools that did not offer all grades from 2 to 4. Although our sample is slightly smaller than anticipated, Nal'ibali’s change in the number of schools per community for the second cohort (2018) allowed us to include more communities in our sample and strengthened the statistical power of our study.8 Overall, this means we have a sufficient number of schools and learners in our sample to conduct the IE as planned. Table 4 below summarizes the planned and achieved sample sizes for the teacher sample. The planned sample included one teacher each from grades 2, 3 and 4. Overall, 32% of schools in the sample had at least some learners of different grades taught in the same classroom. In these schools, there were not necessarily three different teachers across the three grades. The planned sample of 192 teachers for each grade in each province was based on the planned sample of 384 schools. Of the 360 schools that were included in the final sample, we have at least one teacher for each grade in 95% of cases. There are a small number of learners (4%) for whom we do not have a matching teacher. This is due to the teacher for their grade being unavailable for the interview. Table 4: Teacher sample size by province and grade(s) taught Eastern Cape KwaZulu-Natal Planned Completed Planned Completed Grade 2 192 163 192 180 Grade 3 192 167 192 180 Grade 4 192 164 192 171 Individual teachers 495 506 Percentage of learners with a matched teacher 96.0% 96.0% Baseline Instruments Early Grade Reading Assessment: The evaluation measures reading outcomes using subtasks of the Early Grade Reading Assessment (EGRA), a widely used tool to measure various aspects of reading proficiency. The skills tested with the EGRA tool and their corresponding subtask are indicated in Table 5. EGRA assessments were conducted in mother tongue (isiXhosa or isiZulu) for all learners. English EGRA assessments were administered only to Grade 3 and 4 learners. NORC used an existing version of EGRA, created by Lili Pretorius, a professor of linguistics at the University of South Africa (UNISA) 8 See Annex II for full details. 11 | Page for the Zenex Foundation in 2015. NORC also added six receptive vocabulary items where the learner was asked to identify the correct picture to match a particular word. Table 5: Skills and corresponding subtasks included in baseline EGRA tool Skill Subtask & Measurement isiXhosa/isiZulu English Orientation to print Awareness of text direction, where to start reading, how to read down a page Grade 2 Phonetic awareness Number of letters sounds identified in 60 seconds Grades 2, 3 & 4 Grades 3 & 4 Phonological awareness Identify and manipulate phonemes (starting and ending sounds of words, segmenting words) Grades 2, 3 & 4 Grade 4 Word recognition Familiar word reading, number of correct words read in 60 seconds Grades 2, 3 & 4 Grades 3 & 4 Oral reading fluency Oral passage reading, number of words fluently read (with accuracy) from a reading passage in 60 seconds Grades 2, 3 & 4 (different passages for each grade) Grade 4 Reading comprehension Number of questions answered correctly about the passage read aloud by the student Grades 2, 3 & 4 (different questions for each grade) Grade 4 Receptive vocabulary Identify correct picture to match word Grades 2, 3 & 4 Grade 4 Additional instruments: In addition to the EGRA tool, the baseline data collection included: (1) a short student background questionnaire administered immediately following the EGRA to every student sampled, (2) a teacher questionnaire administered to the teacher of each sampled class, and (3) a school inventory completed by the enumerators. NORC drew on questionnaires that had previously been used in South African schools in designing the learner and teacher questionnaires. Draft instruments were shared with USAID/South Africa, USAID/3E/Education Office and Nal'ibali for comments. 12 | Page BASELINE DATA COLLECTION The baseline data collection for the first cohort of schools took place in the first term of the 2017 school year (February/March). Baseline data collection for the second cohort was completed in February and March 2018. NORC plans to re-assess and interview the same learners in the final term of the following school year. Cohort 1 will be revisited in October 2018 and cohort II in October 2019. NORC will also re-interview the same teachers. NORC worked closely with subcontractors Pearson (2017 baseline) and Ikapadata (2018 baseline) to conduct all data collection activities. This study and all associated data collection activities gained approval from both the NORC IRB and the University of Cape Town Ethics Committee. NORC has worked in close collaboration with the Department of Basic Education and the Provincial Education Departments to conduct the data collection necessary for the evaluation. NORC provided ethics training to all team members from the local data collection firm and all team members committed to comply with child protection policies. There were no significant challenges identified during baseline data collection. The data collection schedule for this evaluation is summarized in Table 6. Table 6: Data collection activities Survey Type Instruments Date Baseline (quantitative) EGRA assessment & Learner interview Teacher interview & School inventory Cohort 1: Feb/Mar 2017 Cohort II: Feb/Mar 2018 Midline (qualitative) Classroom observation & Case studies Learner assessments Aug/Sep 2018 Endline (quantitative) EGRA assessment & Learner interview Teacher interview Cohort 1: Aug/Sep 2018 Cohort II: Aug/Sep 2019 13 | Page FINDINGS SAMPLE DESCRIPTION By design, the learner sample comprises roughly equal numbers of boys and girls, and equal numbers of grade 2, 3, and 4 learners. Table 7 shows the average age by grade, gender, and province. If learners start school on time and advance one year per additional year of age, they should be 7 years old in Grade 2, 8 years old in Grade 3 and 9 years old in Grade 4. The averages in Table 7 suggest that a fair number of learners are older than the appropriate age for their grade, with boys being slightly older than girls and learners in the Eastern Cape being slightly older than those in KwaZulu-Natal. Table 7: Age of learner sample by gender, grade and province Overall Eastern Cape KwaZulu-Natal Boys Girls Boys Girls Grade 2 7.67 7.28 7.45 7.20 7.67 Grade 3 8.75 8.27 8.55 8.16 8.75 Grade 4 9.86 9.31 9.71 9.25 9.86 Observations 2,536 2,469 2,723 2,649 2,536 Figure 1 shows, separately by province, grade, and gender, the percentage of learners who are in the correct grade for their age, the percentage who are one year behind and the percentage who are two or more years behind. Overall, 41% of learners are not in the correct grade for their age, with 12% being two or more years behind. This is due to a combination of delayed enrollment and grade repetition. In every grade, girls are significantly more likely to be on-track than boys, and a higher percentage of learners are in the correct grade for their age in KwaZulu-Natal than in the Eastern Cape. Overall, one in five boys in the Eastern Cape is at least two years behind. Figure 1: Percentage of learners at correct grade for age by grade, gender and province 14 | Page LANGUAGE The South African government follows a policy of additive multilingualism, beginning with mother tongue in the first three years of primary school and then transitioning to English as the language of learning and teaching (LOLT) in Grade 4. The LOLT in early grades is isiXhosa for all sample schools in the Eastern Cape and isiZulu for those in KwaZulu-Natal. Table 8 below shows, separated by province, the mother tongue or home language of the learners and teachers in the sample. Almost all (97%) Eastern Cape learners report isiXhosa as the language they speak most frequently at home and a similarly high percentage of teachers report isiXhosa as their mother tongue. The alignment between the LOLT of the school in the early grades and learner and teacher home language is even closer in KwaZulu-Natal with 98% of learners reporting isiZulu as their home language9. Table 8: Home language of learners and teachers by province Eastern Cape KwaZulu-Natal Learners Teachers Learners Teachers isiXhosa 96.9% 96.2% 0.6% 0.2% isiZulu 1.1% 2.3% 98.4% 99.9% Other 2.0% 1.5% 0.1% 0.8% Observations 4754 475 5151 497 Teachers reported overwhelmingly (96%) that they used the LOLT most of the time in the classroom. In addition to asking what language they used most of the time, NORC for all languages that were used for teaching. We also asked the learners what languages their teachers used in the classroom. The results are presented in Table 9. In all grades, the vast majority of teachers report using a combination of English and mother tongue. One in ten Grade 2 teachers use only mother tongue. This decreases to 5% in Grade 3, and by Grade 4 all teachers report using either English only or a combination of English and mother tongue. Learners are much more likely than teachers to report that their teacher only uses mother tongue which may indicate some social desirability bias in the teachers’ responses aligning more closely with official policy. Nevertheless, the transition from mother tongue to English as the medium of instruction is clearly apparent in comparing learner answers across the grades. Table 9: Languages used by teacher in class, as reported by learners and teachers Learners Teachers Grade 2 Grade 3 Grade 4 Grade 2 Grade 3 Grade 4 isiXhosa/isiZulu 59.8% 28.2% 15.2% 10.2% 4.5% 0% English 3.7% 2.7% 9.5% 0% 0% 5.9% Both 36.5% 69.1% 75.3% 89.8% 95.5% 94.1% EGRA SCORES The different subtasks included in the assessment are described in Table 5. Our focus is on the main impact indicators – correct familiar words per minute (CWPM), oral reading fluency (ORF), and reading comprehension (RC) – but we also analyze the rest of the subtasks assessed – Correct Letter Sounds per Minute (CLSPM), Orientation to Print, Phonemic Awareness, and Receptive Vocabulary. The 9 Given this alignment, for the remainder of this report mother tongue and home language will be considered synonymous with isiXhosa in the Eastern Cape and isiZulu in KwaZulu-Natal. 15 | Page aggregated results of the different EGRA subtasks in mother tongue are presented in Table 10 by grade and province.10 Table 10: Summary of EGRA scores by province and grade – home language Eastern Cape KwaZulu-Natal Grade 2 Grade 3 Grade 4 Grade 2 Grade 3 Grade 4 Print orientation [max=10] Mean 6.94 -- -- 7.90 -- -- Std dev. 2.46 -- -- 2.16 -- -- Correct Letter Sounds per Minute (CLSPM) Mean 19.81 31.24 33.90 15.07 18.63 19.18 Std dev. 15.59 18.88 20.56 13.88 15.75 15.40 Correct Words Per Minute (CWPM) Mean 6.59 16.00 22.97 8.99 18.02 24.09 Std dev. 8.45 12.46 14.00 8.85 11.69 12.12 Oral Reading Fluency (ORF) Mean 5.90 13.07 19.57 8.66 19.55 25.29 Std dev. 8.42 12.14 13.84 9.99 14.58 15.21 Read comprehension [max Gr2=10, max Gr3=7, Gr4=8] Mean 1.19 0.83 0.98 1.31 2.24 1.34 Std dev. 1.45 1.21 1.06 1.48 1.81 1.15 Phonemic awareness [max=8] Mean 2.66 3.66 4.12 2.74 3.46 4.15 Std dev. 1.68 1.97 2.08 2.09 2.34 2.32 Vocabulary [max=6] Mean 2.93 4.30 4.83 3.68 4.86 5.30 Std dev. 2.10 1.94 1.69 2.15 1.76 1.43 On average, Eastern Cape Grade 2 learners can correctly answer 6.9 of the 10 print orientation questions, 2.7 of the 8 phonemic awareness questions, and 2.9 of the 6 vocabulary questions. They can identify 19.8 letter sounds (CLSPM), correctly read 6.6 familiar words per minute (CWPM), and read 5.9 words per minute from a paragraph (ORF). This improves to 34 CLSPM, 23 CWPM, and an ORF of 19.6 for Grade 4 learners. Grade 4 learners are able to answer just over half of the phonemic awareness 10 For the full distribution of scores please see Table A1 and Figures A1 to A4 in Annex IV. For the correlation between scores on the different EGRA subtasks please see Table A2 in Annex IV. For reliability statistics please see Table A3 in Annex IV. For means excluding learners scoring zero, please see Table A4 in Annex IV. 16 | Page questions (4.1 out of 8) and 4.83 of the 6 vocabulary questions. In KwaZulu-Natal, the progression from Grade 2 to Grade 4 across the subtasks is similar. Figure 2: Percentage of learners scoring zero on EGRA subtasks by province and grade Figure 2 shows the percentage of learners who scored zero on each subtask by province and grade. Focusing on Grade 2 learners in KwaZulu-Natal, we see that 2% are unable to answer any of the print orientation questions correctly, 18% are unable to correctly identify even one letter sound, 25% cannot identify even one familiar word, 32% could not read even one word of connected text, 18% scored zero 17 | Page on the phonemic awareness questions and 10% could not correctly identify any of the vocabulary items. The percentage of learners scoring zero decreases as the grade level increases across all subtasks. Nevertheless, there are a substantial number of Grade 4 learners who still cannot correctly identify even one word (15% in the Eastern Cape and 11% in KwaZulu-Natal). The progression between grades on the full distribution of CWPM can be seen clearly in Figure 3 which shows the percentage of learners in each grade reading at each level, separated by province. We see a sharp decline in the percentage of non-readers across the grades. Focusing on the children who can identify at least one word, the upwards shift in CWPM across grades is also evident. If we combine all the categories spanning 20 CWPM and higher, we see that the percentage of learners in the Eastern Cape reading at least 20 CWPM ranges from 11% in grade 2, to 41% in grade 3, and up to 64% in grade 4. The corresponding figures for KwaZulu-Natal are 15%, 51%, and 71%. Figure 3: Distribution of correct words per minute for familiar word reading subtask by province and grade This improvement across grades is an indicator of real, albeit slow, progression, rather than being an artifact of dropout or grade retention. Schooling in South Africa is compulsory until Grade 9 or age 15, whichever comes earlier. Figures from nationally representative household surveys consistently show that enrollment is almost universal (in the order of 98-99%) among children aged 7 to 10. Grade 18 | Page repetition is fairly common, but rates are fairly similar across the grades in our sample. Around 7% of the learners in our sample are repeating the grade. Absenteeism is also consistent across grades with 89% of enrolled learners present on the day of the assessments. Each grade was administered a different and grade-appropriate oral reading passage and corresponding comprehension questions. The oral reading fluency rates are therefore not strictly comparable across grades as each a passage was designed for each grade level. This is also the case with the comprehension questions, with the Grade 2 subtask including more literal questions and the higher grades including more inferential questions. It is also important in analyzing the comprehension results to bear in mind that learners are only asked the questions corresponding to the part of the passage that they were able to read. The cut points for each question were different across the grades. Within a grade, those who were asked more questions had to have read further in the passage and therefore have higher ORF. Looking back to Table 10, we see that, on average, Grade 2 learners in the Eastern Cape answered 1.2 comprehension questions out of 10 correctly. Grade 3 and 4 learners in the Eastern Cape, correctly answered 0.8 and 1 comprehension question on average, respectively. However, not all learners read far enough in the passage to attempt any of the comprehension questions. Table 11 shows the percentage of learners attempting at least one comprehension question for each grade and province. Only 56% of Grade 2 learners in the Eastern Cape attempted any of the comprehension questions. Table 11 also shows the percentage of attempted questions that were correctly answered. Of the comprehension questions actually attempted, Grade 2 learners in the Eastern Cape were able to answer 65% correctly. The percentage of attempted questions that were correctly answered were 47% for Grade 3 and 52% for Grade 4. The average number of correctly answered questions in KwaZulu-Natal was 1.3, 2.2, and 1.3 for Grade 2, 3, and 4 respectively (Table 10). Of comprehension questions actually attempted in KwaZulu-Natal, learners answered 60%, 66%, and 57% correctly in Grades 2, 3, and 4, respectively (Table 11). Table 11: Percentage of learners attempting at least one comprehension question and percentage correct of comprehension questions attempted correct by province and grade Eastern Cape KwaZulu-Natal Grade 2 Grade 3 Grade 4 Grade 2 Grade 3 Grade 4 Percentage of learners attempting at least one comprehension question 55.9% 68.2% 81.9% 72.9% 84.7% 87.7% Percentage correct of comprehension questions attempted 65.1% 46.5% 51.8% 59.7% 66.4% 56.6% In South Africa, there are no official reading norms or benchmarks for English or for the nine South African indigenous languages. Neither the DBE nor USAID have proposed or endorsed particular reading targets. In the past years, several methodologies to create benchmarks using the EGRA subtasks have been proposed for different countries and contexts (see for example, Spaull et at. (2018), Room to Read (2018), RTI International (2017)). In a recent study, Spaull et al. (2018) argue that since comprehension is the ultimate goal of reading, it is logical to benchmark to comprehension outcomes. Their empirical investigation “suggests that there are minimum thresholds of accuracy and oral reading fluency in each language, below which it is virtually impossible to read for meaning.” They propose a method for empirically establishing a minimum fluency threshold and a minimum comprehension threshold. Following their methodology,11 we identify a minimum fluency threshold of 20 words per minute in isiXhosa and 23 words per minute in isiZulu. The minimum comprehension thresholds are 38 11 USAID is not proposing that this methodology should be the one to follow, nor that the resulting benchmarks should be the reading targets. 19 | Page words per minute and 35 words per minute in isiXhosa and isiZulu, respectively. See Figure A2 and the accompanying discussion in Annex IV for more details on the identification of these benchmarks. Table 12 shows the percentage of learners reaching these minimum fluency and minimum comprehension thresholds by grade and province. We also show the proportion of learners with ORF greater than 0, 20, 30, 40, and 50. By Grade 4, just over half of the learners in both provinces are reaching the minimum fluency threshold, but only 7% of those in the Eastern Cape and 29% of those in KwaZulu-Natal are reaching the minimum comprehension threshold. Table 12: Percentage of learners reaching ORF and constructed benchmarks by province and grade Eastern Cape KwaZulu-Natal Grade 2 Grade 3 Grade 4 Grade 2 Grade 3 Grade 4 ORF > 0 58.1% 71.6% 84.9% 67.9% 80.5% 89.4% ORF >= 20 8.7% 29.3% 52.4% 17.9% 51.2% 67.2% ORF >= 30 1.6% 11.2% 25.0% 3.5% 26.4% 41.7% ORF >= 40 0.3% 2.4% 5.8% 0.6% 9.9% 16.1% ORF >= 50 0.1% 0.7% 2.3% 0.2% 2.2% 5.0% Minimum fluency threshold (EC=20, KZN=23) 8.7% 29.3% 52.4% 11.5% 41.5% 56.5% Minimum comprehension threshold (EC=38, KZN=35) 0.6% 3.3% 7.3% 1.4% 16.3% 28.8% Before turning to the English EGRA results, it is worth briefly considering the typological and orthographic differences between English and the home languages in our study. isiXhosa and isiZulu are part of the Nguni branch of the family of Southern African Bantu languages. They are agglutinating languages with a conjunctive orthography. In equivalent texts, English therefore presents readers with larger numbers of shorter words. A translation of the 82 word isiXhosa Grade 4 oral reading passage yields a 160 word English passage. The average word length is 7.5 letters for the isiXhosa passage and 4.2 for the English passage. Grade 3 and 4 learners were assessed in English. Grade 3 learners attempted the letter sound and familiar word subtasks in English. Grade 4 learners were assessed on all English EGRA subtasks. A summary of English EGRA scores is presented in Table 13. We will focus on the Grade 4 learners for whom the LOLT is English. On average, Grade 4 learners read 40 and 35 English words per minute in KwaZulu-Natal and the Eastern Cape respectively. This is in contrast to 25 words per minute in isiZulu and 20 words per minute in isiXhosa. In both provinces, around one in ten Grade 4 learners is unable to read even a single English word. Amongst those who are able to read at least one English word, the average words per minute is 44 in KwaZulu-Natal and 39 in the Eastern Cape, but 25% were unable to correctly answer any comprehension questions (see Table A5 in Annex IV, for means excluding zero-scores). 20 | Page Table 13: Summary of English EGRA scores by province and grade Eastern Cape KwaZulu-Natal Grade 3 Grade 4 Grade 3 Grade 4 Correct Letter Sounds per Minute (CLSPM) Mean 26.04 29.60 18.52 20.07 Std dev. 17.57 19.81 15.20 15.59 % scoring zero 12.5% 14.0% 16.2% 15.6% Correct Words per Minute (CWPM) Mean 11.05 21.03 13.85 23.61 Std dev. 12.75 16.86 13.55 16.83 % scoring zero 31.2% 14.9% 21.6% 10.4% Oral Reading Fluency (ORF) Mean 34.01 39.67 Std dev. 25.79 27.65 % scoring zero 12.8% 10.0% % correct of attempted 68.0% 73.7% Reading comprehension [max=10] Mean 1.22 1.90 Std dev. 1.59 2.01 % scoring zero 39.0% 21.8% % correct of attempted 26.1% 36.1% Phonemic awareness [max=8] Mean 2.35 3.05 Std dev. 2.39 2.46 % scoring zero 37.0% 24.9% Vocabulary [ max=6] Mean 3.27 3.67 Std dev. 1.91 1.84 % scoring zero 10.6% 8.0% Table 14 below shows comprehension levels and the proportion of Grade 4 learners reaching various fluency benchmarks. Draper and Spaull (2015) suggest a benchmark of 90-100 English words per minute for Grade 5 second language learners, and consider learners reading below 40 words a minute as non￾readers. Over half of the Grade 4 learners in our sample in both provinces would be classified as non￾readers by this definition. Following Spaull et al. (2018), we establish a minimum fluency threshold of 46 words per minute and a minimum comprehension threshold of 82 words per minute. In Grade 4, only 5% of Eastern Cape learners and 9% of KwaZulu-Natal learners are reaching the minimum comprehension threshold, and over 60% in both provinces cannot even read at the minimum fluency threshold. 21 | Page Table 14: Percentage of Grade 4 learners reaching English benchmarks by province Eastern Cape KwaZulu-Natal Comprehension categories: Unable to read one word 12.4% 8.5% 0% comprehension 28.8% 14.8% 10-29% comprehension 46.4% 52.3% 30-59% comprehension 9.3% 17.3% 60%+ comprehension 3.2% 7.2% ORF = 0 14.8% 11.5% ORF = 1-19 19.7% 16.6% ORF = 20 - 39 24.4% 23.9% ORF = 40 - 59 26.2% 25.8% ORF = 60 - 79 10.3% 12.6% ORF = 80 - 99 3.6% 6.5% ORF >= 100 1.7% 3.5% Minimum fluency threshold = 46 32.0% 39.6% Minimum comprehension threshold = 83 4.9% 9.1% READING ATTITUDES AND BEHAVIORS In addition to reading skills, this IE is interested in the impact of the intervention on learners’ reading attitudes and behaviors. Learners were shown a range of five smiley faces from very happy to very sad and then asked to point out which face most closely corresponded to how they felt about nine statements. The statements asked how much they liked to do a range of activities from playing to reading out loud in class. Figure 4 shows the distribution of answers. Overall, 69% of learners selected the very happy face when asked if they liked to read and a further 20% selected the happy face. Responses were less positive for listening to stories and telling stories with 50% and 44% selecting the very happy face, respectively. In addition to asking whether they liked to read, NORC asked learners whether they liked to read on their own at home, on their own at school, and out loud in class. Responses were very similar to the simple question asking whether they liked to read. 22 | Page Figure 4: Attitudes toward reading and stories Figure 5 shows the attitude toward reading by grade. There is a slight increase in the percentage of learners who select the very happy face as grade increases. Figure 5: Attitude toward reading by grade Overall, three-quarters of learners say that they read at home on their own at least once a week, but only one in four learners reads daily. One in five learners never reads on their own at home. Figure 6 shows a slight increase in reading at home between Grade 2 and the next two grades. 23 | Page Figure 6: Frequency of reading at home by grade HOME ENVIRONMENT The learner questionnaire included questions about the learners’ home environment. Some of the key variables are summarized in Table 15 (additional variables are summarized in Table A8 in Annex IV). As is typical across South Africa, less than three in ten learners live with their father and one in four live with neither parent. The vast majority of learners have electricity at home but are unlikely to have a flush toilet or a computer. Learners in KwaZulu-Natal tend to come from households with more possessions (e.g., refrigerator, car, television) than those in the Eastern Cape. The height of every learner was measured and compared against the World Health Organization’s international child growth standard.12 Children are considered stunted if their height-for-age standardized score is two or more standard deviations below the median child from the international “healthy” reference population. Around 17% of learners in the Eastern Cape and 12% of those in KwaZulu-Natal are classified as stunted. A number of families appear to be relying heavily on the school feeding program, with 14% of learners in the Eastern Cape and 19% in KwaZulu-Natal reporting that they did not eat before coming to school. There is some exposure to English in the home. Only 15% of learners ever speak English at home, but 54% of learners in the Eastern Cape and 69% in KwaZulu-Natal watch TV or listen to the radio in English. Table 15: Average learner characteristics by province Eastern Cape KwaZulu-Natal Lives with mother 68% 72% 12 The height of an individual depends on genetic potential, but also on the socioeconomic and epidemiological circumstances faced from conception until early adulthood (Deaton 2007, Case and Paxson 2008, Strauss and Thomas 2008). Children’s height is considered a good indicator of underlying health, as it reflects cumulative linear growth, and deficiencies show the effects of past or chronic inadequacies in nutrition or exposure to disease. In our data, stunted children are more likely to come from households with fewer assets and books. 24 | Page Eastern Cape KwaZulu-Natal Lives with father 31% 34% Lives with both parents 27% 30% Electricity 84% 85% Radio 52% 72% Television 69% 76% Computer 11% 17% Refrigerator 62% 77% Toilet 10% 19% Mobile phone 94% 94% Bicycle 26% 28% Car 38% 48% Count of household possessions (max=8) 3.53 4.30 Stunted (height-for-age z-score less than -2) 17% 12% Did not eat before coming to school 14% 19% Ever speak English at home 15% 15% Ever watch TV/listen to radio in English 54% 68% Figure 7 shows how frequently learners read with someone at home. Just over a third of learners never read with someone at home. Around half of learners have someone read with them at home at least twice a week and over two-thirds of learners have someone at home who tells them stories. However, access to reading materials in the home is very limited, with 58% of learners in the Eastern Cape and 44% of learners in KwaZulu-Natal reporting that they have no books aside from schoolbooks to read at home (Figure 8). Only 13% and 20% of learners have more than 10 books at home in the Eastern Cape and KwaZulu-Natal respectively. 25 | Page Figure 7: Frequency of reading with someone at home Figure 8: Books other than schoolbooks to read at home SCHOOL ENVIRONMENT School characteristics The Department of Basic Education has classified schools into socio-economic quintiles. The poorest (quintile 1) schools receive the largest allocation of government funding per learner, while quintile 5 schools receive the smallest. Schools in the bottom three quintiles are no-fee schools and may not 26 | Page charge school fees. The majority of schools in the Eastern Cape (70%) are classified as quintile 1 schools, the lowest socio-economic quintile of schools in the country (Figure 9). A further 26% are quintile two schools. In line with socio-economic differences across the two provinces, the proportion of schools in KwaZulu-Natal in quintile one (38%) is lower than that in the Eastern Cape. Table A9 in Annex IV summarizes the basic characteristics of the sample schools as captured by the school inventory instrument. Figure 9: Department of Basic Education school quintile Teachers Table 16 below summarizes key teacher characteristics by province (further variables from the teacher questionnaire are summarized in Table A10 in Annex IV). The vast majority of teachers in both provinces are female. Teachers have, on average, 17 years of teaching experience and almost all have some professional qualification. In KwaZulu-Natal, 47% of teachers have a degree and 46% have a diploma or advanced diploma. Qualification levels in the Eastern Cape are slightly lower, with 36% of teachers having a degree and 43% having a diploma or advanced diploma. Table 16: Teacher characteristics by province Eastern Cape KwaZulu-Natal Female 94% 93% Years of teaching experience 17.7 16.6 Highest professional qualification in teaching: Diploma 31.5% 28.4% Advanced diploma 12.4% 17.8% Degree 35.6% 46.%6 Advanced certificate 18.7% 6.1% None 1.8% 1.1% The teacher questionnaire included a number of questions about the teachers’ own reading habits. Figure 10 shows how often teachers read outside of work requirements. Just over half of the teachers report reading very often or often. Among those that do read, the most common reading materials are newspapers (69%), the Bible and religious texts (56%), and magazines (51%) (Figure 11). Only half report reading any kind of book (fiction or non-fiction). Around 11% of teachers in the Eastern Cape and 6% in 27 | Page KwaZulu-Natal have no books to read in their home, and only 16% of all teachers have more than 25 books at home. Figure 10: Frequency of teachers reading outside work requirements Figure 11: What do teachers read outside work requirements? Figure 12 summarizes the literacy and language teaching practices for teachers in our sample. Teachers were asked how frequently a range of activities with learners took place in their class. The prevalence of rote learning is clear with almost half of teachers reporting that the whole class repeats sentences that they say first on a daily basis. Only 1% of teachers reported that they never use chanting. While almost 28 | Page all teachers report reading a story to learners at least once a week, using games to teach literacy was much less common with around a third of teachers reporting that they never did such activities. Figure 12: Teacher’s practices Access to reading materials Stark differences between the two provinces in the access to reading resources are evident in Table 17. Bookshelves or small libraries in the classroom are very common in KwaZulu-Natal (81%) but only present in just over a third (38%) of Eastern Cape classrooms. While only two in five KwaZulu-Natal schools (43%) have a library, they are twice as likely to have one as Eastern Cape schools (22%). In schools with libraries, learners are also more likely to be allowed to take books home in KwaZulu-Natal (52%) than in the Eastern Cape (39%). This means that only 8% of Eastern Cape learners and 23% of KwaZulu-Natal learners can take a library book home. 29 | Page Table 17: Access to reading materials at school Eastern Cape KwaZulu-Natal Bookshelf or small library in the classroom for learners to use 38.1% 81.3% School has library or mobile library 21.5% 43.3% For schools with library, learners are allowed to take books home 39.2% 52.1% Library has appropriate books 65.7% 77.5% Although the vast majority of teachers report that their learners have readers to read from in class (87% in Eastern Cape and 95% in KwaZulu-Natal), in two-thirds of classrooms the learners have to share readers (Figure 13). Just less than a quarter of teachers allow learners to choose their own readers. Learners are also mostly not allowed to take readers home. In classes with readers, 31% learners in the Eastern Cape and 15% in KwaZulu-Natal are allowed to take the readers home. Figure 13: Use of readers in classrooms with readers Absenteeism Learner absenteeism is a serious challenge in both provinces, with 35% of learners in the Eastern Cape and 42% of learners in KwaZulu-Natal reporting that they were absent at least one day in the previous week (Table 18). Learners also report very high rates of teacher absenteeism with around half of learners reporting that their teacher was absent at least one day in the previous week. Interestingly teachers report much lower rates of absenteeism, 25% in the Eastern Cape and 30% in KwaZulu-Natal. Even these rates, however, suggest that a lot of teaching time is lost over the school year. 30 | Page Table 18: Absenteeism by province Eastern Cape KwaZulu-Natal Learner absent in last week 34.7% 41.5% Teacher absent in last week (learner report) 48.6% 54.0% Teacher absent in last week (teacher report) 25.2% 29.8% GENDER AND READING SKILLS, ATTITUDES AND BEHAVIORS Girls tend to outperform boys in their reading skills, attitudes, and behaviors. On average, girls read around 6 words per minute faster than boys and are significantly less likely than boys to not be able to read one word (16% versus 30%). Figure 14 shows the distribution of oral reading frequency by gender and province. Figure 14: Gender and reading skills Girls are also more likely to report that they really like to read (Figure 15), read on their own at home more often (Figure 16), and someone reads with them at home (Figure 17). 31 | Page Figure 15: Gender and reading attitudes – responses to the question “Do you like to read?” Figure 16: Gender and frequency of reading on your own at home 32 | Page Figure 17: Gender and frequency of reading with someone at home TEACHING ACTIVITIES AND READING FLUENCY This section examines the association between ORF and teachers’ reported practices in their literacy lessons. For each teaching activity shown in Figure 18 below, we compare the ORF of learners in classes where the activity occurs daily against learners in classes where the activity occurs less frequently. We make comparisons within grade as both teaching practices and oral reading fluency change with grade. For each activity, Figure 18 shows how many more (or less) words learners can correctly read in a minute when they do that activity daily as compared to those who do the activity less frequently. The asterisk indicates whether the difference is statistically significant, with ** representing the 1% level and * the 5% level. 33 | Page Figure 18: Association between teaching practices and oral reading fluency In the Eastern Cape, daily practice of almost all of the activities is associated with significantly higher ORF. For example, learners who play games in relation to the literacy/reading lesson can, on average, identify 2.6 more words per minute than other learners in their grade. In KwaZulu-Natal, only four of the activities appear to be associated with better reading outcomes (learners retelling a story, learners repeating sentences that the teacher said first, learners read aloud individually to the teacher or other learners, learners assigned to do reading on their own during school time). It is important to note that these associations do not imply causality from teaching practices to better reading outcomes; reverse causality could be present. That is, it could be the case that teachers adapt their teaching activities to the reading levels of their learners. READING RESOURCES AND READING FLUENCY We also investigate the association between access to various reading resources and ORF. Figure 19 shows how many more words learners are able to read, on average, when they have access to various reading resources. In the Eastern Cape, learners in schools with a library are reading, on average, 2.4 words faster per minute than those in schools without a library. Being allowed to take a reader home is also associated with higher ORF. In KwaZulu-Natal, there does not appear to be any relationship between these reading resources and reading levels. 34 | Page Figure 19: Association between reading resources and oral reading fluency TEACHERS’ ABILITY TO ASSESS LEARNER READING SKILLS Teachers were asked what percentage of their class could read with fluency and comprehension in both mother tongue and in English. Overall, 43% of teachers thought at least half of their class was reading with fluency and comprehension in the mother tongue. Although there are no defined fluency benchmarks for African languages in South Africa, the teachers appear to overestimate their learners’ reading skills by any standard. Not only are teachers underestimating the severity of the reading crisis in their own classrooms, there does not appear to be a strong relationship between teachers’ perceptions and learners’ measured skills. For each Grade 4 teacher, we calculated the proportion of learners that were able to read at least 25 CWPM and the proportion that were able to read at least 40 CWPM, both in mother tongue. We then asked the teacher to estimate of the proportion of his/her class could read with fluency and comprehension in mother tongue. Figure 20 compares actual learner performance on the EGRA assessment against teachers’ perceptions. For each level of teachers’ perception, we show the proportion of learners reaching the 25 CWPM threshold and 40 CWPM threshold. Among teachers that believe 80% to 100% of learners are reading with fluency and comprehension, only 63% and 13% are able to read at least 25 CWPM and 40 CWPM respectively. In general, learner performance tends to improve with teachers’ estimations, but the association between teachers’ perceptions and learner skill is not particularly strong. For example, there is a low degree of discrimination in the 21%-60% range. 35 | Page Figure 20 : Proportion of learners achieving CWPM thresholds by teachers’ report of the percentage of learners reading with fluency and comprehension CHARACTERISTICS OF LEARNERS WHO CANNOT READ ONE WORD As evident in Table 19, there are a substantial number of learners who are unable to correctly identify one word from a passage of connected text in their mother tongue. Overall, 24% of learners scored zero on the oral reading subtask. Table 19 shows the characteristics of these learners (low performing) compared to the 49% of learners who are able to read between one and 24 CWPM (middle performing) and the 27% learners who are able to read at least 25 CWPM (high performing). In a sample that is split evenly between girls and boys, 63% of those who cannot read one word are boys. Compared to either the middle or the top performing groups, learners scoring zero are significantly more likely to be old for their grade, and stunted. They live in households with fewer possessions and fewer books. They are less likely to read with someone at home. They are more likely to have been absent in the past week and repeating their grade at school. 36 | Page Table 19: Characteristics of learners scoring zero compared to those with ORF between 1 and 24, and those with ORF >=25 ORF=0 ORF = 1-24 ORF >= 25 Test for difference in means (p-value) Low Middle High Low vs High Middle vs High Low vs Middle Female 37.5% 46.7% 62.9% 0.000 0.000 0.000 Age for grade: on track 50.1% 58.0% 66.7% 0.000 0.000 0.000 one year behind 31.1% 31.0% 25.3% 0.000 0.000 0.918 two or more years behind 18.8% 11.0% 8.0% 0.000 0.000 0.000 Stunted 18.6% 13.8% 11.5% 0.000 0.001 0.000 Count of household possessions 3.46 3.67 3.82 0.000 0.000 0.000 Learner absent 39.8% 38.9% 36.0% 0.001 0.002 0.472 No books in household 61.7% 50.7% 41.0% 0.000 0.000 0.000 Learner is repeating the grade 10.8% 6.0% 4.7% 0.000 0.078 0.000 Someone reads with the learner at home 60.8% 66.9% 67.1% 0.000 0.974 0.000 CHARACTERISTICS OF LEARNERS WHO READ AT LEAST 25 CWPM We now consider the characteristics of learners whose reading skills are better than average. We focus on those learners who are able to read at least 25 CWPM. Looking at Table 19, we see that all the characteristics that are associated with not being able to read at all are inversely associated with better reading outcomes. For example, the percentage of females in the low, middle and high performing groups is 38%, 47%, and 63% respectively. Two thirds of those in the high performing group are in the correct grade for their age compared to only half of those who score zero. Learners able to read at least 25 CWPM are less likely to be stunted, come from households with more possessions, and are less likely to have been absent in the past week. Six in ten learners who score zero have no books to read in their home compared to four in ten learners who can read at least 25 CWPM. VARIATION IN READING OUTCOMES ACROSS SCHOOLS Overall, 28% of learners in the Eastern Cape and 21% of learners in KwaZulu-Natal are unable to correctly identify one word in the passage of connected text. We calculate the percentage of non￾readers in each school to investigate how much diversity there is in poor reading outcomes across schools. If non-readers were spread evenly across schools, then we would expect 28% of learners in each school in the Eastern Cape to be non-readers. Similarly, 21% of learners in each KwaZulu-Natal school would be expected to be non-readers. Figure 21 depicts the distribution of the percentage of leaners in each school who are non-readers separately by province. The diversity across schools in the prevalence of non-readers is clear with the percentage of non-readers in schools ranging from 0% to 60%. In just under a quarter (24%) of schools in KwaZulu-Natal, all learners are able to identify at least one word. The corresponding figure for Eastern Cape schools is 15%. At the other end, over half the learners are non-readers in 7% of Eastern Cape schools and 2% of KwaZulu-Natal schools. 37 | Page Figure 21: Distribution of percentage of learners in each school who are non-readers by province Figure 22 shows analogous distributions for the percentage of learners in each school who are able to read at least 25 CWPM. In the Eastern Cape, the vast majority (71%) of schools have between 1% and 20% of learners able to read at least 25 CWPM and in 12% of schools, no learners are reaching this fluency level. In KwaZulu-Natal there is more diversity across schools. Figure 22: Distribution of percentage of learners in each school who can read at least 25 CWPM by province 38 | Page DETERMINANTS OF EGRA SCORES In addition to the descriptive analysis presented above, it is possible to use the baseline data to look into how student characteristics, household context, school, and teacher characteristics correlate with observed differences across EGRA scores. In this section, we present results for a series of statistical processes (regressions) for estimating the relationships among available variables and EGRA subtask scores. A large number of variables could be related to EGRA scores. We experimented with a range of specifications that include student characteristics and behavior, household characteristics, proxies for socio-economic status of the household, teacher characteristics and classroom practices, and school characteristics. In particular, we have included: 1. Student’ characteristics: age, sex, grade, learner height-for-age category 2. Home environment: count of household possessions, no books in the household, learner reads with someone at home 3. Teacher characteristics: years of experience, education degree, teacher has more than 100 books at home, teacher absent in the last week 4. School characteristics: has library, has computer lab Regression analysis results for ORF and an indicator for learners not being able to read one word of connected text are included in Table A11 in Annex IV. Here we summarize the results. Eastern Cape: Being a girl has a positive correlation across scores when compared to boys. For instance, when all else is equal, girls correctly identify 5 more words from a paragraph than boys and are 10 percentage points less likely to score zero on this task. This result is consistent with previous findings from South Africa and many other countries (Gambell, T and D. Hunter (1999), Riordan (1999), Mullis et al., 2012, Zuze and Reddy, 2014, Taylor 2004, among many others). Within grade, scores increase with age to a point and then begin to decrease which is consistent with poorer outcomes for learners who are behind in school. Learners who are classified as stunted read 1.2 words per minute slower than other learners and are 6 percentage points more likely to score zero. Learners with no books other than schoolbooks to read in the home correctly identify 2 less words per minute than learners with books in the home. In general, teacher characteristics other than qualification are not significantly associated with learners’ average ORF. Years of experience is associated with better performance at the bottom end of the distribution, with learners who have more experienced teachers slightly less likely to score zero. Having a library and a computer laboratory are both associated with higher fluency of around 2 more words per minute. KwaZulu-Natal: Similar to the Eastern Cape, being a girl has a positive correlation across scores when compared to boys. Learners from homes with more household possessions and books other than schoolbooks to read tend to have better reading performance than those with lower socio-economic status and less reading resources. In KwaZulu-Natal, both learner absenteeism and teacher absenteeism are significantly associated with poorer outcomes. There is no correlation with other teacher characteristics. In general, having a library or computer laboratory in the school does not appear to be significantly associated with learner performance. 39 | Page DETERMINANTS OF ENGLISH READING FLUENCY For our Grade 4 learners, we investigate whether the correlates of English reading fluency are similar to those for mother tongue. We run regressions similar to those for mother tongue EGRA scores but include two proxies of exposure to English at home, specifically an indicator that the learner ever speaks English at home and an indicator that the learner ever watches TV or listens to the radio in English. The full regression results for English ORF are presented in Table A12 in Annex IV. Eastern Cape: Similar to the findings for mother tongue, girls are reading, on average, 9 more correct words per minute than boys, and stunted learners read 5 words slower per minute than their taller peers. Being absent in the past week is associated with lower ORF. Learners with no books other than schoolbooks to read in the home correctly identify 5 less words per minute than learners with books in the home. Though teachers’ qualifications and experience are not significantly associated with learners’ English ORF, learners with teachers who have at least 100 books at home read, on average, 7 more correct words per minute than those whose teachers have fewer books. Having a library in the school is also associated with higher fluency of around 7 words per minute. Exposure to English at home tends to have a positive correlation with English reading fluency. For instance, when all else equal, learners who ever speak English at home read, on average, 15 more correct English words per minute compared to those who never speak English at home. Watching TV or listening to the radio in English is associated with a higher fluency of around 4 words per minute. KwaZulu-Natal: Once again, girls tend to outperform boys with a higher English reading fluency of around 13 words per minute. Learner absenteeism is associated with lower English ORF. Learners from homes with more household possessions have better reading performance than those from homes with less assets. Similar to the Eastern Cape, English ORF is significantly associated with our proxies for exposure to English in the home. Ever speaking English at home is associated with a higher fluency of around 15 words per minute and learners who report that they consume English TV or radio read an average of 15 more words per minute. DETERMINANTS OF READING ATTITUDES AND BEHAVIORS We also investigate the correlates of reading behaviors and attitudes using regressions similar to those described for the EGRA scores. The results for regressions on an indicator for reading at home on your own daily and an indicator for liking reading very much are shown in Table A13 in Annex IV. Here we summarize the findings. Eastern Cape: Girls and learners from homes with more household possessions are more likely to report reading at home daily and liking to read very much. Learners with no books in the home are significantly less likely to read at home daily. Having someone who reads with the learner at home is significantly correlated with learners reporting higher affinity with reading. In general, teacher and school characteristics do not appear to be associated with these reading behaviors and attitudes. KwaZulu-Natal: Girls and learners from homes with more possessions and more books are more likely to read on their own daily. Learner absenteeism is associated with a lower probability of daily reading. Having someone who reads with the learner at home is significantly associated with more frequent home reading and a more favorable attitude towards reading. 40 | Page BALANCE The purpose of a sample balance check exercise is to verify that respondent characteristics, environmental context, and outcome measures of the study sample are similar across treatment groups at baseline. Demonstrating similarity across randomly assigned treatment groups at the start of the study establishes credibility that the untreated group will, indeed, be a viable counterfactual to the treated group at endline. It should be mentioned, however, that differences across groups do not necessary hurt the credibility of the counterfactual and are statistically expected to occur.13 Sample balance checks are not a verification of the randomization process itself; rather, they can verify that the implementation of the process was sound. This can be especially important for studies where randomization is conducted in the field. Given that our random assignment was carried out using statistical software and respected in the field, there is no reason to expect biases. Despite this, we conducted balance checks. The full details of the significance test results are presented in Tables A14 and A15 in Annex IV. As expected, the sample balance checks of the survey data produced few indicators that show differences between the treatment and control groups. As stated earlier, some differences are expected and a natural outcome of a rigorously implemented randomization process. For the EGRA data, we tested a total of 26 variables for balance across the treatment and control arms, disaggregated by language group. This amounted to a total of 52 significance tests, since each variable was tested twice (two languages). Of the 52 significance tests run, 2 (roughly 4%) showed differences between the treatment and control groups, at the 5% significance level. Tests using data from the learner and teacher questionnaires only found 3 differences between the treatment and control average out of 60 tests. Furthermore, corrections for multiple testing using conventional approaches (Bonferroni, Holm-Bonferroni and others) eliminate any significant difference between groups. We also calculated the absolute effect size difference between group means for each variable. All effect size differences were under the 0.25 threshold. Summarizing, the randomization process had no challenges, the sample is properly balanced, and no changes to the evaluation approach are needed. 13 Bruhn M. and D. McKenzie, “In Pursuit of Balance: Randomization in Practice in Development Field Experiments”, American Economic Journal: Applied Economics, 2009, vol. 1, issue 4, 200-232 41 | Page CONCLUSIONS In this report, we present the findings from the baseline collected for the IE of the SPS intervention, which included EGRA assessments, learner interviews, teacher interviews, and a school inventory. The baseline data collection did not face any significant challenges. Average scores in the EGRA subtasks are very low in both provinces. One in four learners are non￾readers in their mother tongue – they cannot read a single word from a short, grade-level paragraph. Excluding non-readers, the average ORF of the rest of the learners is 20 words per minute and 24% of them cannot correctly answer one comprehension question about the passage read. Though there is clear progress from grade to grade, by Grade 4 13% of learners are still unable to read one word and the average oral reading fluency is only 23 words per minute. The very poor foundation in mother tongue literacy at the beginning of Grade 4 is particularly concerning, as learners transition to English as the LOLT in this grade. English EGRA results are similar those for mother tongue, with 32% of Grade 4 learners unable to answer one English comprehension question. On average, girls tend to outperform boys, and better socio-economic status, as proxied by household possessions and child height, is associated with better reading outcomes. Learners who are behind for their age and those repeating the grade also have poorer reading skills. Attitudes to reading are encouraging with 90% of learners selecting the very happy or the happy face when asked whether they like to read. However, 20% of learners never read at home on their own and only one in four learners reads at home daily. Absenteeism is a major challenge for both learners and teachers. Since SPS activities are scheduled to take place in school, absenteeism could reduce the potential impact of the intervention. Access to reading resources is clearly an issue, particularly in the Eastern Cape where only 8% of learners can take a library book home. The vast majority of schools have class readers but learners typically have to share, cannot choose their own reader, and are not allowed to take them home. Fifty percent of learners have no books other than schoolbooks to read at home. The resources that the SPS intervention brings have the potential to make a large difference to the quantity and quality of reading materials available to teachers and learners. Focusing on teachers themselves as readers, 60% read outside of work requirements often or very often, but less than half are reading books (fiction or non-fiction). Only 47% of teachers have more than ten books in their homes. Teachers report using a range of activities in their literacy lessons. Rote learning and chanting remains a dominant pedagogy with 47% of teachers saying that their learners repeat sentences they have said first on a daily basis. Playing games related to the literacy lesson is much less common with only 8% of teacher report this as a daily activity. There is clearly room to incorporate more activities focused on reading for enjoyment. Teachers, on average, feel that 41-50% of their learners are reading with fluency and comprehension in mother tongue. Matching this to learners’ EGRA outcomes would imply a fluency benchmark of 19 words per minute, far from the levels needed to be able to comprehend the text read. Teachers are not only overly optimistic, their assessment of their own learners’ reading levels does not correlate highly with the measured EGRA outcomes. 42 | Page LIMITATIONS The Evaluation Team encountered some limitations inherent to the design of this evaluation. We list some of the more relevant limitations below: Representativeness of the sample. The sample is representative of the areas where SPS is currently working and corresponds to two of the eleven official languages in South Africa – isiZulu and isiXhosa. Therefore, results are not generalizable at the national level or to other geographical areas or other languages. The sample is representative of the children that were in school during the assessment, therefore it does not include the 11% of learners that were absent. Longitudinal attrition and absenteeism. The evaluation employs a longitudinal design where we will interview the same learners and teachers at baseline and endline. All learners will still be in primary school and are unlikely to move schools between term one of the baseline year and term four of the endline year. Absenteeism rates are, however, high and could present a problem. We plan to visit each school ahead of the team of assessors to confirm that all 30 learners are still at the school and to specially request that those learners attend school on the day of the assessments. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 43 | Page REFERENCES Bruhn M. and D. McKenzie, “In Pursuit of Balance: Randomization in Practice in Development Field Experiments”, American Economic Journal: Applied Economics, 2009, vol. 1, issue 4, 200-232 Case, A., Paxson C. and M. Islam, Making sense of the labor market height premium: evidence from the British household panel survey, Economics Letters 102 (2009), pp. 174–176. Deaton 2007 Height, health, and development. Proceedings of the National Academy of Sciences Aug 2007, 104 (33) 13232-13237; DOI:10.1073/pnas.0611500104 Draper, K., and S. Spaull, (2015). Examining Oral Reading Fluency among Grade 5 Rural English Second Language (ESL) Learners in South Africa: Analysis of NEEDU 2013. South African Journal of Childhood Education 5(2) pp.44-77 Gambell, T and D. Hunter (1999) Rethinking Gender Difference in Literacy. Canadian Journal of Education, 24 (1) pp.1-16 Krashen, S. (2004). The Power of Reading: Insights from the Research. Libraries Unlimited: Westport; Evans, M.D.R. et al. (2010) Family scholarly culture and educational success: books and schooling in 27 nations. Research in Social Stratification and Mobility 28: 171-197; UK Department of Education (2012). Research evidence for reading for pleasure (available from www.gov.uk). Mullis et al., 2012 PIRLS 2011 International Results in Reading TIMSS & PIRLS International Study Centre, Chestnut Hill (2012) Needu (2013). NEEDU reading study 2013: the state of reading in Grade 5 in selected rural primary schools. National Education Evaluation and Development Unit. Available at https://nicspaull.files.wordpress.com/2014/12/needu-reading-study-2013_28oct14.pdf. PIRLS Literacy 2016: South African Highlights Report http://www.up.ac.za/media/shared/164/ZP_Files/pirls-literacy-2016-hl-report-3.zp136320.pdf Riordan, C. (1999) The Silent Gender Gap. Education Week, 19 (12) pp.46-47 Room to Read (2018) Data-Driven Methods for Setting Reading Proficiency Benchmarks. Prepared by Matthew Jukes, Chris Cummiskey, Melissa Gargano, and Peggy Dubeck. https://www.roomtoread.org/media/984466/room-to-read_fluency-benchmarking-analysis_report_may￾2018.pdf RTI International (2017) Setting and Using Benchmarks for Reading Performance. https://ierc￾publicfiles.s3.amazonaws.com/public/resources/10746_Benchmarks_brief_02-16-17_r6.pdf Spaull, N. Pretorius, E., & Mohohlwane, N. (2018). Investigating the comprehension iceberg: Developing empirical benchmarks for early grade reading in agglutinating African languages. RESEP Working Paper Series: 01/2018 Taylor D (2004) Not Just Boring Stories: Reconsidering the Gender Gap for Boys. Journal of Adolescent and Adult Literacy, 48 (4), pp. 290-298 Zuze and Reddy 2014, School Resources and Gender Reading Literacy Gap in South African Schools, International Journal of Educational Development, Volume 36, May 2014, Pages 100-107 USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 44 | Page ANNEXES USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 45 | Page ANNEX I: EVALUATION STATEMENT OF WORK USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 46 | Page ANNEX II: EVALUATION METHODS AND LIMITATIONS SAMPLE SELECTION AND EXPERIMENTAL DESIGN In this section, we describe the approach to sample selection and experimental design. We begin with the RCT and then describe the qualitative component of the study. The approach to selecting the sample for the analysis of reading skills outcomes is designed to balance the concerns of the evaluation with those of implementation. District selection: The IE is taking place in two provinces selected by Nal’ibali, DBE and the USAID mission in South Africa, namely the Eastern Cape and KwaZulu-Natal. NORC met with Nal’ibali who undertook to identify the education districts which had not received Nal’ibali programs previously or received very minimal support. In the Eastern Cape, the evaluation is taking place in the Maluti, Mbizana and MtAyliff/Lusiksiki education districts. We are working in the Ilembe, Ugu and Uthukela districts in KwaZulu-Natal. Community design and school selection While SPS is a school based intervention, the Nal’ibali approach involves the community around the school in a number of ways. Community members receive training and are encouraged to run reading clubs. The campaign is also very visible with billboards and encourages interaction between schools through inter school competitions. The SPS intervention employs Story Sparkers from the community with each SPS school receiving weekly visits from their Story Sparker. The sample framework for the evaluation needed to reflect this approach allowing for multiple SPS schools within the same community and ensuring that control schools are sufficiently far away. Treatment was therefore randomized at the community level. Communities of schools needed to be created bearing in mind that one Story Sparker would need to service multiple schools. The initial SPS model had two Story Sparkers working together to service 10 schools. This meant that all 240 schools receiving the intervention in the first year (2017) would need to come from 24 communities of 10 schools each. The evaluation would not be sufficiently powered with only 12 treatment communities in each province and the decision was therefore made to conduct the baseline over two years. Using detailed maps and working closely with Nal’ibali and district officials from the DBE, we defined 48 appropriate communities of 10 schools each (schools within a community needed to be a close as possible and their needed to be sufficient distance between communities to avoid contamination) for the first cohort. Half of these communities were randomly assigned to receive the SPS intervention and the remainder served as control communities. Within each community, we randomly selected four schools to include in the evaluation. Nal’ibali experienced significant logistical challenges in the first year with transport for the Story Sparkers between schools being a key concern. They therefore adapted their model for the second year USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 47 | Page of the program (2018) to have one Story Sparker servicing four schools. We followed a similar process to the previous year but now created 96 communities of four schools each. Half of these communities were randomly assigned to treatment and the other half to control. Within each community, we randomly selected two schools to include in the evaluation. Once the treatment communities had been randomly selected, all schools in those communities were invited to become SPS. Although take up in both years was 100%, we made sure to avoid any potential selection bias by fixing the evaluation sample and treatment status of evaluation schools prior to the invitations going out. Figure 23: Data selection process Teacher selection In each school selected for data collection, we randomly sampled one teacher from each of grade 2 to 4. Where the school was a multi-grade schools with learners from different grades taught by the same teacher, we selected one teacher from each grade combination in the Grade 2 to 4 range. Learner selection Learners were randomly selected from the classes of the sampled teachers. We randomly selected 30 learners from each school (10 from each of grades 2 to 4, 5 girls and 5 boys whenever possible). Learners were only selected if they are present at the school on the day that the field team arrived to complete the assessments. These assessments and accompanying interviews were conducted in the first term of the school year (February/March) prior to the implementation of the SPS program. SAMPLE SIZE The final planned and achieved sample sizes for the baseline are summarized by treatment status and education district in Table 1 below. The number of schools in the sample is slightly lower than planned, particularly in the Eastern Cape. This is largely due to language issues as several selected schools turned out to have seSotho rather than isiXhosa as the medium of instruction in the foundation phase. There were also two schools that did not offer grades 2 to 4, three schools that denied the field team access USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 48 | Page and two schools that needed to be dropped due to errors on the fieldwork company side. All schools that were dropped from the sample were from the first cohort (2017). A combination of an improved school selection process by Nal’ibali and the DBE and a more efficient and prepared subcontractor resulted in no dropped schools in the second cohort. During fieldwork, we replaced 22 schools. The replacement schools were randomly selected from the same community as the school that they were replacing. In 2017, we randomly sampled 4 out of 10 schools in a community to be included in the evaluation giving us up to 6 replacement schools. In 2018, we randomly sampled 2 out of 4 schools, leaving two potential replacements. Table 2 gives details of school replacements by education districts. Table 3 shows the planned and achieved sample sizes for the learner sample by treatment status. POWER CALCULATIONS This section provides updated power calculations for the realised baseline sample. For details of the calculations used to arrive at the recommended sample sizes please refer to the evaluation design report (and annex I to that report) in Annex I of this report. As with any quantitative analysis based on survey data, the statistical power of any sample design is determined by a mathematical calculation that depends on a number of factors. These include features of the study design, properties of the data and outcome variables, and the desired precision of the analysis. In practice, power calculations require choices and assumptions about a range of these parameters. We carry out the power calculations using the Optimal Design software package, which can perform these calculations under a variety of scenarios. In our power calculations, we consider the impact evaluations of the program in Kwazulu-Natal and the Eastern Cape as two separate evaluations. We make the following assumptions about parameters: • α is the significance level of the test, or probability of Type I error. We use the standard value of 0.95. • β is the power of test, where (1-β) is the probability of Type II error. We use the standard value of 0.8. • ρ is the intracluster correlation coefficient, or ICC. The ICC is a measure of the extent to which variation in outcomes is due to cluster-level factors as opposed to individual-level factors. In our case, the ICC reflects the extent to which differences in EGRA scores are due to differences between clusters, rather than differences between individual students. We randomized at the community level and then selected schools within communities. We there need to take into account how much of the total variation in EGRA scores is due to variation between communities and how much is due to variation between schools within communities. Based on results from previous studies, we assume that 10% of the variation in EGRA scores is due to communities and 10% due to variation between schools within communities. • r2 is the proportion of the variation in the outcome due to the covariates anticipated in the regression analysis. In our case, these covariates will include a range of household and individual characteristics, as well as the intertemporal correlation between outcomes values at baseline and follow up. For this parameter, we assume an approximate value of 0.5. • In the event of a longitudinal study, we assume an attrition rate of 15% USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 49 | Page We also take into account the following features of the final realised baseline sample: • Number of communities: We randomized 68 communities to treatment or control in the Eastern Cape. We randomized 72 communities in KwaZulu-Natal. • Number of schools per community: We randomly select 4 schools per community in 2017 and 2 schools per community in 2018. The average number of schools per community is 2.5 in the Eastern Cape and 2.6 in KwaZulu-Natal. • Number of learners per school: We randomly sampled 30 learners per school. • Take up: All schools assigned to treatment were invited to become a SPS but participation was voluntary. Initial take up of the program was 100% These parameters enable us to calculate the minimum detectable effect size (MDES) of our study. The MDES is the smallest impact of the program on the outcome variable that the evaluation will be able to detect. The MDES is 0.23 in the Eastern Cape and 0.22 in KwaZulu-Natal. QUALITATIVE STUDY In addition to the quantitative components, the IE will include a qualitative study on 20 schools from cohort 1I. This study will be conducted towards the end of the third school term in September 2018. Ten schools in each province will be selected: 5 SPS schools and 5 control schools. Using the evaluation baseline tests, we will select two higher performing, two lower performing and one medium performing schools for each treatment-province combination. Within the sub-sample of schools, we will randomly select one teacher from control schools and two teachers from treatment schools, one of whom will have received Nal’ibali training and one who has not received Nal’ibali training. All 30 selected teachers in SPS and comparator schools will be observed teaching Home language and First Additional Language classes, as well as the literacy half hour if this occurs. The focus of observations will be on key dimensions of pedagogy in general; reading and writing pedagogy specifically; and the SPS preferred pedagogy in particular (see conceptual frame below). The observations aim to measure whether there are discernible differences in the pedagogies between SPS trained teachers in SPS schools; non SPS trained teachers in SPS schools; and teachers in comparator schools. The classroom observations will also generate a general description of classroom pedagogies in Home language and First Additional Language across 30 teachers, a description of interest in its own right in relation to well-known features of classrooms in high poverty settings documented in the local research literature. We will also conduct reading skills assessments with 20 randomly selected students in each of the selected teachers’ classes. These students will be randomly selected from all students who are present at school on the day that the field team arrive to complete assessments. Four in depth case studies of SPS schools, their Story Sparkers and their program activities will be undertaken. The case studies will be sampled on the basis of reputation within the program. In each province, one school will be selected based on its reputation for meeting or exceeding the SPS program expectations, and the other will be selected for struggling or failing to meet the SPS program expectations. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 50 | Page EVALUATION MATRIX # Outcomes Questions Data source(s) Example indicators Data analysis method 1 Student reading skills Q1. What is the impact of the Story Powered Schools (SPS) intervention on learners’: C. reading and writing skills in their mother tongue (including letter recognition, word recognition, fluency, comprehension, vocabulary, phonemic awareness, orientation to print)? D. reading and writing skills in English? EGRA assessments letter recognition, word recognition, fluency, comprehension, vocabulary, phonemic awareness, orientation to print RCT – comparison of treatment versus control 2 Student Motivation and Behavior Q2. What is the impact of the SPS intervention on: C. the time or frequency that students spend reading independently? D. student’s motivation to seek out, listen to, read, write, and tell stories? Learner interview Frequency reading at home RCT – comparison of treatment versus control 3 Teachers Attitude and Behavior Q3. What is the impact of the SPS intervention on: C. teachers’ attitudes towards a. reading for enjoyment in the classroom? b. story telling in the classroom? c. reading clubs? D. teachers’ language and literacy instruction practices in the classroom (both those trained by Nal’ibali and those not trained by Nal’ibali but teaching in SPS schools)? Teacher interview Classroom observations Classroom practices: use of stories, for example RCT – comparison of treatment versus control How to describe the analytical method here? 4 Student reading skills Q4. What is the impact of attending a Nal’ibali reading club (in a SPS) on learners’: C. reading and writing skills in their mother tongue D. reading and writing skills in English? EGRA assessments Fluency, comprehension, vocabulary Propensity score matching 5 Student Motivation and Behavior Q5. What is the impact of attending a Nal’ibali reading club (in a SPS) on: C. the time or frequency that students spend reading independently? D. student’s motivation to seek out, listen to, read, write, and tell stories? Learner interview Frequency reading for enjoyment Propensity score matching 6 Student reading skills Q6. What is the association between teacher practices and learners’: C. reading and writing skills in their mother tongue? D. reading and writing skills in English? Teacher interview and EGRA assessments Fluency, comprehension, vocabulary Correlation analysis / Regression USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 51 | Page # Outcomes Questions Data source(s) Example indicators Data analysis method 7 Student Motivation and Behavior Q7. What is the association between teacher practices and: A. the time or frequency that students spend reading independently? B. student’s motivation to seek out, listen to, read, write, and tell stories? Teacher interview Learner interview Frequency reading for enjoyment Correlation analysis / Regression 8 Program take-up (participation in reading activities and use of reading materials/trainings) Q8. What is take-up of and adherence to the SPS intervention? This will include a range of indicators. For example: A. number of invited schools signing up for SPS B. characteristics of schools signing up for SPS C. number of teachers trained D. characteristics of teachers who attend training E. number of community members trained F. implementation of reading for enjoyment sessions during school day G. number of reading clubs operating at the school H. number of children in the reading clubs by grade I. attendance at reading clubs J. other events (competitions, etc.) K. before and after school reading sessions L. use of the mini libraries Nal’ibali M&E data Case studies Teacher interviews Learner interviews Number of reading clubs operating, attendance to reading clubs, number of reading for enjoyment sessions in school, etc. Descriptive analysis USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 52 | Page Table 1: Deviations from planned school sample by treatment status, district and baseline cohort (year) All Treat ment Control Eastern Cape KwaZulu-Natal 2017 2018 All Maluti Mbizana Mt Ayliff/ Lusikisiki All Ilembe Ugu Uthukela Planned sample 384 192 192 192 48 72 72 192 96 48 48 192 192 Dropped due to language issues (2017) 19 9 10 18 17 1 0 1 0 1 0 19 0 Dropped due to incorrect grades (2017) 2 2 0 0 0 0 0 2 0 1 1 2 0 Dropped due to denied access (2017) 3 0 3 2 0 2 0 1 0 1 0 3 0 Dropped due to fieldwork company error (2017) 2 0 2 1 0 1 0 1 0 1 0 2 0 Added due to no Grade 4 in original school (2018) 2 1 1 1 0 0 1 1 1 0 0 0 2 Final sample 360 182 178 172 31 68 73 188 97 44 47 166 194 Percentage of planned sample 93.8% 94.8% 92.7% 89.6% 64.6% 94.4% 101.4% 97.9% 101.0% 91.7% 97.9% 86.5% 101.0% Table 2: Replacement schools Reason for replacement District Maluti Ugu Uthukela Ilembe Mt Ayliff Total Not isiXhosa medium 10 10 Duplicate 1 1 Closed down 1 1 2 Incorrect grades 1 2 2 3 8 Incorrect school visited 1 1 Total 10 3 2 3 4 22 USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 53 | Page Table 3: School and learner sample by treatment status and baseline cohort (year) All Treatment Control 2017 2018 Planned Observed Planned Observed Planned Observed Planned Observed Planned Observed Schools 384 360 192 182 192 178 192 166 192 194 Learners: Grade 2 learners 3840 3456 1920 1777 1920 1679 1920 1574 1920 1882 Grade 3 learners 3840 3455 1920 1772 1920 1683 1920 1578 1920 1877 Grade 4 learners 3840 3493 1920 1789 1920 1704 1920 1616 1920 1877 Percentage of learners with a matched teacher 95.94 95.86% 96.03% 91.36% 99.82% USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 54 | Page ANNEX III: DATA COLLECTION INSTRUMENTS USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 55 | Page We are visiting some schools in your area to better understand how children learn to read and to measure how well reading programmes in schools are working. We will be assessing the reading abilities of some children and also asking them what they do and don’t enjoy about reading. This will take about 30 to 40 minutes for each child. We will also measure the height of each child. The participating children will be selected by a lottery. Your child may or may not be selected. The assessments will happen at school and will be administered by a trained and experienced assessor, who has experience in working with children and making children feel at ease and comfortable during the assessments. The assessments are not a school test and will have no impact on the child’s school grades. Assessors will explain to the children that no one will be informed of their answers at school or anywhere else. They will also explain to the children that they do not have to do the assessments if they do not want to and that they can stop at any time. We plan to revisit the same children after two years to assess how their reading skills have developed. For this reason, the child’s name will be recorded but it will be kept completely confidential by the research team. At the end of the research we will write a report, but no child’s name or personal information will appear in that report. Also, the results from these tests and other data collected during this research may be shared with other researchers but information identifying any child and linking any child to their assessments will not be made public. This research has been approved by the Commerce Faculty Ethics in Research Committee at the University of Cape Town and Department of Basic Education and the schools have given their permission. If you have questions about the study you can contact us at XXXXXXXXXX until DD/MM/YYYY. If you do NOT want your child to be part of this study then please fill in the section below, sign it and send it back with your child to the school tomorrow. If you are happy for your child to be part of the study, you do not need to send the form back. I do NOT give my permission for my child to be part of the study. CHILD’S NAME: _____________________________________________ GRADE: _____________ SCHOOL: ______________________________________ CLASS TEACHER: _____________________ ______________________________________ Signature of parent or main caregiver of the child Date: ____/_____/ ____ USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 56 | Page Story Powered Schools: A South African Reading Revolution • Let me tell you why I am here today. We are trying to understand how children learn to read. You were picked by chance, like in a raffle or lottery. • We would like your help in this. But you do not have to take part if you do not want to. • We are going to play a reading game. I am going to ask you to read letters, words and a short story out loud. • Using a stopwatch, I will see how long it takes you to read. • This is NOT a test and it will not affect your grade at school. • I will also ask you other questions about your family, like what language your family uses at home and some of the things your family has. • This assessment will take between 30 to 40 minutes. • I will NOT show your answers to your teacher, friends or anyone. When we use your answers to understand how children learn to read we will not use your name. • Once again, you do not have to participate if you do not wish to. Once we begin, if you would rather not answer a question, that’s all right. Do you have any questions? Do you want to participate? On the tablet, tick the check box if verbal agreement is obtained:  YES If verbal agreement is not obtained, thank the learner and move on to the next learner. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 57 | Page English Early Grade Reading Assessment (EGRA) SPS Evaluation Baseline March 2017/2018 Instruments developed by Zenex Foundation. Minor adaptations by NORC. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 58 | Page LETTER SOUNDS • Note: Learners must give the letter sound and not the letter name. • Show the learner the chart of letters in the language you are administering: • Here is a page full of letters. I would like you to sound as many letters as you can. You will start here and move across the page. Point to the leftmost letter on the top row of the exercise, moving from left to right. When I say, ‘Begin’, you will sound the letters as best you can. Point to each letter as you sound it. If you don’t know the sound of a letter, just leave it out. • Let’s practice first. (Point to the first example letter, moving from left to right, to practice the instructions given above. Make sure that the learner understands that you are looking for the letter sound and not the letter name.) • Ok, now we’re ready to begin. Put your finger on the first letter. Ready? Begin. • Start the TIMER with 60 seconds on the tablet when the learner starts. • Strike a line through a letter that the learner sounds incorrectly or cannot sound at all. For example: b • If the learner stops for more than 3 seconds, tell the learner to go on and strike a line through the letter. For example: b • If the learner corrects himself/herself, accept it as correct. If a strike has already been made on the letter, circle it to mark it correct. • If the entire first line has strike-through lines across all the letters, stop the letter sounds task, place a bracket (]) after the last letter on the first line and make a tick mark () at the bottom of the exercise (in the box provided) to record that the exercise was discontinued. Move onto the next part of the EGRA. • After sixty (60) seconds, when the TIMER alarm goes off say: “Stop”. Place a bracket (]) after the last letter that the learner has attempted to sound. • Count and record the number of letters that the learner sounded correctly. • If the learner sounds all the letters in less than sixty (60) seconds, record the number of seconds remaining on the timer. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 59 | Page LETTER SOUNDS ENGLISH GRADE 3 & 4 Examples: b M f cr S v n g L y Z h W m /10 k t d ch i th a gl z w /20 h oo sh m dr G j N wh u /30 R pr B s I f gr st T i /40 sl ing C B ph Y F cr a E /50 y s ck P M v O th n pl /60 Z A ow ight fr q h ch A t /70 wh ee H br S fl g l V m /80 a oy sl gr X N E Q p x /90 N k cl D dr y bl j R v /100 V M W e ight h ing pr str y /110 Make a tick mark (  ) if task was discontinued because learner couldn’t read Number of sounds read in 60 seconds Total number of sounds read correctly in 60 seconds If task completed in less than 60 seconds, indicate time remaining (in seconds) USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 60 | Page PHONEMIC AWARENESS (a listening activity) • Tell the learner: “We are going to play a game with the sounds of words and breaking up words” Let’s practice first. Go through the examples so that the learner understands what is expected of him/her. After the 2 practice examples, start with the task. • The learner should not see the words. • You may repeat the word, but do not prompt the learner. • If the child has not given an answer within 5 seconds, move onto the next item. If the learner can’t do the task on 3 successive items, discontinue the phonemic awareness task and move to the next section. In order to finish the task on the tablet you will need to select the D (Don’t know) option for the remaining items. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 61 | Page PHONEMIC AWARENESS (a listening test) ENGLISH GRADE 4 Practice Say Sunday umbrella Now say it again, but don’t say /Sun/ -> day Now say it again, but don’t say /um/ -> brella Correct Incorrect No response Correct form Item 1 Say cowboy Now say it again, but don’t say /boy/ o o o cow 2 Say run What sound comes at the beginning of the word run? o o o /r/ 3 Say fish What sound comes at the end of the word fish? o o o /sh/ 4 Say meat Now say it again, but don’t say /m/ o o o eat 5 Say book Now say it again, but don’t say /k/ o o o boo 6 Say pan Now say it again, but use /f/ instead of /p/ o o o fan 7 Say dot Now can you think of another word that sounds like dot? o o o cot, got, hot, not,pot, rot… 8 Say smack Now say it again, but don’t say /m/ o o o sack Total correct USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 62 | Page WORD READING • Show the learner the chart of words in the language that you are administering. • Here is a page full of words. I would like you to read aloud as many words as you can. You will start here and move across the page. Point to the leftmost word on the top row of the exercise, moving from left to right. When I say, ‘Begin’, you will read the words as best you can. Point to each word as you read it. If you don’t know a word, leave it out. • Let’s practice first. (Point to the first example word to practice the instructions given above.) • Ok, now we’re ready to begin. Put your finger on the first word. Ready? Begin. • Start the TIMER with 60 seconds on the tablet when the learner starts. • Strike a line through a word that the learner reads incorrectly or cannot read at all. For example: sad • If the learner stops for more than three (3) seconds, tell the learner to go on and strike a line through the word. For example: sad • If the learner corrects himself/herself, accept it as correct. (If a strike has already been made on the word, circle it to mark it correct.) • If the entire first line has strike-through lines across all the words, stop the word reading task, place a bracket (]) after the last word on the first line and make a tick mark () at the bottom of the exercise (in the box provided) to record that the exercise was discontinued. • After sixty (60) seconds, when the TIMER alarm goes off say: “Stop”. Place a bracket (]) after the last word that the learner has read correctly. • Count and record the number of words that the learner read correctly. • If the learner reads all the words in less than sixty (60) seconds, record the number of seconds remaining on the timer. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 63 | Page WORD READING ENGLISH GRADE 3 & 4 Examples: cat bell it be do of we to She was /8 but did eat got once you day coat /16 the are girl some here within have came /24 went then frog away from play drum what /32 bear colour shave ideas which their white these /40 square said little there knee things other answer /48 water enough pictures carry wheels straight giraffe bright /56 mother people through jackal quickly hungry together kitchen /64 problem believe question usually brought certain perhaps beautiful /72 uncle thought earth oxygen twisted complete weight necessary /80 giant different autumn general electricity surprise newspaper multiply /88 tongue laugh morning ground knowledge favourite environment journey /96 Make a tick mark (  ) if task was discontinued because learner couldn’t read Number of words read in 60 seconds Total number of words read correctly in 60 seconds If task completed in less than 60 seconds, indicate time remaining (in seconds) USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 64 | Page ORAL READING FLUENCY • Show the learner the reading passage for their Grade and language. Now I’m going to ask you to read this story out loud. If you get stuck, leave the word out and keep on reading. When I say, ‘Stop’, stop reading the story. I will next ask you some questions about what you have just read – so try to remember the story you’re reading. You will start here. (Point to the first word of the passage.) • Ready? Begin. • Start the TIMER with 60 seconds on the tablet when the learner starts. • Strike a line through words that the learner reads incorrectly or cannot read at all. For example: little • If the learner stops for more than three (3) seconds, tell the learner to go on and strike a line through the word. For example: little • If the learner corrects himself/herself, accept it as correct. (If a strike has already been made on the word, circle it to mark it correct.) • If the entire first line has strike-through lines across all the words, stop the assessment, place a bracket (]) after the last word on the first line and make a tick mark () at the bottom of the exercise (in the box provided) to record that the exercise was discontinued. • After 60 seconds when the TIMER alarm goes off, place a bracket (]) after the last word that the learner has read and say Stop. Thank the child and discontinue the task. • Count and record the number of words that the learner read correctly in 60 seconds. • If the learner reads the whole story in less than 60 seconds, record the number of seconds remaining on the timer. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 65 | Page ORAL READING FLUENCY ENGLISH GRADE 4 Words read 12 Jabu had a little dog. It was a fat and happy dog. /12 27 One day Jabu and his dog went to play in the fields behind the house. /27 38 The little dog saw a rabbit and tried to chase it. /38 42 The dog got lost. /42 49 Jabu called him but he didn’t come. /49 58 Jabu got tears in his eyes and went home. /58 66 But before evening the little dog came back. /66 71 He was tired and hungry. /71 79 Jabu was very happy to see his friend. /79 87 He gave the dog some food and water. /87 94 Jabu and his dog went to sleep. /94 103 In the morning they woke up and played again. /103 Make a tick mark (  ) if task was discontinued because learner couldn’t read any of the first line Number of words read in 60 seconds Total number of words read correctly in 60 seconds If task completed in less than 60 seconds, indicate time remaining (in seconds) USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 66 | Page ORAL READING COMPREHENSION • I am going to ask you a few questions about the story you have just read. Try to answer the questions as best you can. • If the learner read only part of the story, only ask the questions related to the part that s/he has read. Enter the option D (Don’t Know) option for questions not covered. • If the learner corrects himself/herself, accept the answer as correct. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 67 | Page ORAL READING COMPREHENSION ENGLISH GRADE 4 Question Words read Answer Correct Incorrect No response 1. In the story who does the little dog belong to? 5 Jabu 2. How can you tell that Jabu liked the dog? 12 The dog was fat and happy/He looked after his dog well/they played together 3. Where did they go and play one day? 27 In the fields (veld) / behind Jabu’s house 4. What animal did the dog chase? 38 A rabbit 5. Why did the little dog get lost? 42 He lost his way when he chased the rabbit/the dog couldn’t find his way back home 6. Why did Jabu have tears in his eyes? 58 He was sad/was crying because his dog was lost/he could not find his dog. 7. When did the dog come home again? 66 Before evening/ before the night came/before dark 8. What did Jabu give his dog when he came home? 87 Some food and water 9. Why do you think the dog was hungry when he got home? 87 He had been lost/he got hungry and thirsty when he tried to find his way home again. 10. Who is Jabu’s friend? 79 His little dog. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 68 | Page VOCABULARY • Show the learner the Vocabulary Show Chart for the language you are testing. • Look at the word in the first box. Show me the correct picture. • Example: show the pictures and point to the cup. • Now you try it. Let learner try the next example. • Is it clear? If not, explain again. • If the learner has not given an answer in approximately 10 seconds, tell them to move to the next item. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 69 | Page VOCABULARY ENGLISH GRADE 4 Examples: cup sun USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 70 | Page bus circle sitting helicopter penguin mountain USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 71 | Page Xhosa Early Grade Reading Assessment (EGRA) SPS Evaluation Baseline March 2017/2018 Instruments developed by Zenex Foundation. Minor adaptations by NORC. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 72 | Page CONCEPT OF PRINT AND ORIENTATION GRADE 2 Ingqikelelo yokushicilela nokwazisa Ndiza kunika incwadi, ndiza kuthanda ukuba undincede ngayo Mnike inqaku elinye 1. Ukubonisa okanye indlela eyondlalwe ngayo itekisi/ Umphandle wencwadi: Nika umntwana incwadi, uyilalisile, ukuze umqolo wencwadi ujonge umntwana. Mbuze: a. “Uphi umphambili wencwadi?” b. “Uphi umva wencwadi?” c. “Vula incwadi apho ibali liqala khona.” Umntwana angavula incwadi kwiphepha lokuqala okanye kwiphepha elinesihloko sencwadi. 2. Imibhalo, hayi imifanekiso, yiyo ephethe umyalezo: Uvule incwadi kwiphepha loku-1, buza umntwana: a. “Ndibonise umfanekiso.” Chaza kwaye uxoxe iinkcukacha ezingomfanekiso. b. “Ndibonise amagama.” 3. Ukukhomba isiqalo semibhalo: Kwa kweli phepha mbuze: a. “Khawundibonise ukuba ndiqale phi ukufunda.” Ukuba umtwana akaphenduli nto inesihlahla, mkhokele. “Ndiqala phi Kanye?” “Ndibonise ngomnwe.” b. Khomba igama lokuqala, lifunde kwaye ubuze: “Ndifunda phi xa ndigqiba apha?” 4. Ukulandelana kwamaphepha: Khomba igama lokugqibela kwiphepha elisekhohlo ubuze: • “Ndifunda phi xa ndisuka apha?” 5. Kwakwiphepha elinye buza: a. “Ngomnwe wakho, ndibonise unobumba wokuqala egameni.” Naliphi igama lilungile, umntwana makakhethe. b. “Ngoku, ndibonise unobumba wokugqibela wegama.” 6. Buyela kwenye indawo: Tyhila iphepha elinemigca emibini ubucinane yetekisi. Funda umgca ophezulu ugcine umnwe wakho ukwigama lokugqibela ukubuza: • “Ndifunda phi xa ndisuka apha?” 7. Impendulo yomntu ngamnye: Khomba igama lokuqala kwiphepha elitsha kwaye phambi kokufunda, buza umntwana: • “Khomba igama ngalinye njengoko ndifunda umgca.” Ingaba umntwana uyalandela kwaye uyadibanisa igama njengoko ufunda? USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 73 | Page LETTER SOUNDS • Note: Learners must give the letter sound and not the letter name. • Show the learner the chart of letters in the language you are administering: • Nasi isiqendu esinonobumba abangcweleyo. Ndingathanda undinike isandi sonobumba abaninzi kangangoko unako. Ndingathanda ukuba uqale apha ufunde uye ngaphaya. Point to the leftmost letter on the top row of the exercise, moving from left to right. Xa ndisithi ‘Qala’, biza oonobumba kangangoko unako. Khomba unobumba ngamnye ombizayo. Ukuba akukwazi ukubiza onobumba yiyeke injalo. • Masifundisane kuqala. (Point to the first example letter, moving from left to right, to practice the instructions given above. Make sure that the learner understands that you are looking for the letter sound and not the letter name.) • Ewe singaqala. Beka umnwe kunobumba wokuqala. Sewulungile? Qala. • Start the TIMER with 60 seconds on the tablet when the learner starts. • Strike a line through a letter that the learner sounds incorrectly or cannot sound at all. For example: b • If the learner stops for more than 3 seconds, tell the learner to go on and strike a line through the letter. For example: b • If the learner corrects himself/herself, accept it as correct. If a strike has already been made on the letter, circle it to mark it correct. • If the entire first line has strike-through lines across all the letters, stop the letter sounds task, place a bracket (]) after the last letter on the first line and make a tick mark () at the bottom of the exercise (in the box provided) to record that the exercise was discontinued. Move onto the next part of the EGRA. • After sixty (60) seconds, when the TIMER alarm goes off say: “Yima”. Place a bracket (]) after the last letter that the learner has attempted to sound. • Count and record the number of letters that the learner sounded correctly. • If the learner sounds all the letters in less than sixty (60) seconds, record the number of seconds remaining on the timer. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 74 | Page LETTER SOUNDS ISIXHOSA GRADES 2, 3 & 4 Imizekelo: b M f ts ntl S v n g L y Z h W m /10 k th G b c hl q d z a /20 p nd z U ng sh j kh X u /30 g sw B qh i f e Z R I /40 S n lw ndl t Y ts bh nc E /50 y tsh Q ph M v O t ny P /60 dl A e mp f ph h u A r /70 W dw H b nz g dl m hl i /80 L o lw a X ntl E ny p x /90 N kw nc D ndl y mb j C mv /100 V ncw W q V h ndl g tsh y /110 Make a tick mark (  ) if task was discontinued because learner couldn’t read Number of sounds read in 60 seconds Total number of sounds read correctly in 60 seconds If task completed in less than 60 seconds, indicate time remaining (in seconds) USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 75 | Page PHONEMIC AWARENESS (a listening activity) • Tell the learner: Sizokudlala umdlalo ngokubiza amagama ngokuwo hlula-hlula. Masifundisane kuqala. • Go through the examples so that the learner understands what is expected of him/her. After the 2 practice examples, start with the task. • The learner should not see the words. • You may repeat the word, but do not prompt the learner. • If the child has not given an answer within 5 seconds, move onto the next item. If the learner can’t do the task on 3 successive items, discontinue the phonemic awareness task and move to the next section. In order to finish the task on the tablet you will need to select the D (Don’t know) option for the remaining items. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 76 | Page PHONEMIC AWARENESS (a listening activity) ISIXHOSA GRADES 2, 3 & 4 Practice Ndibizele eligama Sipho yena Biza eligama kwakhona ususe unobumba u S -> ipho Eligama liqala ngowuphi unobumba > /y/ Correct Incorrect No response Correct form Item 1 Ndibizele eligama vuka Liqala ngowuphi unobumba eligama o o o /v/ 2 Ndibizele eligama hleka Sesiphi isandi osivayo ekuqaleni kweligama o o o /hl/ 3 Ndibizele eligama cula Libize kwakhona eligama ususe ucu o o o la 4 Ndibizele eligama namhlanje Libize kwakhona eligama ususe u nje o o o namhla 5 Ndibizele eligama njalo Libize eligama kwakhona ususe unobumba u n o o o jalo 6 Ndibizele eligama kufanele Libize kwakhona eligama ususe unobumba u f o o o ku-anele 7 Ndibizele eligama kuqala Phinda ulibize eligama kodwa ngoku usebenzise u l endaweni ka q o o o kulala 8 Ndibizele eligama bona Ndibizele elinye igama olicingayo eliqala ngonobumba u b o o o biza, bila, basa, (u)busi… USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 77 | Page WORD READING • Show the learner the chart of words in the language that you are administering. • Nasi isqendu esinamagama agcweleyo. Ngoku ndizokubuza wena ufunde amagama ngokuvakalayo. Ndingathanda ukuba uqale apha ufunde uye ngaphaya. Point to the leftmost word on the top row of the exercise, moving from left to right. Xa ndisithi ‘Qala’, funda amagama kangangoko unako. Khomba igama ngalinye olifundayo. Ukuba akulazi igama liyeke. • Masifundisane kuqala. (Point to the first example word to practice the instructions given above.) • Ewe singaqala. Beka ke umnwe wakho kwigama lokuqala. Sewulungile? Qala. • Start the TIMER with 60 seconds on the tablet when the learner starts. • Strike a line through a word that the learner reads incorrectly or cannot read at all. For example: sad • If the learner stops for more than three (3) seconds, tell the learner to go on and strike a line through the word. For example: sad • If the learner corrects himself/herself, accept it as correct. (If a strike has already been made on the word, circle it to mark it correct.) • If the entire first line has strike-through lines across all the words, stop the word reading task, place a bracket (]) after the last word on the first line and make a tick mark () at the bottom of the exercise (in the box provided) to record that the exercise was discontinued. • After sixty (60) seconds, when the TIMER alarm goes off say: “Yima”. Place a bracket (]) after the last word that the learner has read correctly. • Count and record the number of words that the learner read correctly. • If the learner reads all the words in less than sixty (60) seconds, record the number of seconds remaining on the timer. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 78 | Page WORD READING ISIXHOSA GRADES 2, 3 AND 4 Imizekelo: vula into ikati iti vela lala khala /5 hleka cula imali qala usuku /10 Indoda izitya amanzi xoxa iqanda /15 sela ubisi incwadi indlebe umhlobo /20 Baleka ifama itswele jikeleza ingxowa /25 umntwana ikomityi abafundi impama inkosikazi /30 Ucango iinzwane iminqathe impukane ingonyama /35 wahleka ingxolo inciniba ifesitile umnqweno /40 uMgqibelo isigcawu ukuxhentsa ulwandle intshonalanga /45 ehlathini phandle iphephandaba isibhedlele ukuqhayisa /50 mhlawumbi umntwana ingqukuva inkwenkwe ibhabhathane /55 Make a tick mark (  ) if task was discontinued because learner couldn’t read Number of words read in 60 seconds Total number of words read correctly in 60 seconds If task completed in less than 60 seconds, indicate time remaining (in seconds) USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 79 | Page ORAL READING FLUENCY • Show the learner the reading passage for their Grade and language. Ngoku ndizokucela undifundele eli bali ngokuvakalayo. Ukuba uxakiwe liyeke igama ngaphandle uqhubekeke ngokufunda. Xa ndisithi “yima”, yima ukufunda ibali. Xa ugqibile ukufunda ndizokubuza imibuzwana malunga nokufundileyo lilonke kufuneka uzamile ukulikhumbula ibali ubulifunda. Ndizokuqala apha. (Point to the first word of the passage.) • Sewulungile? Qala. • Start the TIMER with 60 seconds on the tablet when the learner starts. • Strike a line through words that the learner reads incorrectly or cannot read at all. For example: little • If the learner stops for more than three (3) seconds, tell the learner to go on and strike a line through the word. For example: little • If the learner corrects himself/herself, accept it as correct. (If a strike has already been made on the word, circle it to mark it correct.) • If the entire first line has strike-through lines across all the words, stop the assessment, place a bracket (]) after the last word on the first line and make a tick mark () at the bottom of the exercise (in the box provided) to record that the exercise was discontinued. • After 60 seconds when the TIMER alarm goes off, place a bracket (]) after the last word that the learner has read and say Yima. Thank the child and discontinue the task. • Count and record the number of words that the learner read correctly in 60 seconds. • If the learner reads the whole story in less than 60 seconds, record the number of seconds remaining on the timer. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 80 | Page ORAL READING FLUENCY ISIXHOSA GRADE 2 Words read 3 UJabu wayenenja encinane. /3 8 Ngenye imini uJabu nenja yakhe /8 12 bayakudlala kwibala elalisemva kwendlu. /12 17 Injana yabona umvundla yazama ukuwuleqa. /17 19 Yalahleka inja. /19 24 Wabangayibiza uJabu kodwa ayizange ibuye. /24 31 UJabu wabonakala eneenyembezi emehlweni akhe waza wagoduka. /31 37 Kodwa kwathi phambi kwangorhatya yabuya injana. /37 39 Yayidiniwe, ilambile. /39 45 Wavuya kakhulu uJabu akumbona umhlobo wakhe. /45 51 UJabu wayinika ukutya namanzi inja. /51 56 UJabu nenja yakhe baya kulala. /56 Make a tick mark (  ) if task was discontinued because learner couldn’t read any of the first line Number of words read in 60 seconds Total number of words read correctly in 60 seconds If task completed in less than 60 seconds, indicate time remaining (in seconds) USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 81 | Page ORAL READING FLUENCY ISIXHOSA GRADE 3 Words read 4 Kwakukho umhambi owayelambe kunene. /4 8 Wahamba engena ecela amalizo. /8 14 Kwakungekho kutya, kwanto tu kwaphela emizini. /14 17 Umhambi wachola imbiza. /17 22 Wachola namatye agudileyo wawafaka embizeni. /22 27 Wagalela amanzi wabasa umlilo wapheka. /27 32 Wachopha walinda de yabila imbiza. /32 38 Kwafika umfazana wafuna ukwazi okuphekwa ngumhambi. /38 50 “Ndipheka isuphu yelitye emnandi. Ukuba ndingathi chatha nto ukuyinika isongo?” waphendula umhambi. /50 56 “Ndinayo iminqathe,” wabe selenika umhambi umfazana /56 Make a tick mark (  ) if task was discontinued because learner couldn’t read any of the first line Number of words read in 60 seconds Total number of words read correctly in 60 seconds If task completed in less than 60 seconds, indicate time remaining (in seconds) USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 82 | Page ORAL READING FLUENCY ISIXHOSA GRADE 4 Words read 7 Ngenye imini kudala, kwakukho umhambi owayelambe kunene. /7 10 Wayengenakutya tu kwaphela, /10 17 engenayo nendlela awayengenza ngayo ukuncedakala afumane ukutya. /17 26 Wafika lalini ithile, wahamba, enkqonkqoza, engena emizini, ecela amalizo. /26 31 Abahlali belali nabo babengenako ukutya. /31 35 Babeswele, bengamahlwempu, bengathathi ntweni. /35 40 Umhambi waphelelwa lithemba, wemka ke, /40 47 ebhuqwa yindlala engazi ukuba uza kuthini na. /47 55 Akuphumela, edlula elalini leyo, umhambi wabona umthi omkhulu, /55 61 wakhe wahlala phantsi komthunzi wawo, wacinga. /61 70 Esahleli phantsi komthunzi lowo, wabona imbiza enkulu emnyama, kufutshane. /70 76 Wafikelwa yingcinga, waqokelela iinkuni waphemba umlilo, /76 82 wahlamba amatye acoceka. Wawafaka embizeni enamanzi. /82 Make a tick mark (  ) if task was discontinued because learner couldn’t read any of the first line Number of words read in 60 seconds Total number of words read correctly in 60 seconds If task completed in less than 60 seconds, indicate time remaining (in seconds) USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 83 | Page ORAL READING COMPREHENSION • Ndizokubuza imibuzo embalwa malunga nokufundileyo. Zama ukuphendula imibuzo kanga ngoko unako. • If the learner read only part of the story, only ask the questions related to the part that s/he has read. Enter the option D (Don’t Know) option for questions not covered. • If the learner corrects himself/herself, accept the answer as correct. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 84 | Page ORAL READING COMPREHENSION ISIXHOSA GRADE 2 Umbuzo Words read Impendulo Correct Incorrect No response 1. Ngokweli bali, yekabani le njana? 3 YekaJabu 2. Umbona njani okokuba uJabu wayeyithanda lenja? 12 Wayeyikhathalele eyiphethe kakuhle inja yakhe/ babedlala kunye 3. Bahamba bayakudlala phi ngenye imini? 12 Kwibala elalisemva kwendlu 4. Sisiphi isilwanyana esaleqwa yile nja? 17 Ngumvundla 5. Kwakutheni ize ilahleke injana? 19 Yalahleka xa yayisendleleni ileqa umvundla 6. Kwakutheni ukuze uJabu abeneenyembezi emehlweni akhe? 31 Wayekhathazekile/wayelila kuba inja yakhe yayilahlekile/wayengayiboni inja yakhe. 7. Yabuyela nini inja ekhaya? 37 Phambi kokuba kuhlwe/phambi kokuba kufike ubusuku. 8. Wayinika ntoni uJabu inja yakhe yakufika ekhaya? 51 Ukutya namanzi 9. Ucinga okokuba kutheni inja yayilambile ngethuba ifika ekhaya? 51 Yayilahlekile/yalamba yanxanwa ngethuba eyayizama ukufumana indlela ebuyela ekhaya. 10. Ngubani umhlobo kaJabu? 56 Yinjana yakhe/ngumbundlwana wakhe. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 85 | Page ORAL READING COMPREHENSION ISIXHOSA GRADE 3 Umbuzo No of words read Impendulo Correct Incorrect No response 1. Ngubani lo wayelambile? 4 Umhambi 2. Wenza ntoni umhambi ngamatye? 22 Waawafaka embizeni kuba ezakwenza isuphu yamatye. 3. Yintoni enye into eyafakwa ngumhambi embizeni? 27 Yagalela amanzi / amanzi 4. Kwakutheni ukuze achophe ecaleni kwembiza? 32 Wayelinde ukuba amanzi ade abile. 5. Ngubani owafuna ukwazi kumhambi ukuba upheka ntoni? 38 Ngumfazana 6. Kwakutheni ukuze umfazana anike lomhambi iminqathe? 56 Wayefuna ukumnceda ukuze enze isuphu emnandi / enesongo / enencasa 7. Ucinga okokuba lesuphu yamatye izakuvuthwa? 56 Ewe / hayi USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 86 | Page ORAL READING COMPREHENSION ISIXHOSA GRADE 4 Umbuzo Words read Impendulo Correct Incorrect No response 1. Laliqhubeke nini elibali? 7 Kudala 2. Kwakutheni ukuze umhambi ahambe engena ezindlwini zabantu? 26 Wayelambile efuna ukutya 3. Kwakutheni ukuze abantu belali bangamniki umhambi ukutya? 31 Babeswele bengamahlwempu/ babengenako nabo ukutya 4. Kwakutheni ukuze umhambi ahlale phantsi acinge? 61 Wayelambile ecinga indlela anookufumana ngayo ukutya 5. Wahlala phantsi kwantoni umhambi? 61 Wahlala phantsi komthi omkhulu 6. Yintoni eyabonwa ngumhambi ileli engceni? 70 Yimbiza 7. Wawunjani umbala walembiza? 70 Wawumnyama 8. Ucinga ukuba isuphu yelitye ingatyeka? Kutheni usitsho? 82 Hayi/ewe USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 87 | Page VOCABULARY • Show the learner the Vocabulary Show Chart for the language you are testing. • Jonga igama elisebhokisini yokuqala. Ubonise umfanekiso walo. • Example: show the pictures and point to the dog. • Ngoku yizame. Let learner try with second example. • Ingaba icacile? If not, explain again. • If the learner has not given an answer in approximately 10 seconds, tell them to move to the next item. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 88 | Page VOCABULARY ISIXHOSA GRADES 2, 3 & 4 Examples: Inja Amehlo USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 89 | Page Isandla inkukhu izihlangu ibhulukhwe indlulamthi inkwenkwe zi USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 90 | Page Zulu Early Grade Reading Assessment (EGRA) SPS Evaluation Baseline March 2017/2018 Instruments developed by Zenex Foundation. Minor adaptations by NORC. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 91 | Page PRINT CONCEPT AND ORIENTATION GRADE 2 Amatemu ombhalo nokuma kwencwadi Ngizokunika incwadi, ngizocela ukuthi ungisize ngayo Nika iphuzu elilodwa 1. Ukuma kwenzcwadi noma isakhiwo sombhalo / isembozo sebhuku sangaphambili: Nika umntwana incwadi, uyibambe ibheke phezulu ukuze ugogodla wencwadi ubheke umntwana. Buza: a. “Likuphi ingaphambili lencwadi?” b. “Likuphi ingemuva lencwadi?” c. “Vula incwadi lapho kuqala khona indaba.” Umntwana angayivula incwadi athole ikhasi lokuqukethwe noma ikhasi olokuqala lendaba. 2. Imibhalo, hhayi izithombe iqukethe umyalezo: Njengoba incwadi ivulwe ekhasini loku-1 encwadini, buza umntwana: a. “Ngikhombise isithombe.” Chaza bese uxoxa ngemininingwane esesithombeni. b. “Ngikhombise amagama” 3. Ukuma kombhalo: Buza lokhu mayelana nekhasi elivuliwe: Direction of print: On the same page ask: a. “Ngikhombise lapho kumele uqale khona ukufunda.”Uma impendulo yomfundi ingaqondisiseki, musize ngokumbuza; Kuphi kahle kahle? Ngikhombise ngomunwe”. Khomba igama lokuqala, lifunde bese ubuza: b. “Ngifunda kuphi emuva kwalokhu?” 4. Ukulandelaniswa kwamakhasi: Khomba igama lokugcina ekhasini elingakwesokunxele bese ubuza: • “Ngifunda kuphi emuva kwalokho?” 5. Usekulona leli khasi, buza: a. Ngomunwe wakho, ngikhombise uhlamvu lokuqala egameni.” Kungaba yinoma kuliphi igama. Umntwana makazikhethele yena igama. b. “Manje, ngikhombise uhlamvu lokugcina egameni.” 6. Ukusuka ekugcineni komugqa kuya ekuqaleni komugqa olandelayo: Pheqa ikhasi elinemibhalo enemigqa emibili. Funda umugqa ongenhla, usabeke umunwe wakho egameni lokugcina buza: “Ngifunda kuphi emuva kwalokhu?” 7. Ukuqondanisa igama nokufundwayo: Khomba igama lokuqala ekhasini elisha, ngaphambi kokufunda, buza umntwana lokhu: • “Khomba egameni ngalinye njengoba ngifunda lo mugqa.” Kungabe umntwana ulandela kahle futhi lokho akukhombisayo kuyahambelana nokufundayo? USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 92 | Page LETTER SOUNDS • Note: Learners must give the letter sound and not the letter name. • Show the learner the chart of letters in the language you are administering: • Nali ikhasi eligcwele imibhalo. Ngicela wenze umsindo wemibhalo eminingi ukuya ngekhonolwakho. Uzoqala lapha, uvundle nekhasi. Point to the leftmost letter on the top row of the exercise, moving from left to right. Manje, umangithi, ‘Qala’, phumisa izinhlamvu ngendlela ozokwazi ngayo. Khomba uhlamvu ngalunye ngenkathi uluphimisa. Uma kukhona umbhalo ongawazi, uyeke. • Makesenze Isibonelo. (Point to the first example letter, moving from left to right, to practice the instructions given above. Make sure that the learner understands that you are looking for the letter sound and not the letter name.) • Sesilungele ukuqala. Beka umunwe wakho ohlamvini lokuqala. Usulungile? Qala. • Start the TIMER with 60 seconds on the tablet when the learner starts. • Strike a line through a letter that the learner sounds incorrectly or cannot sound at all. For example: b • If the learner stops for more than 3 seconds, tell the learner to go on and strike a line through the letter. For example: b • If the learner corrects himself/herself, accept it as correct. If a strike has already been made on the letter, circle it to mark it correct. • If the entire first line has strike-through lines across all the letters, stop the letter sounds task, place a bracket (]) after the last letter on the first line and make a tick mark () at the bottom of the exercise (in the box provided) to record that the exercise was discontinued. Move onto the next part of the EGRA. • After sixty (60) seconds, when the TIMER alarm goes off say: “Yima”. Place a bracket (]) after the last letter that the learner has attempted to sound. • Count and record the number of letters that the learner sounded correctly. • If the learner sounds all the letters in less than sixty (60) seconds, record the number of seconds remaining on the timer. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 93 | Page LETTER SOUNDS ISIZULU GRADE 2, 3 AND 4 Isibonelo: b M f ts S v n g L y Z h W m /10 k th G b c hl q d z a /20 p nd z U ng sh j kh X u /30 g sw B qh i f e Z R I /40 S n lw ndl t Y ts bh nc E /50 y tsh Q ph M v O t ny P /60 dl A e mp f ph h u A r /70 W dw H b nz g dl m hl i /80 L o lw a X nhl E ny p x /90 N kw nc D ndl y mb j C mv /100 V ncw W q V h nhl g tsh y /110 Make a tick mark (  ) if task was discontinued because learner couldn’t read entire first line Number of sounds read in 60 seconds Total number of sounds read correctly in 60 seconds If task completed in less than 60 seconds, indicate time remaining (in seconds) USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 94 | Page PHONEMIC AWARENESS (a listening activity) • Tell the learner: “Sizodlala umdlalo ngemisindo yamagama siphinde sihlukanise amagama.” Makesenze Isibonelo. Go through the examples so that the learner understands what is expected of him/her. After the 2 practice examples, start with the task. • The learner should not see the words. • You may repeat the word, but do not prompt the learner. • If the child has not given an answer within 5 seconds, move onto the next item. If the learner can’t do the task on 3 successive items, discontinue the phonemic awareness task and move to the next section. In order to finish the task on the tablet you will need to select the D (Don’t know) option for the remaining items. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 95 | Page PHONEMIC AWARENESS (a listening activity) ISIZULU GRADES 2, 3 & 4 Practice Yisho Yisho Sipho yena Manje isho igama ususe umsindo u /s/ -> ipho Yisho umsindo otholakala ekuqaleni kwegama - > /y/ Correct Incorrect No response Correct form Item 1 Yisho vuka Yisho umsindo otholakala ekuqaleni kwegama o o o /v/ 2 Yisho hleka Yisho umsindo otholakala ekuqaleni kwegama o o o /hl/ 3 Yisho cula Manje isho igama ususe umsindo u /cu/ o o o la 4 Yisho namhlanje Manje isho igama ususe umsindo u /nam/ o o o hlanje 5 Yisho njalo Manje isho igama ususe umsindo u /n/ o o o jalo 6 Yisho kufanele Manje isho igama ususe umsindo u /f/ o o o ku-anele 7 Yisho kuqala Manje isho igama, usebenzise /l/ esikhundleni sika /q/ o o o kulala 8 Yisho bona Manje cabanga amanye amagama aqala ngomsindo /b/? o o o biza, bila, basa… USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 96 | Page WORD READING • Show the learner the chart of words in the language that you are administering. • Nalikhasi eligcwele amagama. Ngicela ungifundela ngelizwi eliphezulu amagama amaningi ukuya ngekhonolwakho. Uzoqala lapha, uvundle nekhasi. Point to the leftmost word on the top row of the exercise, moving from left to right. Uma ngithi, ‘Qala’, funda amagama ngendlela ozokwazi ngayo. Khomba egameni ngalinye ngenkathi ulifunda. Uma kukhona igama ongalazi, liyeke. • Makesenze Isibonelo. (Point to the first example word to practice the instructions given above.) • Sesilungele ukuqala. Beka umunwe wakho egameni lokuqala. Usulungile? Qala. • Start the TIMER with 60 seconds on the tablet when the learner starts. • Strike a line through a word that the learner reads incorrectly or cannot read at all. For example: sad • If the learner stops for more than three (3) seconds, tell the learner to go on and strike a line through the word. For example: sad • If the learner corrects himself/herself, accept it as correct. (If a strike has already been made on the word, circle it to mark it correct.) • If the entire first line has strike-through lines across all the words, stop the word reading task, place a bracket (]) after the last word on the first line and make a tick mark () at the bottom of the exercise (in the box provided) to record that the exercise was discontinued. • After sixty (60) seconds, when the TIMER alarm goes off say: “Yima”. Place a bracket (]) after the last word that the learner has read correctly. • Count and record the number of words that the learner read correctly. • If the learner reads all the words in less than sixty (60) seconds, record the number of seconds remaining on the timer. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 97 | Page WORD READING ISIZULU GRADE 2, 3 AND 4 Isibonelo: into jika ikati itiye vela lala khala /5 hleka cula imali qala umoba /10 indoda izitsha amanzi xoxa iqanda /15 phuza ubisi incwadi indlebe umhlobo /20 baleka umlimi kuqala ingulube igatsha /25 umntwana inkomishi abafundi impama inkosikazi /30 isicabha izinzwane izaqathe impukane ingonyama /35 inxiwa ingxube ncibilika ifasitela isigqoko /40 mpilonhle isibhedlela uvemvane uMgqibelo intshonalanga /45 umshushuluzo intothoviyane iphephandaba isicabucabu ukuqhenya /50 uMvelinqangi ubukhwebezane umgqugquzeli echwebeni inxanxathela /55 Make a tick mark (  ) if task was discontinued because learner couldn’t read Number of words read in 60 seconds Total number of words read correctly in 60 seconds If task completed in less than 60 seconds, indicate time remaining (in seconds) USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 98 | Page ORAL READING FLUENCY • Show the learner the reading passage for their Grade and language. Manje sengizocela ukuthi ungifundele lendaba ngephimbo eliphezulu. Uma uhluleka, yeqa lelogama uqhubekele phambili ngokufunda. Mangithi ‘yima’, yima ukufunda lendaba. Ngizolandelisa ngokukubuza imibuzo ngalendaba osandakuyifunda – ngakoke zama ukukhumbula lendaba oyifundayo. Uzoqalalapha. (Point to the first word of the passage.) • Usulungile? Qala. • Start the TIMER with 60 seconds on the tablet when the learner starts. • Strike a line through words that the learner reads incorrectly or cannot read at all. For example: little • If the learner stops for more than three (3) seconds, tell the learner to go on and strike a line through the word. For example: little • If the learner corrects himself/herself, accept it as correct. (If a strike has already been made on the word, circle it to mark it correct.) • If the entire first line has strike-through lines across all the words, stop the assessment, place a bracket (]) after the last word on the first line and make a tick mark () at the bottom of the exercise (in the box provided) to record that the exercise was discontinued. • After 60 seconds when the TIMER alarm goes off, place a bracket (]) after the last word that the learner has read and say Yima. Thank the child and discontinue the task. • Count and record the number of words that the learner read correctly in 60 seconds. • If the learner reads the whole story in less than 60 seconds, record the number of seconds remaining on the timer. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 99 | Page ORAL READING FLUENCY (ORF) ISIZULU GRADE 2 Words read 3 UJabu wayenenja encane. /3 7 Bekuyinja ekhuluphele futhi ejabulile. /7 14 Ngelinye ilanga uJabu nenja yakhe baya kodlala /14 17 enkundleni engemuva komuzi. /17 22 Inja yabona unogwaja yase iyawujaha. /22 24 Inja yalahleka. /24 28 UJabu wayimemeza kodwa yangabuya. /28 36 UJabu wagcwala izinyembezi emehlweni akhe wase ephindela ekhaya. /36 43 Kodwa kwathi ngaphambi kokushona kwelanga inja yabuya. /43 46 Yayikhathele futhi ilambile. /46 51 UJabu wajabula ukubona umngani wakhe. /51 56 UJabu wanika inja ukudla namanzi. /56 Make a tick mark (  ) if task was discontinued because learner couldn’t read any of the first line Number of words read in 60 seconds Total number of words read correctly in 60 seconds If task completed in less than 60 seconds, indicate time remaining (in seconds) USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 100 | Page ORAL READING FLUENCY (ORF) ISIZULU GRADE 3 Words read 3 Kunesihambi esasilambile kakhulu. /3 10 Sahamba sicela emizini yabantu. Abantu babengenakho ukudla. /10 20 Isihambi sathola isu. Isihambi sathola ibhodwe. Sathatha amatshe sawafaka ebhodweni. /20 27 Sathela amanzi. Sabasa umlilo, sabeka ibhodwe eziko. /27 32 Sama salinda ibhodwe laze labila. /32 40 Kwafika intombazane yacela ukwazi ukuthi siphekani isihambi eziko. /40 44 “Ngipheka isobho elimnandi lamatshe. /44 52 Kodwa kumele ngilifake into ukuze linongeke,” kusho isihambi. /52 59 “Nginezaqathe mina,” wabe esenika isihambi. Sazifaka ebhodweni. /59 Make a tick mark (  ) if task was discontinued because learner couldn’t read any of the first line Number of words read in 60 seconds Total number of words read correctly in 60 seconds If task completed in less than 60 seconds, indicate time remaining (in seconds) USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 101 | Page ORAL READING FLUENCY ISIZULU GRADE 4 Words read 6 Kwasukasukela kudala, isihambi safika ndawana thize. /6 10 Sasingenakho ukudla, sesifile yindlala. /10 17 Sasuka sangena umuzi nomuzi sicela okuya ethunjini. /17 22 Kodwa bonke abantu basendaweni babempofu, /22 30 batshela isihambi ukuthi okukhona ukudla kwanele bona bodwa. /30 37 Isihambi sasuka sahamba, singazi ukuthi sizokwenzani ngokulandelayo. /37 43 Sesisukile endaweni, sathola isiqu esibanzi sesihlahla, /43 48 sahlala phansi sadla amathambo engqondo. /48 52 Sabona ibhodwe lobumba otshanini. /52 57 Lokhu kwasenza safikelwa yisu elithile. /57 63 Isihambi sasuka lapho sacoshela izinkuni zokubasa. /63 69 Sokhela umlilo sabeka ibhodwe phezu komlilo /69 73 saligcwalisa ngamanzi acwebile asemfuleni. /73 80 Sahamba sayobutha amatshe amakhulu. Sawageza sawafaka ebhodweni. /80 Make a tick mark (  ) if task was discontinued because learner couldn’t read any of the first line Number of words read in 60 seconds Total number of words read correctly in 60 seconds If task completed in less than 60 seconds, indicate time remaining (in seconds) USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 102 | Page ORAL READING COMPREHENSION • Ngizokubuza imibuzo embalwa ngalendaba osandakuyifunda. Zama ukuphendula imibuzo ngendlela ozokwazi ngayo. • If the learner read only part of the story, only ask the questions related to the part that s/he has read. Enter the option D (Don’t Know) option for questions not covered. • If the learner corrects himself/herself, accept the answer as correct. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 103 | Page ORAL READING COMPREHENSION ISIZULU GRADE 2 Imibuzo Words read Izimpendulo Correct Incorrect No response 1. Ngekabani inja kule ndaba? 3 NgekaJabu 2. Wazi kanjani ukuthi uJabu wayeyithanda inja yakhe? 7 Inja yayikhuluphele futhi ijabulile/Wayeyinakekela kahle/babedlala ndawonye 3. Baya kodlala kuphi nenja yakhe? 17 Enkundleni ngemuva komuzi 4. Yisiphi isilwane esajahwa yinja? 22 Unogwaja 5. Kungani inja yalahleka? 24 Yalahlekelwa yindlela ngenkathi igijimisa unogwaja 6. Kungani uJabu wayegcwele izinyembezi emehlweni akhe? 36 Wayedabukile/ekhala ngoba inja yakhe ilahlekile/wayengayitholi inja yakhe. 7. Yabuya nini inja ekhaya? 43 Ngaphambi kokushona kwelanga 8. Ucabanga ukuthi kungani inja yayilambile ngenkathi ifika ekhaya? 46 Yayilahlekile/yalamba yonywa ngenkathi izama ukuthola indlela yokubuyela 9. Wayiphani uJabu inja yakhe ngenkathi ifika ekhaya? 56 Ukudla namanzi. 10. Ngubani umngani kaJabu? 51 Inja yakhe encane. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 104 | Page ORAL READING COMPREHENSION ISIZULU GRADE 3 Umbuzo Words read Impendulo Correct Incorrect No response 1. Ngubani owayelambile? 3 Isihambi. 2. Yini abantu ababengenayo? 10 Babengenakho ukudla. 3. Senzani isihambi ngamatshe? 20 Sawafaka ebhodweni. 4. Yini enye into eyenziwa isihambi? 27 Sathela amanzi ebhodweni / sabasa umlilo / sabeka ibhodwe eziko. 5. Sasimeleni isihambi eduze kwebhodwe? 32 Sasilinde ukuthi libile ibhodwe. 6. Yabuza yathini intombazane kwisihambi? 40 Yabuza ukuthi siphekani eziko. 7. Kungani intombazane yanika isihambi izaqhathe? 59 Yayifuna ukusiza ukuze isobho libemnandi. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 105 | Page ORAL READING COMPREHENSION ISIZULU GRADE 4 Umbuzo Words read Impendulo Correct IncorrectNo response 1. Yenzeka kuphi lendaba? 6 Emandulo 2. Yini indaba isihambi sasingena umizi nomuzi? 17 Wayelambile efuna ukudla 3. Yini indaba izakhamuzi zazingayiniki lendonda ukudla? 30 Babempofu bengenakho ukudla 4. Yini indaba yahlalaphansi lendoda yadla amathambo enqondo? 48 Wayelambile edinga ukucabanga indlela yokuthola ukudla 5. Yabonani indoda ngesikhathi ihleli otshanini? 52 Ibhodwe elakhiwe ngobumba 6. Yayizichoshelani izinkuni ? 69 Wayefuna ukubasa umlilo abeke ibhodwe 7. Yini yayifaka kuqala ebhodweni? 73 Amanzi 8. Yayiwahlanzelani kuqala amatshe? 80 Wayengafuni ukufaka amatshe angcolile emanzini abilayo USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 106 | Page VOCABULARY • Show the learner the Vocabulary Show Chart for the language you are testing. • Bheka igama elingaphakathi kwebhokisi. Ngikhombise isithombe okuyisona esifanele. • Example: show the pictures and point to the dog. • Manje, zama ngokwakho. Let learner try it. • Uyaqondisisa? If not, explain again. • If the learner has not given an answer in approximately 10 seconds, tell them to move to the next item. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 107 | Page VOCABULARY ISIZULU GRADES 2, 3 & 4 Examples: inja amehlo USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 108 | Page isandla inkukhu izicathulo ibhulukwe indlulamit hi inkanyezi USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 109 | Page SPS Evaluation Baseline Learner Survey Name of Learner Gender Boy Girl Grade and class Grade Class Is this child currently wearing spectacles/glasses? Yes No Teacher’s name Date of Assessment Day Month Year Name of evaluator A. Learner context Do not read the response options aloud unless otherwise instructed. Wait for the learner to respond then write this response in the space provided, or circle/tick the code of the response option that corresponds to the learner’s response. If there is no special instruction to the contrary, only one response is permitted. Now I am going to ask you a few questions about yourself. No. Questions Response Options Code A1 How old are you? __ Years Don’t know _ _ 99 USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 110 | Page No. Questions Response Options Code A2 What is your date of birth? (Enter 99/99/9999 if the child does not know their date of birth. If a child knows some parts of their date of birth (e.g. day and month but not year, then enter 99 for the parts they don’t know) DD MM YYYY _ _ / _ _ / _ _ _ _ A3 Who do you stay with? (Tick apply that apply) Mother Father Grandparent Aunt Uncle Older sibling Guardian/Other Caregiver Other Do not know/No response 1 2 3 4 5 6 7 8 9 A4 What language do you speak most often at home? Zulu Xhosa Afrikaans English --> skip to Question A7 S.Sotho (Sesotho) N. Sotho (Sepedi) Tswana Venda Tsonga si Swati Ndebele Other Do not know/No response 1 2 3 4 5 6 7 8 9 10 11 12 99 A5 Do you ever speak English at home? Yes No Don’t know/No response 1 0 9 A6 Do you ever watch TV or listen to the radio in English? Yes No Don’t know/No response 1 0 9 A7 In the classroom, what languages does your teacher use to teach? (Tick all that apply) Zulu Xhosa Afrikaans English Other (specify) Do not know/No response 1 2 3 4 5 9 A8 What language do you speak when you play with your friends? (Tick all that apply) Zulu Xhosa Afrikaans English Other (specify) Do not know/No response 1 2 3 4 5 9 USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 111 | Page No. Questions Response Options Code A9 What grade were you in last year? Grade R Grade 1 Grade 2 Grade 3 Grade 4 Not in school Don’t know/No response 1 2 3 4 5 6 9 A10 Is there any day you did not come to school last week? Yes No (skip to A12) Don’t know/No response (skip to A12) 1 0 9 A11 If yes, how many days did you not come to school last week? (number of absent days) Enter 9 for Don’t know ____ A12 Is there any day your teacher did not come to school last week? Yes No Don’t know/No response 1 0 9 A13 Did you eat before school this morning? Yes No Don’t know/No response 1 0 9 A14 Apart from schoolbooks, do you have any other books that you can read at home? Yes No (skip to A16) Don’t know/No response (skip to A16) 1 0 9 A15 Look at the pictures, how many books would you say that you have at home? (show picture with book piles) Few books Some books Many books A lot of books Don’t know/No response 1 2 3 4 9 A16 Household Possessions (show pictures of household possessions) a. Is there a radio at your home? Yes No Don’t know/No response 1 0 9 b. Does anyone at your home have a mobile phone? Yes No Don’t know/No response 1 0 9 c. Do you have electricity at your home? Yes No Don’t know/No response 1 0 9 d. Is there a television at your home? Yes No Don’t know/No response 1 0 9 e. Is there a computer at your home? Yes No Don’t know/No response 1 0 9 f. Is there a refrigerator at your home? Yes No Don’t know/No response 1 0 9 g. Is there a toilet inside your home? Yes No Don’t know/No response 1 0 9 USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 112 | Page No. Questions Response Options Code h. Does anyone at your home have a bicycle? Yes No Don’t know/No response 1 0 9 i. Does anyone at your home have a car, or a truck? Yes No Don’t know/No response 1 0 9 A17 The next questions ask about difficulties you may have doing certain activities because of a HEALTH PROBLEM. a. Do you have difficulty seeing, even if wearing glasses? No – no difficulty Yes - some difficulty Yes – a lot of difficulty Cannot do at all Don’t know/No response 1 2 3 4 9 b. Do you have difficulty hearing, even if using a hearing aid? No – no difficulty Yes - some difficulty Yes – a lot of difficulty Cannot do at all Don’t know/No response 1 2 3 4 9 c. Do you have difficulty walking or climbing steps? No – no difficulty Yes - some difficulty Yes – a lot of difficulty Cannot do at all Don’t know/No response 1 2 3 4 9 d. Do you have difficulty remembering or concentrating? No – no difficulty Yes - some difficulty Yes – a lot of difficulty Cannot do at all Don’t know/No response 1 2 3 4 9 e. Do you have difficulty (with self￾care such as) washing all over or dressing? No – no difficulty Yes - some difficulty Yes – a lot of difficulty Cannot do at all Don’t know/No response 1 2 3 4 9 f. Using your usual (customary) language, do you have difficulty communicating, for example understanding or being understood? No – no difficulty Yes - some difficulty Yes – a lot of difficulty Cannot do at all Don’t know/No response 1 2 3 4 9 USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 113 | Page B. Reading Practices Now, I am going to ask you some questions. If you do not want to answer any question, or do not know the answer to a question, you do not have to answer. Can I begin asking you these questions? I am going to ask you some questions about what you like and do not like. I’m going to show faces. SHOW SMILEY FACE SCALE. Explain the scale. This face:  means that you really something very much. This face (point to face 2) means that you like it. This face means that you don't care (point to face 3); you don't particularly like it but you don't dislike it either. This one means that you don't like it (point to face 4). This face (point to face 5) means you really don't like it at all (hate it?) When I ask you a question you will point to the face that shows how you feel. 1 2 3 4 5 No. Questions Response Options Code B1 [SHOW SMILEY FACE SCALE] Do you like to play? Point to how you feel. Very happy Happy Indifferent Sad Very sad Don’t know/No response 1 2 3 4 5 9 B2 [SHOW SMILEY FACE SCALE] Do you like to go to school? Point to how you feel. Very happy Happy Indifferent Sad Very sad Don’t know/No response 1 2 3 4 5 9 B3 [SHOW SMILEY FACE SCALE] Do you like to listen to stories? Point to how you feel. Very happy Happy Indifferent Sad Very sad Don’t know/No response 1 2 3 4 5 9 B4 [SHOW SMILEY FACE SCALE] Do you like to tell stories? Point to how you feel. Very happy Happy Indifferent Sad Very sad Don’t know/No response 1 2 3 4 5 9 USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 114 | Page No. Questions Response Options Code B5 [SHOW SMILEY FACE SCALE] Do you like to look at books? Do you like to look at the pictures, at the words, or play with books? Point to how you feel. Very happy Happy Indifferent Sad Very sad Don’t know/No response 1 2 3 4 5 9 B6 [SHOW SMILEY FACE SCALE] Do you like to read? Point to how you feel. Very happy Happy Indifferent Sad Very sad Don’t know/No response 1 2 3 4 5 9 B7 Do you read on your own at home? [DO NOT READ THE OPTIONS OUT LOUD] Yes No (skip to B10) I cannot read (skip to B10) Don’t know/No response 1 2 3 9 B8 [SHOW SMILEY FACE SCALE] Do you like to read on your own at home? Point to how you feel. Very happy Happy Indifferent Sad Very sad Don’t know/No response 1 2 3 4 5 9 B9 How often do you read on your own at home? Everyday or almost everyday Two to four days a week One day a week Once or twice a month Less than once a month Never Don’t know/No response 1 2 3 4 5 6 9 B10 Do you ever read on your own in school? [DO NOT READ THE OPTIONS OUT LOUD] Yes No (skip to B12) I cannot read (skip to B12) Don’t know/No response 1 2 3 9 B11 [SHOW SMILEY FACE SCALE] And, do you like to read on your own in school? Very happy Happy Indifferent Sad Very sad Don’t know/No response 1 2 3 4 5 9 B12 Do you read in class out loud? [DO NOT READ THE OPTIONS OUT LOUD] Yes No (skip to B15) I cannot read (skip to B15) Don’t know/No response (skip to B15) 1 2 3 9 B13 [SHOW SMILEY FACE SCALE] Do you like to read in class out loud? Very happy Happy Indifferent Sad Very sad Don’t know/No response 1 2 3 4 5 9 USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 115 | Page No. Questions Response Options Code B14 How often do you read in class out loud? Everyday or almost everyday Two to four days a week Less than once a week Don’t know/No response 1 2 3 9 B15 Does anyone at home tell you stories? Yes No Don’t know/No response 1 2 9 B16 Does anyone read with you at home? Yes No (skip to B20) Don’t know/No response (skip to B20) 1 2 9 B17 Who do you read with at home? [SELECT ALL THAT APPLY] Mother Father Stepmother Stepfather Aunt Uncle Grandfather Grandmother Sister Brother Other Specify ________ 1 2 3 4 5 6 7 8 9 10 11 B18 How often do you read with someone at home? Four days a week or more Two or three days a week One day a week Once or twice a month Less than once a month Never Don’t know/No response 1 2 3 4 5 6 9 B19 [SHOW SMILEY FACE SCALE] Do you like to read with someone at home? Point to how you feel. Very happy Happy Indifferent Sad Very sad Don’t know/No response 1 2 3 4 5 9 B20 [SHOW SMILEY FACE SCALE] Do you like to write? Point to how you feel. Very happy Happy Indifferent Sad Very sad Don’t know/No response 1 2 3 4 5 9 B21 Do you ever write things just because you like it, even if the teacher doesn't ask you to do it? Yes No (skip to C1) Don’t know/No response (skip to C1) 1 2 9 USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 116 | Page No. Questions Response Options Code B22 What do you like to write? [SELECT ALL THAT APPLY] Words/sentences Letters Poems/Songs Stories/Plays Books Other Don’t know/No response 1 2 3 4 5 6 9 No. Questions Response Options Code C1 Learner’s height (cm) ____ ____ ____ . ____ C2 Was the assessment done in appropriate conditions? Yes, it was quiet, there were no interruptions and no distracting noises or activities around. Most of the time. There were some minor interruptions or distractions but didn’t create a problem No, there were interruptions or noises that interfered with the assessment. 1 2 3 OBSERVATIONS ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ___________________________________________________________ USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 117 | Page Measuring Height Disassembled stadiometer 1. When assembling the stadiometer be sure to match the sections together properly. Look for matching symbols at the end of each measuring piece. Failure to assemble the stadiometer properly will result in incorrect measurements. 2. Stand the learner next to the stadiometer. Explain that you are going to measure their height. 3. Ask and/or help learner to stand: • up straight • feet and heels together • heels back against the upright section of the stadiometer • arms relaxed (floppy) by sides. 4. Ask learner to look straight ahead. Arms should be relaxed by sides, with palms facing inwards. Gently position their head: hold your hand around their jaw, so that the top of their head and the stadiometer form a right angle and their chin is not pointing down towards their chest. 5. Check that their head is still positioned correctly and their heels are still flat on the floor. Bring the head plate down to rest on the learner’s head. 6. Crouch down so your eye is level with where the red arrow in the reading window. Record the measurement to nearest 0.1cm. Make a note of this measurement immediately. Sliding head plate Foot plate Reading window with arrow pointer at exact height USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 118 | Page In this example the measurement is 162.0 cm USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 119 | Page Story Powered Schools: A South African Reading Revolution Hello, my name is ____________________________ . I am working with NORC at University of Chicago and the University of Cape Town, on a research study on reading in collaboration with the Department of Basic Education and USAID. We are undertaking a study to better understand how children learn to read. Along with assessing the reading abilities of learners, we are interviewing teachers regarding teaching about reading. We would now like to ask you a few questions about your classroom and your preparation and support for teaching about reading. If you agree to participate in the survey, all the observations and answers that you provide will be kept confidential. Though the information acquired through this data collection process will be shared with the Department of Basic Education with the hope of identifying areas where additional support may be needed, your school will not be identified by name nor will you be identified individually by name. There are no known risks or dangers to you associated with this study. This research has been approved by the Institutional Review Board at NORC and the Commerce Faculty Ethics in Research Committee at the University of Cape Town. The research has also been approved by the Department of Basic Education. You are free to choose not to answer any question and you can stop the interview at any time, ask me to clarify questions or ask me to repeat something you did not hear or understand. You can also withdraw from the survey at any time. Stopping the survey, not answering any question or answering the full survey will not impact you negatively in any way, nor will it bring you any additional benefits. However, your honest answers to our questions will help us understand better how learners learn to read. We greatly appreciate your cooperation. This interview will take about 20 minutes. Do you have any questions for me now? Do you understand that you may refuse to participate or stop participating at any time without penalty? Do you agree to participate? Check box if verbal agreement is obtained:  YES (If verbal agreement is not obtained, thank the teacher and move on to the next teacher, using this same form.) USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 120 | Page SPS Evaluation Baseline Teacher Interview Enumerator Name Teacher Name Type of school? Monograde Multi-grade Teacher Interview - Part 1 1) What class do you teach? Check all that apply if multi-grade school. Grade R Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 2) What is your gender? Male Female Other__________ Prefer not to say/Refuse 3) What is your age? |_ _| YEARS Enter a number from 0 to 99. If unsure of exact age, have respondent estimate. Don't know/ Prefer not to say/Refuse 4a) What is your mother tongue? isiZulu isiXhosa Afrikaans English Southern Sotho (Sesotho) USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 121 | Page Northern Sotho (Sepedi) Tswana Venda Tsonga siSwati Ndebele Other (specify in 4b) 4b) If other, please specify: ___________________________ 5) What is the mother tongue of the majority of learners in your class? isiZulu isiXhosa Afrikaans English Southern Sotho (Sesotho) Northern Sotho (Sepedi) Tswana Venda Tsonga siSwati Ndebele 6a) What is the language of learning and teaching (LOLT) in the grade that you teach in this school? If there is more than one LOLT in the school, then we want to know the LOLT in the class they teach. If it is a multi-grade school and the LOLT is different for different grades that they teach then we want to know the LOLT in the lowest grade that they teach. isiZulu isiXhosa Afrikaans English Other (specify in 6b) 6b) If other, please specify: __________________________ 7) Do you speak this language? Yes, very well Yes, but not fluently No Refuse 8) Have you received any instruction on teaching in this language? Yes No Don't know/Refuse 9a) Is this the same language that you use in your class for teaching most of the time? Yes -->skip to 10 No Refuse USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 122 | Page 9b) If no, what language do you use for teaching most of the time? isiZulu isiXhosa Afrikaans English Other (please specify) 9c) If other, please specify___________________________. 10) Now I would like to know about all the languages that you use to teach. Which languages do you use? Check all that apply. isiZulu isiXhosa Afrikaans English Other (please specify) 11a) What is your highest level of academic or professional qualification? Less than Grade 12 Grade 12/Matric Less than Grade 12 with certificate Grade 12 with certificate Diploma Degree PhD Other (please specify) Don't Know / Refuse 11b) If other, please specify: 12) What is your highest professional qualification in teaching? Diploma in Grade R Teaching Bachelor of Education Degree (B.Ed.) Postgraduate Certificate in Education (PGCE) Advanced Certificate Diploma Advanced Diploma Postgraduate Diploma Bachelor of Education Honours Degree Master of Education Degree Professional Master's Degree in Education Doctoral Degree in Education Professional Doctoral Degree in Education No teaching qualification Other: _______________________________ Don’t know / Refuse 13) How many years have you been teaching overall? |_ _| YEARS Enter a number from 0 to 99. Don’t know / Refuse USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 123 | Page 14) Think about the last 5 school days. How many days, if any, were you unable to come to school? Please do not count days where the entire school was closed for a holiday or another reason. |_| DAYS MISSED Enter a number from 0 to 5. Don't know/refuse 15) How often do read outside of work requirements? Very often Often Sometimes Hardly ever Never, there is not enough time - skip to 17 Never, I don’t enjoy reading  skip to 17 Don’t know/Refuse - skip to 17 16a) What do you read? Check all that apply Novel/fiction Non-fiction books Magazines Children’s books Newspapers Bible or religious texts Online news and information Other 16b) If other, please specify 17) Approximately how many books do you have in your home? Read options None (0) Very few (1 to 10) Enough to fill a shelf (11 to 25) Enough to fill a bookcase (25-100) Enough to fill more than one bookcase (more than 100) Don’t know/Refused 18) Aside from textbooks, readers and other school books, what is your favourite children’s book? Part 2 19) Since you started working as a teacher, have you ever attended training in how to teach reading? Yes No Don't Know / Refuse USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 124 | Page 20) Which of the following were covered in the training? Read out all options. Circle all that apply. How to teach phonics in African languages How to teach phonics in English How to motivate children to read How to select appropriate, interesting storybooks for children 21) How often does the Principal, Deputy Principal, HOD, lead teacher, subject advisor or anyone else at the school check or review your lesson plans? Never Once a year Once per term Once every month Once every week Daily Don't Know / Refuse 22) Aside from IQMS, how frequently does your Principal, Deputy Principal, HOD, lead teacher, subject advisor or someone else at this school observe your teaching? Never Once a year Once every term Once every month Once every week or more frequently Don't Know / Refuse 23) How many learners are in your class? |_ _ _| LEARNERS Don’t know / Refuse 24a) Do the learners in your class have readers to read from? Yes No ---> skip to 28 Don't Know / Refuse ---> skip to 28 24b) How many isiXhosa/isiZulu readers do you have in your classroom? None 0-10 11-20 21-40 More than 40 Don’t know 24c) How many English readers do you have in your classroom? None 0-10 11-20 21-40 USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 125 | Page More than 40 Don’t know 25) I would like to know about the readers learners in your class have to read from. Please check all that apply. The learners in my class have readers but they have to share. There are enough readers for each child to have their own. There are one or more sets of graded readers for the learners to read from. Learners are sometimes allowed to choose their own reader to read from. Learners are allowed to take readers home. 26) What is your opinion about these readers? Would you say they are: Read options Very good Good Fair Poor Very poor Don't Know / Refuse 27) How often do you use these readers? Read options Every day A few times a week Sometimes a month Rarely Never Don't Know / Refuse 28) Do you have access to other books, textbooks or reading material to use in the lesson to read to the learners or to share with them? Yes No Don't Know / Refuse 29) Is there a bookshelf or small library in your classroom for the learners to use? Yes No Don't Know / Refuse 30) Does the school have a library, or get visited by a mobile library? Yes No -- skip to 33 Don't Know / Refuse -- skip to 33 USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 126 | Page 31) Does the library have books that you feel have appropriate content and language for the learners in your class? Yes No Don’t Know / Refuse 32) Are the learners allowed to take readers or story books from the library home with them? Yes No Don't Know / Refuse 33) Do you ever do reading activities with the learners using Nal’ibali supplements? Yes No Don't Know / Refuse 34) What kind of support would be most useful for you to improve your ability to teach reading? Mark all that apply Training In-class support Materials/books Observation and feedback Other: _____________________ Don't Know / Refuse 35a) Do you have any assistants or volunteers who help you with reading activities in class? No-one Volunteers Assistants HOD Other teachers Other 35b) If other, please specify __________________________________________________________________________ Part 3 36) I would like to ask you about activities you might do with your learners in class. Think about the last 5 teaching and learning days. How often did each of the following activities take place? Read each activity and frequency options. If it is a multi-grade school, report on activities for the lowest grade that you teach. Never 1 day a week 2 days a week 3 days a week 4 days a week 5 days a week a. The whole class repeated sentences that you said first. 0 1 2 3 4 5 USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 127 | Page Never 1 day a week 2 days a week 3 days a week 4 days a week 5 days a week c. Learners retold a story that they read. 0 1 2 3 4 5 d. Learners sounded out unfamiliar words. 0 1 2 3 4 5 e. Learners learned meanings of new words. 0 1 2 3 4 5 f. Learners read aloud (individually) to teacher or to other learners. 0 1 2 3 4 5 g. Learners read with the teacher in small ability groups with the same reader 0 1 2 3 4 5 h. Learners were assigned reading to do on their own during school time. 0 1 2 3 4 5 i. Learners were assigned reading to do on their own at home 0 1 2 3 4 5 k. Learners played games related to the literacy/reading lesson 0 1 2 3 4 5 l. You read a story to the learners 0 1 2 3 4 5 m. You discussed a story with the learners 0 1 2 3 4 5 o. Learners used a dictionary 0 1 2 3 4 5 q. Learners answered reading comprehension questions about what they have read 0 1 2 3 4 5 37) How do you determine if your learners are grasping the content of the lesson/s? Do not provide response options. Tick all that apply. Learners are not evaluated Routine written tests (weekly, monthly, etc) Routine oral tests (weekly, monthly, etc) Lesson recitation Homework End of term evaluation End of Year examinations Other_________________________ Don’t know/Refuse 38) Think about your class, how many learners are reading below their grade level? Read options None/almost none A few/less than half About half Many/more than half All/almost all Don’t know/Refuse 39) What percentage of learners in your class can read fluently AND with understanding in isiXhosa/isiZulu? 0-10% 11-20% 21-30% 31-40% 41-50% 51-60% 61-70% 71-80% 81-90% 91-100% USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 128 | Page 40) What percentage of learners in your class can read fluently AND with understanding in English? 0-10% 11-20% 21-30% 31-40% 41-50% 51-60% 61-70% 71-80% 81-90% 91-100% 41) What support, if any, do you give to learners who are unable to keep up with the language/reading lessons? Do not provide response options. Tick all that apply Not applicable (no learners needed support) No support was given I conduct a whole class revision I give the learner individualised remedial support inside the class I give the learner individualised remedial support outside the class I assign the learner additional practice time during school hours I give the learner additional reading materials or work assignments as homework I have other learner/s work on reading with the learner having difficulty I communicate the problem to the parents/caregivers I ask the parents to help the learner with reading I run a reading club Other ____________________________ Don’t know/Refuse 42) Think about your class, how many learners do you think will not have learnt to read with understanding by Grade 6? None/almost none A few/less than half About half Many/more than half All/almost all Don’t know/Refuse 43) Do you know about the “Drop Everything and Read” (DEAR) campaign? Yes No Don't Know / Refuse 44) Do you have a specific period in your timetable where everyone across the school reads at the same time? Yes No Don't Know / Refuse USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 129 | Page 45) Which of these statements comes closest to how you feel? Schools are entirely responsible for teaching children how to read Schools are mostly responsible for teaching children to read, but parents should help occasionally Both schools and parents are responsible for teaching children how to read Don’t know / no response 46) Do you agree or disagree with each of the following statements? Agree Disagree Reading to children before they know how to talk or read helps them learn. Children need English for their future so there is no point teaching them to read in an African language. Reading for enjoyment is nice for children, but it doesn’t help achieve CAPS outcomes. There’s no point reading to children until they have learned to read themselves. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 130 | Page SPS Evaluation Baseline School Inventory 1) Are the school buildings and grounds clean and neat? Yes No 2) Does the school have electricity? If YES, is it functioning today (i.e. the day of the learner visit)? No Yes, but not functioning today Yes, and functioning today 3) What is the main source of water for the school? Municipal or communal 1 Borehole/ Well on site 2 Rainwater harvest 3 Mobile Tankers 4 Other 5 None/No water supply 6 4) What type of toilets/latrines does the school have? (Tick all that apply) Flush toilet municipal 1 Flush toilet septic tank 2 VIP (Ventilated Improved Pit) 3 Pit Latrine 4 Enviro Loo 5 Chemical 6 Mobile 7 Other 8 None/No facilities 9 skip to 8 5) Are there private/separate toilet facilities for girls? Yes No 6) Is there a library (either in the classroom or for the whole school or mobile)? If YES, are students using the library at the time of visit? USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 131 | Page No, there is no library Yes, but no students are using it Yes, and students are using it 7) Which of the following does the school have in working order? [Tick all that apply] Mobile phone 1 Landline 2 Internet for administrative purposes 3 Internet for teaching and/or learning 4 None 5 8) Does the school have a computer lab? [A COMPUTER LAB IS A SPACE THAT HAS AT LEAST 5 COMPUTERS, IN WORKING ORDER, THAT ARE NORMALLY USED BY THE LEARNERS] If YES, are the computers connected to the internet NO YES and the computers are connected to internet YES but there is no internet connection 9) Is there a functional chalk board/white board/smart board in each classroom? YES NO 10) Is there a school nutrition programme at this school? YES NO - skip to 13 11) Does it feed all learners or just selected learners? All learners Selected learners 12) On the day of the learner visit, did the school nutrition programme impact on teaching and learning time: By more than 20 minutes By less than 20 minutes Did not see the school nutrition programme during visit 13) Does the school have a boundary fence or wall? No fencing Palisade Wire fence Razor wire fence Pre cast Brick wall 14) Does the school have any security? (Tick all that apply) No security Electric fence Gate Access control Security room guard USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 132 | Page Alarm system Security lights Surveillance cameras 15) What type of school is this? Monograde Multigrade 16) What grades are taught in this school? Check all that apply Grade R Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 17) How many learners are there in this school in total? _________ 18) How many teachers are there in this school in total? _________ 19) How many Grade 2 learners are there? _________ 20) How many Grade 3 learners are there? _________ 21) How many Grade 4 learners are there? _________ 22) How many Grade 2 teachers are there? _________ 23) How many Grade 3 teachers are there? _________ 24) How many Grade 4 teachers are there? _________ 25) What is the language of learning and teaching (LOLT) in the foundation phase (Grades R to 3) in this school? isiZulu isiXhosa English Afrikaans Other (specify) USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 133 | Page 26) What is the language of learning and teaching (LOLT) in the intermediate phase (Grades 4 to 7) in this school? isiZulu isiXhosa English Afrikaans Other (specify) USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 134 | Page ANNEX IV: ADDITIONAL ANALYSES USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 135 | Page FULL DISTRIBUTION OF EGRA SCORES Table A1: Percentiles of EGRA subtasks Eastern Cape KwaZulu-Natal Percentile Percentile 10th 25th 50th 75th 90th 10th 25th 50th 75th 90th Letter sounds Grade 2 1 7 17 30 42 0 3 12 24 36 Grade 3 5 17 31 45 56 0 5 16 28 41 Grade 4 3 18 34 49 60 0 7 17 28 41 Familiar words Grade 2 0 0 2 11 20 0 0 7 15.5 22 Grade 3 0 4 16 26 32 0 8 20 27 32 Grade 4 1 12 25 33 40 3 17 26 33 38 Oral reading Grade 2 0 0 1 10 19 0 0 4.5 15.5 24 Grade 3 0 0 12.5 21 31 0 7 20 31 39 Grade 4 0 8 21 29.5 35 0 16 25 36 43 English Letter sounds Grade 3 0 12 26 38 48 0 5 16 29 39 Grade 4 0 14 31 43 55 0 7 19 30 42 English familiar words Grade 3 0 1 10 22 31 0 4 15 25 32 Grade 4 0 7 22 32 41 2 12 25 33 41 English oral reading Grade 4 0 10 34 51 68 0 18 39 57 79 USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 136 | Page Figure A1: Distribution of EGRA subtasks by grade and province USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 137 | Page USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 138 | Page USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 139 | Page USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 140 | Page USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 141 | Page USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 142 | Page USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 143 | Page Table A2: Correlation between EGRA subtasks Panel A: Eastern Cape Correct words per minute (cwpm) Oral reading fluency (orf) Correct letter sounds per minute (clspm) Phonemic awareness percent correct Vocabulary percent correct Reading comprehension percent correct Correct words per minute (cwpm) 1 Oral reading fluency (orf) 0.930 1 Correct letter sounds per minute (clspm) 0.716 0.669 1 Phonemic awareness percent correct 0.525 0.494 0.490 1 Vocabulary percent correct 0.727 0.681 0.597 0.457 1 Reading comprehension percent correct 0.623 0.706 0.496 0.387 0.450 1 Panel B: KwaZulu-Natal Correct words per minute (cwpm) Oral reading fluency (orf) Correct letter sounds per minute (clspm) Phonemic awareness percent correct Vocabulary percent correct Reading comprehension percent correct Correct words per minute (cwpm) 1 Oral reading fluency (orf) 0.933 1 Correct letter sounds per minute (clspm) 0.494 0.472 1 Phonemic awareness percent correct 0.453 0.436 0.441 1 Vocabulary percent correct 0.727 0.680 0.414 0.401 1 Reading comprehension percent correct 0.636 0.711 0.398 0.356 0.497 1 USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 144 | Page Table A3: Cronbach’s apha for EGRA subtasks by grade and province Eastern Cape KwaZulu-Natal Grade 2 0.792 0.790 Grade 3 0.778 0.743 Grade 4 0.730 0.704 Table A4: Summary of EGRA scores excluding learners scoring zero by grade and province – home language Eastern Cape KwaZulu-Natal Grade 2 Grade 3 Grade 4 Grade 2 Grade 3 Grade 4 Print orientation [max=10] Mean 7.05 -- -- 7.93 -- -- Std dev. 2.31 -- -- 2.10 -- -- Correct Letter Sounds per Minute (CLSPM) Mean 21.76 33.13 36.65 18.32 21.38 21.91 Std dev. 14.99 17.76 18.88 13.21 15.03 14.55 Correct Words Per Minute (CWPM) Mean 10.50 19.23 25.31 12.04 20.53 25.82 Std dev. 8.53 11.50 12.74 8.40 10.26 10.69 Oral Reading Fluency (ORF) Mean 10.14 18.28 23.03 12.77 24.31 28.30 Std dev. 9.01 10.70 12.07 9.73 12.26 13.18 Read comprehension [max Gr2=10, max Gr3=7, Gr4=8] Mean 2.04 1.92 1.70 2.04 2.99 1.84 Std dev. 1.36 1.14 0.85 1.39 1.46 0.94 Phonemic awareness [max=8] Mean 2.92 3.75 4.27 3.34 3.99 4.51 Std dev. 1.53 1.90 1.96 1.83 2.06 2.06 Vocabulary [max=6] Mean 3.48 4.60 5.02 4.09 5.11 5.45 Std dev. 1.83 1.62 1.42 1.87 1.42 1.13 USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 145 | Page Table A5: Summary of English EGRA scores excluding learners scoring zero by grade and province Eastern Cape KwaZulu-Natal Grade 3 Grade 4 Grade 3 Grade 4 Correct Letter Sounds per Minute (CLSPM) Mean 29.8 34.4 22.1 23.8 Std dev. 15.5 17.0 14.0 14.1 Correct Words per Minute (CWPM) Mean 17.3 24.3 19.1 25.6 Std dev. 12.1 14.3 11.6 13.2 Oral Reading Fluency (ORF) Mean 39.2 44.4 Std dev. 24.3 26.4 Reading comprehension [max=10] Mean 2.0 2.4 Std dev. 1.6 2.0 Phonemic awareness [max=8] Mean 3.7 4.1 Std dev. 2.0 2.0 Vocabulary [ max=6] Mean 3.6 4.0 Std dev. 1.6 1.5 ORAL READING FLUENCY BENCHMARKS Table A6 below shows the proportion of learners who were asked each comprehension question. Two thirds of the grade 2 learners in the Eastern Cape were asked at least one out of the 10 comprehension question, while the rest did not read enough to participate. On average, the grade 2 learners that participated in the reading comprehension answered 65% of the questions that they were asked correctly. Looking across all the grades, oral reading fluency obviously increases with the number of questions asked but so too does the percentage of learners who are providing correct answers for the comprehension questions. With the increase in oral reading fluency there tends to be an increase in oral reading accuracy (percentage of attempted words that were read correct). For example, among those Eastern Cape grade 3 learners who were only asked one comprehension question, not only was oral reading fluency low at 11.6 but accuracy was also low with mistakes on every fourth word attempted. Those who were asked four or more questions not only were reading much faster (36 correct words per minute) but with much greater accuracy (96%). The percentage of learners USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 146 | Page answering the comprehension questions correctly climbs from 33% for those who were only asked one question to 69% for those answering four or more. Table A6: Comprehension Panel A: Eastern Cape Comprehension questions attempted 0 1 2 3 4 or more Grade 2: % of learners 44.1% 25.8% 0.0% 11.5% 18.6% % of answers correct -- 69.3% -- 52.2% 66.2% Oral reading fluency 0.44 3.46 -- 9.38 19.50 Oral reading accuracy 7.7% 46.5% -- 69.1% 85.3% Grade 3: % of learners 31.8% 40.2% 11.3% 5.9% 10.9% % of answers correct -- 32.5% 65.2% 65.0% 69.1% Oral reading fluency 1.62 11.61 21.55 26.31 36.23 Oral reading accuracy 12.4% 77.9% 93.1% 93.5% 96.4% Grade 4: % of learners 18.1% 40.8% 14.3% 25.8% 1.0% % of answers correct -- 35.6% 62.7% 71.0% 67.5% Oral reading fluency 1.48 14.53 25.40 35.01 69.71 Oral reading accuracy 14.1% 80.7% 93.2% 96.3% 92.6% Panel B: KwaZulu-Natal Comprehension questions attempted 0 1 2 3 4 or more Grade 2: % of learners 27.1% 24.7% 22.1% 9.8% 16.4% % of answers correct -- 51.1% 54.8% 65.3% 75.6% Oral reading fluency 0.07 3.15 8.64 16.78 26.29 Oral reading accuracy 3.2% 47.1% 70.2% 90.1% 97.4% Grade 3: % of learners 15.3% 10.3% 20.1% 18.5% 35.8% % of answers correct -- 9.3% 63.5% 77.8% 78.5% Oral reading fluency 0.07 2.60 13.22 21.36 35.37 Oral reading accuracy 2.6% 40.8% 88.3% 95.7% 98.5% Grade 4: % of learners 12.3% 22.2% 23.4% 38.2% 3.9% % of answers correct -- 33.1% 59.7% 66.4% 74.9% Oral reading fluency 0.55 14.83 22.98 37.62 56.30 Oral reading accuracy 9.6% 85.3% 95.1% 97.5% 98.8% USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 147 | Page Adapting from Spaull et al (2018) we classify learners into a) those who were unable to read one word, b) those who were able to read at least one word of connected text but were unable to correctly answer any of the comprehension questions, c) those who correctly answered at least one but less than 30% of comprehension questions d) those who correctly answered 30 to 59% of comprehension questions and e) those who correctly answered 60% of more of the comprehension questions. Table A7 shows the distribution of the sample by these categories. While there appears to be a drop off in comprehension between Grade 3 and Grade 4, this is because the oral reading passage and associated comprehension questions were designed for each grade level. The Grade 4 passage is more complex with more inferential questions and learners had to read further in the passage to attempt each additional question. Table A7: Comprehension categories Eastern Cape KwaZulu-Natal Grade 2 Grade 3 Grade 4 Grade 2 Grade 3 Grade 4 Unable to read one word 43.1% 29.4% 15.2% 32.0% 19.6% 10.7% 0% comprehension 8.0% 31.3% 29.8% 10.2% 9.4% 20.2% 10-29% comprehension 33.1% 19.7% 44.8% 41.4% 13.0% 54.1% 30-59% comprehension 15.6% 18.5% 10.0% 15.9% 45.0% 14.5% 60%+ comprehension 0.2% 1.1% 0.2% 0.5% 13.0% 0.4% There are no official reading norms or benchmarks for the nine South African indigenous African languages. In a recent study, Spaull et al. (2018) argue that since comprehension is the ultimate goal of reading, it is logical to benchmark to comprehension outcomes. Their empirical investigation “suggests that there are minimum thresholds of accuracy and oral reading fluency in each language, below which it is virtually impossible to read for meaning.” We follow Spaull et al. (2018) and identify the minimum fluency threshold as the 25th percentile of oral reading fluency scores for learners in the 30-59% comprehension category and the minimum comprehension threshold as the 25th percentile of oral reading fluency scores for learners in the 60% plus comprehension category. Figure A2 shows the oral reading fluency distribution by these comprehension categories. While oral reading fluency clearly increases with comprehension score, there is a considerable range in fluency at each comprehension level. For example, a quarter of those learners who were unable to correctly answer one comprehension question, were able to correctly read at least 18 words per minute. The minimum fluency threshold is 20 in Eastern Cape and 23 in KwaZulu-Natal. Minimum comprehension thresholds are 38 and 35 in the Eastern Cape and KwaZulu-Natal respectively. USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 148 | Page Figure A2: Oral reading fluency by comprehension category Figure A3 shows the relationship between oral reading fluency and accuracy. Accuracy rates reach levels of 95% and higher in the oral reading fluency range of 20-30 correct words per minute, which corresponds with our minimum fluency thresholds. Figure A3: Accuracy by correct words per minute in connected text USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 149 | Page LEARNERS, SCHOOL AND TEACHER CHARACTERISTICS Table A8: Learner characteristics by province Eastern Cape KwaZulu-Natal Female 0.496 0.493 Grade 2 0.333 0.331 Grade 3 0.332 0.332 Grade 4 0.335 0.337 Repeating grade 0.072 0.066 Height-for-age z-score -1.021 -0.755 Stunted (z-score less than -2) 0.169 0.117 z-score between -2 and -1 0.349 0.296 z-score between -1 and 0 0.323 0.357 z-score above 0 0.159 0.229 Disability threshold 1 0.455 0.483 Disability threshold 2 0.223 0.265 Disability threshold 3 0.134 0.163 Disability threshold 4 0.011 0.012 Child wearing spectacles 0.015 0.014 USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 150 | Page Table A9: School characteristics by province Eastern Cape KwaZulu-Natal School buildings and grounds clean and neat 0.673 0.807 Toilets: Flush toilet municipal 0.025 0.123 Flush toilet septic tank 0.012 0.107 VIP (Ventilated Improved Pit) 0.068 0.123 Pit Latrine 0.907 0.711 Enviro Loo 0.049 0.011 Chemical 0.000 0.059 Mobile 0.000 0.080 Other 0.000 0.011 None/No facilities 0.019 0.011 Telecommunications: Mobile phone 0.790 0.877 Landline 0.043 0.198 Internet for administrative purposes 0.395 0.166 Internet for teaching and/or learning 0.019 0.032 None 0.136 0.080 Security: Electric fence 0.000 0.005 Gate 0.648 0.850 Access control 0.019 0.037 Security room guard 0.284 0.529 Alarm system 0.037 0.102 Security lights 0.012 0.128 Surveillance cameras 0.025 0.021 No security 0.259 0.080 Computer lab: No 0.918 0.789 Yes, but not connected to the internet 0.070 0.158 Yes, and connected to the internet 0.013 0.052 Functional board in each classroom 0.949 0.984 Nutrition programme 1.000 0.989 Electricity: USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 151 | Page Eastern Cape KwaZulu-Natal No 0.168 0.086 Yes, but not functioning on day of visit 0.149 0.118 Yes, and functioning on day of visit 0.683 0.796 Water source: Municipal or communal 0.074 0.273 Borehole/ Well on site 0.105 0.180 Rainwater harvest 0.463 0.372 Mobile Tankers 0.340 0.126 River 0.006 0.005 None/no water supply 0.012 0.044 Library: No library 0.821 0.661 Yes, but no students are using it 0.049 0.086 Yes, and students are using it 0.130 0.253 Fencing: Palisade 0.025 0.032 Wire fence 0.465 0.385 Razor wire fence 0.396 0.364 Pre cast 0.107 0.219 Brick wall 0.006 0.000 Table A10: Teacher characteristics by province Eastern Cape KwaZulu-Natal Ever received training in teaching reading 0.867 0.789 Number of learners in class 35.48 33.92 Books at home: None 0.113 0.059 Very few (1 to 10) 0.479 0.424 Enough to fill a shelf (11-25) 0.249 0.349 Enough to fill a bookcase (26-100) 0.126 0.137 Enough for more than one bookcase (101+) 0.034 0.030 Do you agree with the following statements: USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 152 | Page Eastern Cape KwaZulu-Natal Reading to children before they know how to talk or read, helps them learn to read 0.960 0.930 Children need English for their future so there is no point in teaching them to read in mother tongue 0.152 0.087 Reading for enjoyment is nice for children but it does not help achieve CAPS outcomes 0.251 0.308 There is no point reading to children until they have learned to read themselves 0.072 0.042 Reading/hearing stories for enjoyment is important for learning to read 0.985 0.996 Do you know about the Drop Everything and Read (DEAR) campaign? 0.755 0.267 Do you have a specific period in your timetable where everyone across the school reads? 0.504 0.621 Do you ever do reading activities with the learners using Na’libali supplements? 0.127 0.227 What support do you give to learners who are unable to keep up with reading? No learners need support 0.000 0.010 No support was given 0.002 0.010 Conduct a whole class revision 0.352 0.308 Give the learner individualised remedial support inside the class 0.636 0.618 Give the learner individualised remedial support outside the class 0.274 0.296 Assign the learner additional practice time inside school hours 0.436 0.370 Give the learner additional reading materials or work assignments as homework 0.362 0.451 Have other learners work on reading with the learner 0.288 0.233 Communicate the problem to parents/caregivers 0.284 0.354 Ask the caregivers to help the learner with reading 0.345 0.362 Run a reading club 0.006 0.052 USAID/SOUTH AFRICA SPS IMPACT EVALUATION BASELINE REPORT 153 | Page Table A11: Correlates of oral reading fluency Eastern Cape KwaZulu-Natal Dependent variable: ORF Zero correct ORF Zero correct Grade 3 7.103*** -0.130*** 12.797*** -0.162*** (0.491) (0.019) (0.557) (0.017) Grade 4 14.953*** -0.300*** 20.993*** -0.312*** (0.603) (0.023) (0.705) (0.021) Age 4.699*** -0.096* 2.246 -0.076 (1.298) (0.051) (1.594) (0.047) Age squared -0.323*** 0.007** -0.254*** 0.007*** (0.070) (0.003) (0.088) (0.003) Female 5.475*** -0.104*** 5.878*** -0.089*** (0.354) (0.014) (0.385) (0.011) Stunted (zhfa <-2) -1.239* 0.062** -1.217* 0.019 (0.633) (0.025) (0.717) (0.021) -2