This publication was produced for review by the United States Agency for International Development. It was prepared by NORC at the University of Chicago. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. IMPACT EVALUATION OF USAID/SOUTH AFRICA STORY POWERED SCHOOL PROGRAM Endline Report Prepared under Contract No.: GS-10F-0033M/AID-OAA-M-13-00010 NORC, QUALITATIVE DATA COLLECTION. CUTTING AND FOLDING. PHOTO CREDIT: URSULA HOADLEY USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | i USAID READING AND ACCESS STORY POWERED SCHOOL PROGRAM IMPACT EVALUATION Endline Report DECEMBER 2019 Prepared under Contract No.: GS-10F-0033M/AID-OAA-M-13-00010 Submitted to: USAID Submitted by: Cally Ardington, Ursula Hoadley, and Alicia Menendez Contractor: NORC at the University of Chicago 4350 East West Highway, 8th Floor Bethesda, MD 20814 Attention: Varuni Dayaratna DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States. This publication was produced at the request of the United States Agency for International Development. ii | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV TABLE OF CONTENTS ACRONYMS...................................................................................................................................................................... vi EXECUTIVE SUMMARY..................................................................................................................................................1 1 INTRODUCTION.........................................................................................................................................................6 1.1 CONTEXT AND PROJECT BACKGROUND..............................................................................................6 1.2 EVALUATION PURPOSE AND AUDIENCE...............................................................................................10 2 EVALUATION QUESTIONS AND METHODOLOGY OVERVIEW ............................................................12 2.1 EVALUATION QUESTIONS ............................................................................................................................12 2.2 METHODOLOGY OVERVIEW.......................................................................................................................13 2.2.1 SAMPLE SELECTION AND DESIGN .....................................................................................................14 2.2.2 SAMPLE SIZE.................................................................................................................................................15 2.2.3 INSTRUMENTS ............................................................................................................................................16 2.2.4 LIMITATIONS...............................................................................................................................................19 3 FINDINGS .....................................................................................................................................................................20 3.1 SAMPLE DESCRIPTION.....................................................................................................................................20 3.2 IMPACT OF SPS PROGRAM ON LEARNERS.............................................................................................20 3.2.1 IMPACT OF BEING IN A SPS SCHOOL ON LEARNERS’ READING ATTITUDES, BEHAVIORS AND SKILLS...................................................................................................................................20 3.2.1.1 Attitude to reading...............................................................................................................................21 3.2.1.2 Reading Behavior..................................................................................................................................23 3.2.1.3 Reading proficiency ..............................................................................................................................25 3.2.1.4 SPS program effects.............................................................................................................................27 3.2.2 IMPACT OF ATTENDING A NAL’IBALI READING CLUB ON LEARNERS’ READING ATTITUDES, BEHAVIORS AND SKILLS ....................................................................................................30 3.3.1 IMPACT OF BEING IN A STORY POWERED SCHOOL ON TEACHERS’ READING ATTITUDES AND PRACTICES .........................................................................................................................34 3.3.2 IMPACT OF ATTENDING BIG 5 TRAINING ON TEACHERS’ READING ATTITUDES AND PRACTICES .............................................................................................................................................................38 3.3.4 ASSOCIATION BETWEEN TEACHER PRACTICES AND LEARNERS’ READING ATTITUDES, BEHAVIORS AND SKILLS .........................................................................................................45 3.4 UPTAKE AND ADHERENCE TO THE SPS PROGRAM..........................................................................46 3.4.1 INSIGHTS FROM NAL’IBALI MONITORING DATA .......................................................................47 3.4.2 INSIGHTS FROM QUANTITATIVE EVALUATION DATA.............................................................52 3.5 FINDINGS FROM STORY SPARKER CASE STUDIES AND INTERVIEWS ........................................54 3.5.1 STRUCTURE OF STORY SPARKER ACTIVITY IN SCHOOLS......................................................54 USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | iii 3.5.2 OBSERVATIONS OF FOUR STORY SPARKERS IN CASE STUDY SCHOOLS.........................55 3.5.2.1 Reading Activity ....................................................................................................................................58 3.5.2.2 Structuring and Managing Pedagogic Activity.................................................................................59 3.5.2.3 Discussion – Story Sparker Observations......................................................................................60 3.5.3 FACTORS IMPACTING ON THE UPTAKE OF THE PROGRAM IN SCHOOLS .........................61 3.5.3.1 The structure of the school day .......................................................................................................61 3.5.3.2 School and teacher acceptance.........................................................................................................61 3.5.3.3 Number of functions and activities..................................................................................................62 3.5.3.4 Program focus and support................................................................................................................63 3.5.3.5 Understanding the development of reading habits.......................................................................63 4. COSTS...........................................................................................................................................................................65 5. CONCLUSIONS AND RECOMMENDATIONS ...............................................................................................67 REFERENCES....................................................................................................................................................................70 iv | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV TABLE OF TABLES Table 1: Evaluation questions........................................................................................................................................12 Table 2: Data collection activities................................................................................................................................13 Table 3: Baseline and endline sample size by province and treatment status - schools and learners.........15 Table 4: Teacher endline sample size by province and treatment status...........................................................16 Table 5: Skills and corresponding subtasks including Reading Assessment tool ..............................................17 Table 6: Learner Interview............................................................................................................................................18 Table 7: Endline learner sample by grade and province .........................................................................................20 Table 8: Program impact on learner reading attitudes, behaviors and skills.....................................................28 Table 9: Program impact on learners scoring zero on EGRA sub-tasks............................................................29 Table 10: Percentage of learners attending reading clubs in treatment and control schools.......................31 Table 11: Impact of currently attending reading club in SPS school on learner reading attitudes, behaviors and skills.............................................................................................................................................................................32 Table 12: Impact of attending reading club in SPS school in previous year on learner reading attitudes, behaviors and skills – cohort I only.............................................................................................................................33 Table 13: Program impact on frequency of teacher reading and literacy practices ........................................35 Table 14: Program impact on teachers' reading for enjoyment activities..........................................................36 Table 15: Program impact on teachers’ attitude to learning to read..................................................................37 Table 16: Impact of attending Big 5 training on teachers’ reading practices and attitudes............................39 Table 17: Concept, level and indicator for analysis of reading for enjoyment in instructional practice.....40 Table 18: Dominant type of reading in the classroom............................................................................................41 Table 19: Reading for enjoyment concepts and indicators....................................................................................42 Table 20: Association between learners’ reading attitudes, behaviors and skills and teachers’ reading for enjoyment score ..............................................................................................................................................................45 Table 21: Percentage of schools meeting SPS component expectations............................................................48 Table 22: Number of Story Sparker visits in 2018 ..................................................................................................48 Table 23: Story Sparker daily reports.........................................................................................................................51 Table 24: Learner access to reading materials at home.........................................................................................54 Table 25: Time Breakdown of Story Sparker Activity over Four Days of Observation ................................56 Table 26: SPS expenses and in-kind contributions, in US dollars.........................................................................65 TABLE OF FIGURES Figure 1: Reading for enjoyment....................................................................................................................................9 Figure 2: SPS theory of change .....................................................................................................................................10 Figure 3: Attitudes to reading.......................................................................................................................................21 Figure 4: Distribution of reading attitudes index for learners in control schools...........................................22 Figure 5: Reading attitudes index by province and treatment status..................................................................23 Figure 6: Reading attitudes index by province and treatment status..................................................................24 Figure 7: Frequency of reading on own at home by treatment status................................................................24 Figure 8: EGRA fluency sub-tasks by province and treatment status, endline..................................................25 Figure 9: EGRA comprehension sub-tasks by province and treatment status, endline..................................26 Figure 10: Cohort II additional EGRA sub-tasks by province and treatment status, endline........................27 Figure 11: Daily frequency of literacy and language activities with learners by treatment status................35 USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | v Figure 12: Learner reports on taking reading materials home and choosing their own readers by treatment status...............................................................................................................................................................37 Figure 13: Frequency of teacher reading books aloud to class just for fun by treatment status .................38 Figure 14: Reading for enjoyment score - teachers in control schools..............................................................43 Figure 15: Reading for enjoyment score - teachers in SPS schools.....................................................................44 Figure 16: Percentage of schools meeting SPS component expectations by year and province ..................47 Figure 17: Distribution of number of Story Sparker visits in term II of 2018 .............................................................49 Figure 18: Story Sparker Reported Activities...........................................................................................................49 Figure 19: Percentage of visits activity happens during the school day ..............................................................50 Figure 20: Percentage of schools with registered reading clubs, by cohort and year.....................................52 Figure 21: School reports on Story Sparker Activities, by province...................................................................52 Figure 22: Learner reports on readers and storybooks in classrooms, by province ......................................53 Figure 23: Time Allocation to Different Activities by Four Story Sparkers Over Four Days.......................58 vi | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV ACRONYMS DEAR Drop Everything and Read DBE Department of Basic Education EC Eastern Cape EDR Evaluation Design Report EGRA Early Grade Reading Assessment HOD Head of Department ICC Intra-Cluster Correlation IE Impact Evaluation KZN KwaZulu-Natal LOLT Language of learning and teaching MDES Minimum Detectable Effect Size ORF Oral Reading Fluency RCT Randomized Control Trial RFE Reading for Enjoyment SPS Story Powered Schools: A South African Reading Revolution USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 1 EXECUTIVE SUMMARY NORC at the University of Chicago, through the USAID Reading and Access Evaluation Contract, has been charged with conducting the external impact evaluation (IE) of Story Powered Schools: A South African Reading Revolution (SPS). The main purpose of this evaluation is to assess the causal impact of SPS on reading outcomes of primary school children. In this report we present summary findings from the endline evaluation. PROJECT BACKGROUND SPS aims to develop and sustain a culture of reading for enjoyment in 720 primary schools in the Eastern Cape (EC) and KwaZulu-Natal (KZN) provinces of South Africa. The program focuses on nurturing a love of reading, in mother tongue and English, to spark children’s potential and unlock their capacity to learn. The project is run by South African non-profit organization, Nal’ibali, in partnership with the United States Agency for International Development (USAID), the DG Murray Trust, and national, provincial and local Departments of Education. The following activities take place at each SPS school: • “Big 5” training – 5 teachers trained in reading for enjoyment (2 days). • Community training for 10 (in 2016-17) and 12 (in 2018-19) community volunteers in reading for enjoyment (1 day). • Providing five hanging libraries – canvas wall hangings with clear pockets to display the 30 books each and 15 editions per year of the bilingual Nal’ibali reading newspaper supplement. • Supporting schools to register reading clubs. • Weekly visits from Story Sparkers – Nal’ibali mentors. These are high school or university graduates from the community that receive training and support in the field by SPS Literacy Mentors and Provincial Coordinators. On their weekly visit, Story Sparkers assist schools in creating a culture of reading for enjoyment with activities scheduled before, during, and after the school day. • Holiday programs in June/July that include multiple reading activities for learners. • Events, campaigns, and competitions, which engage the broader school community. • Reading club showcases – competitions where reading clubs are adjudicated according to established criteria and where children from reading clubs share and perform their achievements. These SPS activities aim to increase: 1) awareness about the importance of reading for enjoyment to contribute to children’s literacy development, and learning; 2) motivation to read, write, and share stories; 3) positive behaviors and habits that support reading development; 4) access to reading materials; 5) children’s skills and confidence relating to reading, writing, and storytelling; and 6) parents’ and communities’ involvement in promoting reading. EVALUATION METHODOLOGY OVERVIEW The methodology for this evaluation is a randomized-controlled trial (RCT), with communities of schools randomly assigned to treatment and control groups. We complemented these data with qualitative data from classroom observations in a sub-sample of treatment and control schools and with in-depth case studies in four treatment schools. In addition, we used program monitoring data provided by SPS. Quantitative data was collected in two rounds: a baseline prior to program implementation, and an endline two (academic) years later, after each school received direct SPS program support for the first 2 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV year and had the opportunity to continue with SPS activities during the second. By design, the learner baseline sample comprises roughly equal numbers of boys and girls, and equal numbers of grade 2, 3, and 4 learners .We collected learner longitudinal data by administering assessments to the same individual learner at both, baseline and endline. Our learner assessments include the most common EGRA subtasks, and several other creative writing, reading comprehension, productive listening comprehension and written comprehension exercises. Additionally, this IE assesses SPS’s impact on other intermediate outcomes such as participation of children in reading activities at school and in the community/home, time spent reading independently, children’s attitudes toward reading, and teacher attitudes and behavior toward RFE and storytelling. To measure these outcomes, the evaluation uses survey instruments to collect data at the level of the learner, teacher, and school. Estimating the effect of attending Nal’ibali reading clubs in the SPS schools is somewhat challenging. Participating in the clubs is not a random event, but rather a choice made by each learner. Learners that choose to attend reading clubs and learners that do not cannot be directly compared as they might differ, not only in terms of their club participation but also in taste for reading, reading skills, family support, etc. Therefore, the effect of attending a Nal’ibali reading club in a SPS school is estimated using statistical techniques (coarsened exact matching) to construct a comparison group of learners from control schools that is very similar to the group of learners who attend reading clubs. Similarly, teachers in SPS schools who attend Big 5 training likely differ from teachers in those schools that did not attend in systematic ways. We therefore use matching techniques to construct a comparison group of teachers from control schools to estimate the effect of attending Big 5 training. ANSWERING THE EVALUATION QUESTIONS EVALUATION QUESTION 1: What is the impact of the SPS project on learners’ reading and writing skills in their mother tongue? And reading and writing skills in English? SPS does not show an impact on learners’ reading and writing skills in mother tongue or in English. We tested a wide array of reading and writing skills, including those that are most likely to have been impacted by the SPS program, such as creative writing, productive listening, oral reading and written comprehension. The estimated effects are all very small and not statistically different from zero for any of the outcomes of interest in either province. We find the same results when we analyze separately cohorts, boy and girls, different grades, or learners’ proficiency groups. EVALUATION QUESTION 2: What is the impact of the SPS intervention on the time or frequency that students spend reading independently? And students’ motivation to seek out, listen to, read, write, and tell stories? We do not find an impact due to the SPS intervention on the time or frequency that the average student spends reading independently, or his/her motivation and attitude towards reading, writing, listening or telling stories. The estimated effects are all very small and not statistically different from zero for any of the outcomes of interest in either province. We find the same results when we analyze separately cohorts, boy and girls, different grades, or learners’ proficiency groups. EVALUATION QUESTION 3: What is the impact of SPS on teachers’ attitudes towards reading for enjoyment in the classroom, storytelling in the classroom, and reading clubs? And on teachers’ language and literacy instruction practices in the classroom (both those trained by Nal’ibali and those not trained by Nal’ibali but teaching in SPS schools)? Teachers trained by Nal’ibali and those not trained but teaching in SPS schools are very similar. Compared to teachers in control schools, they are significantly more likely to report that: learners retell USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 3 stories that they read, read with the teacher in small ability groups, and use a dictionary; that they allow learners to take readers home and choose their own reader; and to run a reading club at their school. There is no impact of SPS on teachers’ attitudes towards reading for enjoyment or in their pedagogic practice in language/literacy lessons. There is little evidence of pedagogies or activities that foster reading for enjoyment in both SPS and control schools. EVALUATION QUESTIONS 4 & 5: What is the impact of attending a Nal’ibali reading club (in a SPS) on learners’ reading and writing skills in their mother tongue? And reading and writing skills in English? What is the impact of attending a Nal’ibali reading club (in a SPS) on the time or frequency that students spend reading independently? And students’ motivation to seek out, listen to, read, write, and tell stories? Around 15 percent of learners in SPSs attend reading clubs. Compared to learners in control schools with similar baseline characteristics, these learners appear to read on their own at home more frequently and for fun in the holidays. In the Eastern Cape, reading club participants also have higher fluency and do better on the oral reading and productive listening comprehension tasks than the matched comparison learners in control schools. The effects range from 0.15 to 0.26 standard deviations. EVALUATION QUESTIONS 6 & 7: What is the association between teacher practices and learners reading and writing skills in their mother tongue? And in English? What is the association between teacher practices and the time or frequency that students spend reading independently? Students’ motivation to seek out, listen to, read, write, and tell stories? There is no significant association between learners’ reading and writing skills –in mother tongue or English- and their teacher’s instructional practice in relation to reading for enjoyment. There is no evidence of students seeking out different forms of engagement with stories. Lessons observed were wholly teacher-directed and there were no activities initiated by learners. EVALUATION QUESTIONS 8: What is take-up of and adherence to the SPS intervention? On average, the take-up and adherence to SPS is low. Program activities did not take place in the schools as expected. A large number of schools did not fulfill the minimum expectations of the program and very few met the target for all components. Story Sparker daily reports suggest that reading clubs only took place during one in five visits and that in many schools neither teachers nor community members were taking on the running of reading clubs between visits. Sustainability was an issue with the percentage of schools with at least one reading club dropping sharply after the Story Sparker withdrawal. Even low-effort components of the programs were missing, for example, hanging libraries on display. CONCLUSIONS AND RECOMMENDATIONS In general, we found that the SPS intervention was successful in providing reading materials at school and at home however this did not translate into changes of learners’ reading proficiency, attitudes or behaviors towards reading, such as the time that they spent reading independently at home. Only 23 percent of learners in SPSs attended reading clubs in the year that the Story Sparker visited and less than half of these learners were still attending a reading club the following year. When compared to learners in control schools with similar baseline characteristics, these learners appear to read on their own at home more frequently and for fun in the holidays. In the Eastern Cape, attending a reading club was associated with better reading outcomes. We explored the potential reasons behind these results, making use of all the quantitative, qualitative and program monitoring data available. Taken together these data point towards four main problems: 4 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV 1) Partial implementation. Story Sparkers visits to schools were less frequent than anticipated. Although teachers tend to make very positive comments about the program, the central challenge identified by the Story Sparkers was the lack of support from teachers in the schools. Story Sparkers’ reports suggest that there was a serious dosage issue with reading clubs during the period they visited SPS schools. Reading clubs only happen during one in five Story Sparkers’ visits and very seldom between visits. 2) Low take-up by the schools. Despite universal sign up amongst schools invited to participate in SPS, the program activities do not take place in the schools as expected. A large number of schools did not fulfill the minimum expectations of the program and very few were meeting the target for all components. Story Sparkers reported lack of support in the SPS schools, and even low-effort components of the programs are missing, for example, hanging libraries were not on displayed in almost half of the schools when the Story Sparkers arrived. Reading clubs require at least half-hour and are very unlikely to run before or after school for multiple reasons. Neither teachers nor community members have taken on the activity of running reading clubs. We saw a sharp decline in the percentage of learners who report attending a reading club after the Story Sparker has left and the majority of SPS schools report no community involvement in reading activities. The lack of commitment and apathy of the schools’ staff towards the program is a significant factor impacting on uptake and seriously compromises its sustainability. 3) Program limitations. There are areas of the program that need improvement. Although the program was expected to hold activities before and after the school day or during breaks, this was clearly not possible. Story Sparker activities predominately took place during the school day, crowding out other activities and generating some resentment from teachers. This is a central problem that needs to be addressed. Story Sparkers do not have pedagogical expertise and they have a great deal of autonomy. Many SPS activities included very little storytelling, reading, or writing. Story Sparkers followed in the case study, only spent a quarter of the time engaging with texts, and this only occurred during normal instructional time with whole classes and not in the Reading Clubs. Activities in all three entailed singing and cut, paste and draw activities. In general, there was little evidence of reading for enjoyment or its development. We recommend that the SPS program is reviewed taking into account the following considerations: • The feasibility of reading clubs needs to be revisited given the impossibility of before and after school activity • Increase the amount of reading the Story Sparker does. Consider making the Story Sparker’s work more structured and directed. • Develop a clearer concept of how to engender reading for enjoyment in learners who have very low reading proficiency. • Revisit the issue of sustainability of the program once the Story Sparkers depart from the school. • Weigh up costs and benefits of the Reading Showcase competition –or at least its frequency- in terms of program aims. • More support needs to be provided for Story Sparkers in negotiating relationships with schools and in delivering reading for enjoyment activities. USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 5 • In training on the use of Nal’ibali supplements focus on practical ways to use sections other than “Cut Out and Keep”. • Review the role of Literacy Mentors as literacy mentors to the Story Sparkers, providing the latter with support and development in their primary task – developing reading for enjoyment. 4) Existing policies, programs and practices. SPS or similar programs need to be designed in relation to the programs currently in place in schools. While SPS attempts to fill a neglected area of the curriculum – reading for enjoyment and for choice, this is undermined by the fact that the ongoing programs (such as NECT and Jika iMfundo) have greater levels of accountability and monitoring. SPS and programs like it need to try and align their interventions more carefully with those that are already in place. The broad instructional interventions have strong pacing requirements built into their program, and make full use of curriculum time. This is a crucial aspect to take into account in trying to shift practice: whether there is time available for teachers to innovate or extend their practice beyond official requirements. While the goals of the SPS program are critical and laudable, the program did not take sufficient cognizance of existing policies, program and practices of teaching reading in order that their program could gain traction in the school lives of the pupils who so desperately need to develop a love of reading. 6 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV 1 INTRODUCTION 1.1 CONTEXT AND PROJECT BACKGROUND In 2016, almost 80 percent of South African Grade 4 learners fell below the lowest international level of reading proficiency, which means they could not locate explicit information or reproduce information from text. This effectively means they cannot read for meaning. Grade 4 learners living in remote rural areas or townships are at a particular disadvantage; they have lower reading scores than learners in other locations. 1 The ability to read for meaning by Grade 4 is considered crucial by education experts, since students are expected to utilize their reading skills to learn other subjects from Grade 4 onward. To tackle this challenge, the Story Powered Schools (SPS) program aims to develop and sustain a culture of reading for enjoyment in 720 primary schools in the Eastern Cape and KwaZulu-Natal provinces of South Africa. The SPS program is part of the broader Nal'ibali (isiXhosa for “here’s the story”) campaign, which is a national reading-for-enjoyment campaign to spark children’s potential by making reading and storytelling part of their daily lives. The SPS program focuses on nurturing a love of reading, in mother tongue and English, to spark children’s potential and unlock their capacity to learn. The program is run by South African non-profit organization Nal’ibali in partnership with USAID, the DG Murray Trust, and national, provincial, and local Departments of Education. The campaign is based on research on the link between reading for pleasure and improved education outcomes, and identifies the necessary conditions to support children’s literacy development.2 The degree to which children acquire language skills and become motivated, habitual readers, is a strong predictor of future academic success, educational attainment, employment, and income.3 To support children’s development as readers, the SPS project seeks to create the following conditions: • Increase awareness that reading for enjoyment is important and powerful, and knowledge of how to do it; • Create more opportunities for children to read, write, and share stories – alone and with others; • Inspire, equip, and encourage adults to be reading role models who share stories with children, encourage them to read, and model that reading matters; and • Increase access to quality reading material, especially in children’s mother tongues. The SPS project aims to give children new opportunities to experience books and stories; to create the conditions where children’s literacy development can flourish; and to equip, inspire, and nurture teachers on how motivation, confidence, and linking reading and writing to children’s lived experiences can support curriculum objectives, accelerate children’s literacy development, and enable school success. Ultimately, it wants schools to embrace and take ownership of a strong culture of reading for enjoyment. 1 PIRLS Literacy 2016: South African Highlights Report http://www.up.ac.za/media/shared/164/ZP_Files/pirls-literacy-2016-hl-report￾3.zp136320.pdf 2 For example, Krashen, S. (2004). The power of reading: insights from the research. Libraries Unlimited: Westport; Evans, M.D.R. et al. (2010) Family scholarly culture and educational success: books and schooling in 27 nations. Research in Social Stratification and Mobility 28: 171-197; United Kingdom Department of Education (2012). Research evidence for reading for pleasure (available from www.gov.uk). 3Needu (2013). NEEDU reading study 2013: the state of reading in Grade 5 in selected rural primary schools. National Education Evaluation and Development Unit. Available at https://nicspaull.files.wordpress.com/2014/12/needu-reading-study-2013_28oct14.pdf USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 7 The SPS program has seven main components designed to engender reading for enjoyment among learners at the level of the school. These are: Story Sparkers; the Big 5; Hanging Libraries; Nal’ibali Supplements; Drop Everything and Read (DEAR); Reading Clubs; and community and family involvement. The different school level program components are discussed in more detail below. STORY SPARKERS The SPS recruits young community members to function in schools as ‘Story Sparkers’. Central to the program, they are also tasked with mentoring teachers in reading for enjoyment activities in preparation for their withdrawal from the school after a year. Each Story Sparker is allocated four schools4. Each school is visited once a week, and on the Friday the Story Sparkers meet with their Literacy Mentor and other Story Sparkers. The Story Sparker is meant to spend the full school day at school, from before the start of the school day (to do ‘before school activities’) until after school (for ‘after school activities’). Story Sparkers are contracted to a school for a year. Thereafter the school is assumed to take responsibility for implementation of the program. There is a very wide range of activities that Story Sparkers are expected to undertake in the schools. These include: • Organizing ‘Reading Clubs’; • Coordinate before and after school activities; • Mentor teachers in techniques to develop reading for enjoyment, especially leading reading clubs • Participate in DEAR periods • Hold meetings and information sessions for teachers and community members; • Coordinate learners for the annual “Reading Showcase” • Conduct songs and games with learners • Conduct storytelling sessions • Organize activities around the following events and competitions: • World Read Aloud Day: February/March • International Mother Language Day: 21 February • Storytelling Day: 20 March • World Book Day: 23 April • Get-Caught-Reading Month: May • Book Lovers’ Day: 9 August • International Literacy Day: 8 September • International Picture Book Month: November Thus the scope of the Story Sparkers’ work is very broad. There is also a wide degree of autonomy granted to Story Sparkers in terms of what they do when, and in what way. “The Big 5” A selection of teachers 5, known as the “Big 5”, are given a once-off two-day training focused on integrating reading for enjoyment into their classrooms. The “Big 5” are largely self-selected and comprise teachers from Grade 1 to Grade 4 who are motivated and committed to the SPS program. Their training focused on the SPS program principles and on the idea of reading for enjoyment in particular. The various components of the program, including the use of the Nal’ibali supplements, 4 In 2017 each SS was assigned to 5 schools and visited one per day. 5 In 2017, Nal’ibali trained 3 teachers and 2 community volunteers per school at the “Big 5” training. It was found that a critical mass of trained teachers is needed, so in 2018 they shifted to train 5 teachers per school. 8 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV before and after school activities, the role of communities and creating reading clubs were all covered. Coverage generally was superficial given the brevity of the training. The idea is that the Story Sparker acts as a mentor to the Big 5 with the intention that they will take over all the Story Sparker’s functions once he or she departs at the end of the year. Thus they would continue to run reading clubs, before and after school activities, draw in community members, etc. Drop Everything and Read period (DEAR) DEAR is a national campaign that mandates schools to allocate time in the school day where everyone in the school engages in independent or shared reading activity. The DEAR period is meant to last for 30 minutes every day. Launched in 2015, it has yet to gain traction in most schools. The SPS project is meant to support DEAR periods, with Story Sparkers running or assisting in running DEAR periods, especially conducting read alouds. Reading clubs Reading clubs are clubs established in schools that are meant to meet at break or after school to engage with stories and text in various ways – through songs and games, read alouds and drama. The groups are supposed to be small, mixed grade and voluntary, and the aim is to have a particular, print-rich space marked out for reading for enjoyment activity, rather than reading instruction. Each school is expected to establish at least three reading clubs, which may be run by the Story Sparkers, teachers and/or community volunteers. Critical to the withdrawal of the Story Sparker from the school is that the school take ownership of some of these Reading Clubs. Hanging libraries Each SPS school is provided with five ‘Hanging Libraries’, each with 30 books. The books are graded from Grade R to Grade 3, and are in English and isiXhosa or isiZulu. The idea is that one hanging library is distributed to each of the Foundation Phase grades (i.e. Grade R to 4) and shared among the teachers in that grade. Supplements The Nal’ibali supplement, is distributed to every SPS school every second week during term time. The supplements are brightly colored newsprint magazines that include different sections, all related to reading for enjoyment: • Stories and pictures printed to be made into small books (‘Cut Out and Keep’ books) • Stories to read aloud (‘Story Corner’ stories) • Activities to go with stories (Get Story Active) • Puzzles and games • Reading tips for teachers and caregivers • Competitions and ‘Story Stars’, profiles of Nal’ibali materials users. Involvement of community members in reading aloud to children and story-telling Community volunteers and parents are invited to participate in the SPS program, and receive training to this end. The idea of drawing in community volunteers stems from the idea of extending storytelling traditions in learners’ homes to the school. In addition to these school-level components, the SPS program also has a number of “Milestones’ that are achieved at the district level. These are the Big 5 training; the training of community volunteers; running holiday programs and the Reading Club Showcase competition. The latter brings together selected members of reading clubs, teachers and Story Sparkers from different schools to showcase what they have been doing in the program in their schools. Ultimately, the changes the SPS project expects to see at schools include: USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 9 • Increased awareness that reading for enjoyment (RFE) is important and contributes to children’s literacy development, and knowledge/understanding of how RFE helps children learn; • Shifts in attitudes: children who are more excited/motivated to read, write and share stories; teachers who view reading for enjoyment as central to learning; parents who are more invested in reading and education; • Positive behaviors and habits that support reading development, such as children attending reading clubs, reading voluntarily in their free time, choosing what they want to read, taking books and supplements home, and reading a wide variety of books and stories; • Improved access to reading materials, with reading materials accessible to children in schools, and children taking books and supplements home; • Children’s reading, writing and storytelling skills improve; • Parents and communities more involved in promoting reading. Figure 1: Reading for enjoyment The SPS conceptualization of reading for enjoyment is summarized in Image 1. A range of affective elements (interest, excitement, inspiration) and activities (reading and storytelling) lead to a shift in reading attitudes and in turn a change in reading behaviors and finally attainment. The program also refers to immersion, the idea that readers need plenty of time and plenty of reading to develop a positive affective relationship to reading. In its elaboration of what the term means, the program refers to a number of different aspects – choice, fun, immersion, free reading – and it also defines reading for enjoyment in relation to what it is not, drawing on Kraschen (1993): Reading for enjoyment/Free voluntary reading means reading because you want to. For school age children, it means no book report, no questions at the end of the chapter, and no looking up every vocabulary word. It means putting down a book you do not like and choosing another one instead. It is the kind of reading that highly literate people do obsessively all the time. The SPS program’s theory of change is presented in Figure 2. 10 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV Figure 2: SPS theory of change INPUTS Training of teachers, officials, community volunteers Multilingual reading materials Parents and community engagement around the importance of reading At least 3 reading clubs per school, run by Story Sparkers, teacher or volunteers and supported by Nal’ibali staff Weekly visits to schools by Story Sparkers Events, campaigns and competitions Holiday programs OUTPUTS Schools invited to sign up for SPS Reading clubs operating at the school Reading materials available at school and home OUTCOMES Learners more motivated to read More time devoted to reading for enjoyment Increased support of teachers, parents, and community members around reading Use of reading materials Learners attending reading clubs IMPACT Reading skills improved Reading comprehension improved Vocabulary improved 1.2 EVALUATION PURPOSE AND AUDIENCE NORC at the University of Chicago, through the USAID Reading and Access Evaluation Contract, has been charged with conducting the IE of SPS. The main purpose of this evaluation is to assess the causal impact of SPS on reading outcomes of primary school children. In this report, we present summary findings from the endline evaluation. The evaluation measures reading skills using subtasks of the EGRA, a widely used tool to measure various aspects of reading proficiency, and several other creative writing, reading comprehension, productive listening comprehension and written comprehension exercises. Additionally, this IE assesses SPS’s impact on other intermediate outcomes such as participation of children in reading activities at school and in the community/home, time spent reading independently, children’s attitudes toward reading, and teacher attitudes and behavior toward RFE and storytelling. To measure these outcomes, the evaluation uses survey instruments to collect data at the level of the learner, teacher, and school. The IE assesses the impact of SPS over a period of two school years. It was implemented in the provinces of Eastern Cape and Kwazulu-Natal, and focuses on isiXhosa and isiZulu languages, as well as English. USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 11 The findings, conclusions and recommendations generated by this evaluation and research study will contribute toward understanding how activities devoted to RFE contribute to create a culture of reading and to improve reading, comprehension, and vocabulary. The audiences for this evaluation and research study comprise USAID Operating Units, notably, USAID/South Africa, the Africa Bureau, and the E3/Education Office. Other important audiences are the Government of South Africa, primarily, the Department of Basic Education, and the Eastern Cape and Kwazulu-Natal education departments, South African and international researchers, donors committed to improving learners’ reading skills around the world, and the implementing partner, Nal’ibali. 12 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV 2 EVALUATION QUESTIONS AND METHODOLOGY OVERVIEW 2.1 EVALUATION QUESTIONS Representatives from USAID/South Africa, USAID/E3/Education Office, Nal’ibali SPS team, and NORC met in South Africa in April 2016 to jointly discuss the SPS program and the evaluation objectives. The evaluation approach and evaluation questions in Table 1 below are results of those discussions. Table 1: Evaluation questions # OUTCOMES QUESTIONS 1 Student reading skills Q1. What is the impact of the SPS project on learners’: A. reading and writing skills in their mother tongue (including letter recognition, word recognition, fluency, comprehension, vocabulary, phonemic awareness, orientation to print)? B. reading and writing skills in English? 2 Student Motivation and Behavior Q2. What is the impact of the SPS intervention on: A. the time or frequency that students spend reading independently? B. students’ motivation to seek out, listen to, read, write, and tell stories? 3 Teachers Attitude and Behavior Q3. What is the impact of the SPS intervention on: A. teachers’ attitudes towards a. reading for enjoyment in the classroom? b. story telling in the classroom? c. reading clubs? B. teachers’ language and literacy instruction practices in the classroom (both those trained by Nal’ibali and those not trained by Nal’ibali but teaching in SPS schools)? 4 Student reading skills Q4. What is the impact of attending a Nal’ibali reading club (in a SPS) on learners’: A. reading and writing skills in their mother tongue? B. reading and writing skills in English? 5 Student Motivation and Behavior Q5. What is the impact of attending a Nal’ibali reading club (in a SPS) on: A. the time or frequency that students spend reading independently? B. Students’ motivation to seek out, listen to, read, write, and tell stories? 6 Student reading skills Q6. What is the association between teacher practices and learners’: A. reading and writing skills in their mother tongue? B. reading and writing skills in English? 7 Student Motivation and Behavior Q7. What is the association between teacher practices and: A. the time or frequency that students spend reading independently? B. students’ motivation to seek out, listen to, read, write, and tell stories? USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 13 # OUTCOMES QUESTIONS 8 Program take-up (participation in reading activities and use of reading materials/ trainings) Q8. What is take-up of and adherence to the SPS intervention? This will include a range of indicators. For example: A. number and characteristics of invited schools signing up for SPS B. number and characteristics of teachers trained C. number of community members trained D. implementation of reading for enjoyment sessions before, during, and after school day E. number of reading clubs operating at the school, number of children in the reading clubs F. attendance at reading clubs G. other events (competitions, etc.) H. use of mini libraries 2.2 METHODOLOGY OVERVIEW The methodology for this evaluation is a randomized-controlled trial, with communities of schools randomly assigned to treatment and control groups. We complemented these data with qualitative data from classroom observations in a sub-sample of treatment and control schools and with in-depth case studies in four treatment schools. Quantitative data was collected in two rounds: a baseline prior to program implementation, and an endline two (academic) years later, after each school received direct SPS program support for the first year and had the opportunity to continue with SPS activities during the second. We collected student longitudinal data by administering assessments to the same individual students at both, baseline and endline. The main advantage of a student-level panel is that it allows for student-level fixed effects, which can improve the power and precision of the analysis if baseline outcomes are highly correlated with endline outcomes. We also collected qualitative data, which included classroom observations and case studies. Table 2 below shows the data collection activities that took place over the evaluation period. Table 2: Data collection activities SURVEY TYPE INSTRUMENTS DATE Baseline (quantitative) EGRA assessment & Learner interview Teacher interview & School inventory Cohort 1: Term 1 2017 Cohort II: Term I 2018 Midline (qualitative) Classroom observation & Case studies Learner assessments Cohort II: Term III 2018 Endline (quantitative) EGRA assessment & Learner interview Teacher interview Cohort 1: Term III 2018 Cohort II: Term III 2019 Estimating the effect of attending Nal’ibali reading clubs in the SPS schools is somewhat challenging. Participating in the clubs is not a random event, but rather a choice made by each learner. Learners that choose to attend reading clubs and learners that do not cannot be directly compared as they might differ, not only in terms of their club participation but also in taste for reading, reading skills, family support, etc. Therefore, the effect of attending a Nal’ibali reading club in a SPS school is 14 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV estimated using statistical techniques (coarsened exact matching) to construct a comparison group of learners from control schools that is very similar to the group of learners who attend reading clubs. Similarly, teachers in SPS schools who attend Big 5 training likely differ from teachers in those schools that did not attend in systematic ways. We therefore use matching techniques to construct a comparison group of teachers from control schools to estimate the effect of attending Big 5 training. More details on data sources and data analysis methods can be found in Annex II. 2.2.1 SAMPLE SELECTION AND DESIGN Community randomization and school selection: While SPS is a school based intervention, the Nal’ibali approach involves the community around the school in a number of ways. Randomization was, therefore, done at the community level, with all schools in a community assigned to either treatment or control. The design considerations and process of creating appropriate communities of schools is described in detail in the baseline report6 and briefly included in Annex II. Within each community, between two and four schools were randomly selected to be part of the evaluation. The same schools were visited at baseline and endline. Teacher selection: In each school selected for data collection, we randomly sampled one teacher each from grades 2, 3 and 4 at baseline. Where the school was a multi-grade schools with learners from different grades taught by the same teacher, we selected one teacher from each grade combination in the Grade 2 to 4 range. We interviewed the same teachers at endline. Learner selection: Learners were randomly selected from the classes of the sampled teachers. We randomly selected 30 learners from each school (10 each from grades 2, 3, and 4, comprising of five girls and five boys whenever possible). Learners were only selected if they were present at the school on the day that the field team arrived to complete the assessments7. We re-interviewed and re￾assessed the same children at endline when they were finishing grades 3, 4 and 5. Classroom selection: Classroom observations that were conducted in 20 schools as part of the evaluation of SPS. Ten treatment and 10 control schools were selected from the larger evaluation sample to compare practices across lessons. We observed 1 Big 5 trained teacher and 1 untrained teacher in each of 10 treatment schools, and 2 teachers in each control school, with the exception of one school were 3 teachers were observed. We also randomly selected 15 learners from the class of each teacher selected for observation. Reading proficiency of these learners was assessed using the enline EGRA assessment. These learners also completed the endline learner interview. During the course of the fieldwork, the qualitative field team had to make some changes to the teachers selected for observation and we therefore have learner data for 30 of the 40 classrooms observed. We assessed and interviewed 453 learners in these classes. The process of selecting schools and teachers is described in detail in the Classroom Observations Report8. Case studies: In 2018, two schools in rural KwaZulu-Natal and two Eastern Cape schools were selected in consultation with the SPS program. In each province, researchers wanted to observe the program in a school that was regarded as having successful uptake of SPS, and one where uptake was regarded as problematic, and sought the advice of Nal’ibali on the selection. Unfortunately, the schools suggested by the program manager of Nal’ibali did not accord with these criteria, although observations did take place in one school where the program was regarded as working well. Four Story Sparkers were ‘shadowed’ for a full school day recording their activities in detail. The Story Sparkers and the Heads of Department in the schools were also interviewed using semi-structured 6 Ardington, C. and A. Menendez (2018) Impact Evaluation of USAID/South Africa Story Powered School Program – Baseline. NORC at The University of Chicago 7 The absenteeism rate is approximately 11% in all grades, and it is the same in treatment and control schools. 8 Hoadley, U. (2019) Impact Evaluation of USAID/South Africa Story Powered School Program – Classroom Observation Report. NORC at The University of Chicago USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 15 interview schedules. On different days, the Literacy Mentors that oversaw the work of the Story Sparkers were also interviewed. Finally, one of the Provincial Coordinators of the SPS program was interviewed. 2.2.2 SAMPLE SIZE Due to programmatic considerations, the baseline was conducted over two years with 166 schools in cohort I (term 1 2017) and 193 schools in cohort II (term 1 2018). Within each school, we randomly selected 10 learners for each of grades 2, 3 and 4 for assessment and interview. The total number of learners assessed at baseline was slightly lower than planned (30 per school) due to very low enrolment in some schools and a handful of schools that did not offer all three grades of interest. The endline assessments were conducted with the same learners at the same schools in the third term of the following year. The baseline and endline sample sizes are summarized in Table 3. Table 3: Baseline and endline sample size by province and treatment status - schools and learners EASTERN CAPE KWAZULU-NATAL TREATMENT CONTROL TREATMENT CONTROL BASELI NE ENDLI NE BASELI NE ENDLI NE BASELI NE ENDLI NE BASELI NE ENDLI NE Schools 87 86 85 85 97 97 91 91 Grade 2 learners 853 706 823 714 924 778 856 699 Grade 3 learners 848 720 823 711 924 789 860 746 Grade 4 learners 860 731 825 729 929 738 879 736 Overall learner attrition 16% 13% 17% 16% Aside from one school in the Eastern Cape that closed down, we successfully revisited all baseline schools at endline and were able to re-assess 85 percent of the 10,404 learners assessed at baseline. Attrition rates were marginally higher among treatment learners than control learners – 16 percent versus 13 percent in the Eastern Cape and 17 percent versus 16 percent in KwaZulu-Natal. The overall attrition rate and the differential attrition rate between treatment and control are both comfortably within the What Works Clearinghouse (WWC) conservative limits for low expected bias (WWC Standards Brief for Attrition). Learners who were not re-assessed had significantly poorer outcomes across a range of baseline EGRA tasks. They were also more likely to be classified as stunted and less likely to co-reside with their mother. The full details of the correlates of attrition are presented in Tables A1 and A2 in Annex IV. Baseline equivalence of the full sample was guaranteed through the random assignment of schools to treatment and control using statistical software. The low overall attrition rate and the minimal differential attrition between treatment and control groups suggests that the final sample is balanced. We conducted balance checks on the endline sample by comparing a range of baseline learner characteristics and reading outcomes across treatment and control. As expected, the differences 16 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV between treatment and control on all variables are within the limits to satisfy baseline equivalence9. The full details of the balance checks are presented in Tables A3 and A4 in Annex IV. Table 4 below summarizes the baseline and endline sample sizes for the teacher sample. The planned sample included one teacher each from grades 2, 3 and 4. Overall, 32 percent of schools in the sample had at least some learners of different grades taught in the same classroom. In these schools, there were not necessarily three different teachers across the three grades. In total, the baseline sample included 495 teachers in the Eastern Cape and 506 teachers in KwaZulu-Natal. At endline, we were able to successfully re-interview 78 percent of these teachers. At endline, 16 percent of teachers had left the school, 5 percent were absent and 2 percent refused to be interviewed. Teacher attrition was higher in the Eastern Cape than in KwaZulu-Natal. There is no difference in the attrition rate between treatment and control teachers. Baseline equivalence checks were conducted on variables and revealed no imbalances. Full details of the covariates of attrition and balance checks are presented in in Tables A5 and A6 in Annex IV. Table 4: Teacher endline sample size by province and treatment status EASTERN CAPE KWAZULU-NATAL TOTAL CONTROL TREATMENT CONTROL TREATMENT Successfully re-interviewed 183 181 195 216 775 Refused 2 10 3 0 15 Left school 46 40 32 41 159 Absent 16 14 10 10 50 Attrition rate 26% 26% 19% 19% 22% 2.2.3 INSTRUMENTS Reading Assessment: The evaluation measures reading outcomes using subtasks of the Early Grade Reading Assessment (EGRA), a widely used tool to measure various aspects of reading proficiency, and other reading and writing exercises. In Table 5 we indicate the skills tested and their corresponding subtask. The assessments were conducted in mother tongue (isiXhosa or isiZulu) for all learners. English assessments were not administered to Grade 2 learners. For baseline assessments, NORC used an existing version of EGRA, created by Lili Pretorius, a professor of linguistics at the University of South Africa (UNISA) for the Zenex Foundation in 2015. In addition, NORC included six receptive vocabulary items where the learner was asked to identify the correct picture to match a particular word. At endline, productive listening comprehension tasks designed by Dr. Pretorius were added in both mother tongue and English. For these tasks, the leaner listens to a short story and responds questions about it. To further broaden the scope of literacy outcomes that the SPS program could potentially impact, we randomly selected half of the schools in each stratum of cohort II and administered three additional tasks. These were 1) a vocabulary subtask, where learners identify the correct word out of three choices to complete sentences, 2) a reading comprehension task, where learners read short sentences 9The WWC standard for baseline equivalence considers absolute standardized mean differences (effect sizes) below or equal to 0.05 to satisfy baseline equivalence and effect sizes above 0.25 to violate baseline equivalence. Across 35 variables, we have no effect sizes greater than 0.25. For effect sizes in between 0.05 and 0.25, the WWC requires statistical adjustment through the use of baseline controls in analyses. Of the 35 variables tested, we find 12 and 14 to be in this range for KwaZulu-Natal and the Eastern Cape respectively. All main analyses include a range of baseline covariates to ensure that there is no bias due to any baseline imbalances between the two groups. USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 17 and indicate whether they make sense or not and 3) a creative writing subtask, where learners were shown two pictures and asked to write a story about the pictures. Table 5: Skills and corresponding subtasks including Reading Assessment tool SKILL SUBTASK & MEASUREMENT WAVE ISIXHOSA/ISIZULU ENGLISH Orientation to print Awareness of text direction, where to start reading, how to read down a page Baseline Grade 2 Alphabet knowledge Number of letters sounds identified in 60 seconds Baseline Endline Grades 2, 3 & 4 Grade 3 Grades 3 & 4 Grade 3 Phonological awareness Identify and manipulate phonemes (starting and ending sounds of words, segmenting words) Baseline & Endline (Cohort II only) Grades 2, 3 & 4 Grades 3, 4 & 5 Grade 4 Grades 4 & 5 Word recognition Familiar word reading, number of correct words read in 60 seconds Baseline & Endline Grades 2, 3 & 4 Grades 3, 4 & 5 Grades 3 & 4 Grades 3, 4 & 5 Oral reading fluency Oral passage reading, number of words fluently read (with accuracy) from a reading passage in 60 seconds Baseline & Endline Grades 2, 3 & 4 Grades 3, 4 & 5 (different passages for each grade) Grade 4 Grades 4 & 5 Reading comprehension Number of questions answered correctly about the passage read aloud by the student Baseline & Endline Grades 2, 3 & 4 Grades 3, 4 & 5 (different questions for each grade) Grade 4 Grades 4 & 5 Receptive vocabulary Identify correct picture to match word Baseline Grades 2, 3 & 4 Grade 4 Productive Listening Comprehension Number of questions answered correctly about the passage read to the student twice Endline Grades 3, 4 & 5 (different questions for each grade) Grades 3, 4 & 5 Receptive Listening Comprehension Perform actions following verbal instruction from the enumerator Endline Grades 3, 4 & 5 Written Comprehension Number of questions answered correctly in written comprehension Endline Grade 5 Grade 5 Reading Comprehension Identify whether 20 short sentences make sense Endline (50% of Cohort II schools) Grades 3, 4 & 5 Vocabulary Identify correct word out of three choices to complete sentences Endline (50% of Cohort II schools) Grades 3, 4 & 5 Grades 4 & 5 18 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV SKILL SUBTASK & MEASUREMENT WAVE ISIXHOSA/ISIZULU ENGLISH Creative Writing Write a story about two pictures, marked on meaning making; spelling; organization /length; and vocabulary and language use Endline (50% of Cohort II schools) Grades 3 & 4 Grade 5 Learner Interview: after completing the reading assessment, all learners answered a survey that included questions about their background, reading preferences, attitudes and behaviors. NORC drew on questionnaires that had previously been used in South African schools in designing the learner tool and augmented it. In Table 6, we show a summary of the different dimensions measured in the survey. Table 6: Learner Interview COMPONENTS DESCRIPTION Learner background Learners’ characteristics including age, home language, disabilities, height, school absenteeism, repeated grade, had breakfast, access to library in school, books at home, mother and father at home, household assets, questions about teacher, etc. Nal’ibali (NB) resources Knowledge about NB, access to NB books, ownership of NB books, reading NB books, reading clubs in school, attendance to reading club (details) Reading behaviors Reading at home (alone and with someone) and its frequency, storytelling at home, reading during holidays Reading attitudes Reading preferences statements Teacher questionnaire: The teacher interview includes questions about the teacher education and background, availability of teaching reading resources, and opinion about them, classroom practices related to teaching reading, opinions and attitudes about teaching reading School Inventory: The school inventory includes information about the school infrastructure, security, access to electricity, phone and internet, language of instruction, feeding programs, among others. This instrument was only administered at baseline. Classroom Observation Tool: Two data collection instruments were used for the classroom observations. The first instrument consists largely of close-ended items to determine the presence and absence of a range of indicators across classrooms, and requires the recording of an open, detailed narrative of the lesson observed. The instrument also directs fieldworkers to the collection of qualitative image data, including photos of the classroom walls and text used in the course of lessons. The second instrument includes only the capturing of a lesson narrative. This second instrument is both a check on the first narrative and a supplement to it. In this way two forms of data are collected simultaneously – quantifiable closed-ended data, and rich open-ended qualitative data. Story Sparker Case Study: The selected Story Sparkers were ‘shadowed’ for a full school day. The observer recorded detailed notes of all activities, at least every 5 minutes or when the task changed, whatever happened first. Directly after each observation episode (lesson, or reading club) the observer sought clarification from the Story Sparker where language may have presented a barrier to understanding any activity or discussion. Semi-structured interview schedules were used for the interviews with Story Sparkers, school HODs, Literacy Mentors and the Provincial Coordinator. USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 19 2.2.4 LIMITATIONS The Evaluation Team encountered some limitations inherent to the design of this evaluation. We list some of the more relevant limitations below: Representativeness of the sample. The sample is representative of the areas where SPS is currently working and corresponds to two of the eleven official languages in South Africa – isiZulu and isiXhosa. Therefore, results are not generalizable at the national level or to other geographical areas or other languages. The sample is representative of the children that were in school during the assessment, therefore it does not include the 11percent of learners that were absent. English Literacy Lessons. All the classroom observations were conducted while the teacher was instructing the learners in reading in home language, isiZulu or isiXhosa. There were no English lessons to observe and therefore we cannot make statements about teachers’ practices during English lessons. Estimation of Reading Club Effects. Learners that participate in reading clubs are not a random group in the population but rather, they are self-selected into the activity. These learners differ from others in ways we can observe and in ways that we cannot observe. Given this, we cannot directly compare learners that choose to attend reading clubs and learners that do not. We use instead statistical techniques (matching) to construct a comparison group of learners from control schools that is very similar to the group of learners who attend reading clubs in all the characteristics that we can observe. The methodological approach assumes that the learners are also very similar in terms of other characteristics that we cannot directly observe. Given that learners self-select into the reading club this is a very on a strong assumption. 20 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV 3 FINDINGS 3.1 SAMPLE DESCRIPTION By design, the baseline learner sample comprises roughly equal numbers of boys and girls, and equal numbers of grade 2, 3, and 4 learners. Table 7: Endline learner sample by grade and province7 shows the endline learner sample by grade, and province. The sample remains fairly evenly split by grade and gender. The percentage of learners found to be repeating the grade they were in during baseline is also shown. For all grades and in both provinces, girls are less likely to repeat the grade than boys and repetition rates are highest in the Eastern Cape. Table 7: Endline learner sample by grade and province EASTERN CAPE KWAZULU-NATAL BOYS GIRLS BOYS GIRLS Grade 2 (baseline) 696 713 744 733 Grade 2 (endline) 10% 4% 8% 4% Grade 3 (endline) 90% 96% 92% 96% Grade 3 (baseline) 702 721 757 777 Grade 3 (endline) 9% 4% 5% 2% Grade 4 (endline) 91% 96% 95% 98% Grade 4 (baseline) 722 729 754 720 Grade 4 (endline) 13% 3% 11% 3% Grade 5 (endline) 87% 97% 89% 97% Observations 2,120 2,163 2,255 2,230 3.2 IMPACT OF SPS PROGRAM ON LEARNERS 3.2.1 IMPACT OF BEING IN A SPS SCHOOL ON LEARNERS’ READING ATTITUDES, BEHAVIORS AND SKILLS In this section we address the first two evaluation questions. Q1. What is the impact of the SPS project on learners’ reading and writing skills in their mother tongue? And reading and writing skills in English? SPS does not show an impact on learners. The estimated effects are all very small and not statistically different from zero for any of the outcomes of interest in either province. We find the same results when we analyze separately cohorts, boy and girls, different grades, or learners’ proficiency groups. Q2. What is the impact of the SPS intervention on the time or frequency that students spend reading independently? And students’ motivation to seek out, listen to, read, write, and tell stories? USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 21 We begin by presenting graphical summaries of the various outcomes used to measure learners’ reading attitudes, behaviors and skills. These figures show the mean and 95% confidence intervals for each outcome separately for treatment and control learners. Thereafter, we present the results from regressions that include a range of baseline covariates. The inclusion of these baseline covariates improves the precision of our estimates and also accounts for any non-random attrition and incidental variation between treatment and control groups at baseline. 3.2.1.1 ATTITUDE TO READING Measuring attitudes to reading proves to be somewhat challenging as it relies on self-reports and responses tend to be overly positive. At baseline, learners were shown a range of five smiley faces from very happy to very sad and then asked to point out which face most closely corresponded to how they felt about nine statements. The statements asked how much they liked to do a range of activities from playing to reading out loud in class. Overall, 69 percent of learners selected the very happy face when asked if they liked to read and a further 20 percent selected the happy face. The responses to questions about reading on their own, reading at home and reading at school were all very similar. At endline, we therefore decided to include a battery of 20 questions probing attitudes to reading in order to build an index that would allow a finer discrimination between learners. Figure 3 below shows the distribution of responses to these questions for learners in the control schools. The battery included 11 positively worded questions and nine negatively worded questions. Before proceeding with the battery of questions we asked learners whether they agreed or disagreed with the statements “You like to play” and “You like to be punished”. The responses to these two items are included in the first two lines of the figure and indicate that learners were able to understand the instructions and distinguish between agreeing and disagreeing. Figure 3: Attitudes to reading 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% like to play like to be punished like to read like to listen to stories like reading with a grown-up to help you like reading stories like reading comics or magazines like reading information books like to tell stories like to write reading is cool think reading is important reading is one of your favorite activities prefer watching TV to reading don’t like reading at home only read when you have to cannot find things to read that interest you would be embarrassed if your friends saw you read are not interested in books feel anxious/nervous when you have to read think reading is difficult think reading is boring Attitudes to reading - control learners Strongly agree Agree Indifferent Disagree Strongly Disagree 22 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV During the interview the positive and negative statements were mixed up. The figure groups the positive and negative statements to highlight learners’ tendency to strongly agree with the positively worded statements. The negatively worded statements elicit a wider range of responses. Figure 4 shows the distribution of a reading attitude index constructed from all 20 items with reverse scoring for the negative statements10. Despite the overly positive reporting on individual positive items, combing all the responses creates a pleasing distribution of responses with good discrimination between learners. Figure 4: Distribution of reading attitudes index for learners in control schools Figure 5 plots the distribution of the reading attitude index separately for treatment and control learners in each of the provinces. The distributions are fairly similar across treatment and control groups. In both provinces, the treatment distribution appears to be slightly to the right (i.e. more positive) of the control distribution. In the program impact section below, we test whether these observed differences are statistically significant in regressions that control for a range of baseline characteristics. We find that the differences are not large enough for us to conclude that the SPS program has a significant impact on the reading attitude index. 10 Scores for each statement ranged from 5 for strongly agree to 1 for strongly disagree. Negatively worded statements were inversely scored. The index is then the sum of scores across the 20 statements, and ranges from 20 to 100. 0 .02 .04 .06 .08 Fraction 40 60 80 100 Reading Attitude Index USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 23 Figure 5: Reading attitudes index by province and treatment status 3.2.1.2 READING BEHAVIOR Learners were asked whether they read on their own at home and at school and also whether they read for fun during the holidays. Figure 6 shows the percentage of learners who report independent reading by province and treatment status. There are no clear differences in these reading behavior 0 .01 .02 .03 .04 Density 40 60 80 100 Reading attitude index Control Treatment Eastern Cape 0 .01 .02 .03 .04 Density 40 60 80 100 Reading attitude index Control Treatment KwaZulu-Natal 24 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV measures between treatment and control learners. Further investigation into the frequency of these activities also reveals no differences between treatment and control learners (see Figure 7 for example). Figure 6: Reading attitudes index by province and treatment status Figure 7: Frequency of reading on own at home by treatment status Read on own at home Read on own at school Read for fun in holidays EC treatment 87% 85% 77% EC control 85% 85% 74% KZN treatment 86% 84% 77% KZNcontrol 86% 82% 78% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Percent of learners 18% 19% 50% 50% 15% 13% 2% 3% 15% 14% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Control Treatment Frequency of reading on own at home Every day or almost everyday Two to four days a week One day a week Less than once a week Never USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 25 3.2.1.3 READING PROFICIENCY Figure 8 presents the mean score for each of the endline EGRA fluency subtasks by province and treatment group. As observed at baseline, learners in the Eastern Cape are able to identify more letter sounds per minute than learners in KwaZulu-Natal, but are slower at reading connected text. Within each province, treatment and control leaners average performance is very similar across both the home language and English fluency subtasks. Figure 8: EGRA fluency sub-tasks by province and treatment status, endline Letter sounds Familiar words Connected text EFAL Letter sounds EFAL Familiar words EFAL Connected text EC treatment 38 27 25 32 27 53 EC control 38 26 24 32 26 51 KZN treatment 21 28 32 19 30 60 KZNcontrol 19 27 31 18 30 60 0 10 20 30 40 50 60 70 Correct items per minute Figure 9 summarizes the comprehension sub-tasks. Similar to the fluency items, we see better performance in KwaZulu-Natal than in the Eastern Cape but no significant differences in the performance of learners by treatment status. The reading comprehension questions are based on the passage used to measure oral reading fluency with learners only asked questions related to the parts of the passage that they have read. Low fluency levels present a challenge for assessing comprehension as even learners who can read, do not get very far in the passage in the one minute. During the cohort II endline, we therefore allowed learners an additional two minutes to continue reading the passage. The oral reading fluency measure was still based on the first minute but this change allowed learners to attempt more reading comprehension questions. Focusing only on Cohort II learners, the average percent correct for reading comprehension is 43 percent, 42 percent, 50 percent and 50 percent for Eastern Cape treatment and control and KwaZulu-Natal treatment and control respectively. 26 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV Figure 9: EGRA comprehension sub-tasks by province and treatment status, endline Productive listen Oral reading Written EFAL receptive listen EFAL productive listen EFAL oral reading EFAL written EC treatment 74% 43% 37% 69% 17% 19% 35% EC control 74% 42% 37% 68% 17% 18% 32% KZN treatment 78% 50% 45% 78% 22% 24% 42% KZNcontrol 77% 50% 43% 79% 22% 24% 38% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Percent correct To further broaden the scope of literacy outcomes that the SPS program could potentially impact, we randomly selected half of the schools in each stratum of cohort II and administered four additional tasks to each learner. The average scores, together with 95% confidence intervals, for these tasks are presented in Figure 10. As discussed above, low fluency levels present significant challenges for measuring reading comprehension. We included an additional comprehension task that aimed to get around some of these challenges. This untimed subtask consists of 20 short sentences (typically two words in isiXhosa and isiZulu) that the learner needs to read and then indicate whether the sentence makes sense or not. For example, “Cars have wheels” makes sense while “Fire is cold” does not. Each sentence has a matching pair (e.g. “Fire is cold” and “Fire is hot”) and the learner only receives a point if they score correctly on both items in the pair. Learners performed fairly well on these task with an average score of 70 percent in the Eastern Cape and 76 percent in KwaZulu-Natal. Learners in treatment schools do not appear to have out-performed learners in control schools. Learners with greater exposure to stories and texts should have larger vocabularies than those with less exposure. We assessed both home language and English vocabulary through a task which required the learner to select the correct word among three options to complete a sentence. Learners found this task to be quite challenging with average scores well below 50 percent. Scores are very similar across treatment and control learners. The final additional task was a creative writing exercise. Learners were shown two pictures and asked to write a story about the pictures. The stories were scored according to a rubric with a maximum of five marks each in the following categories – meaning making; spelling; organization/length; and vocabulary and language usage. Performance across all four categories and the overall score were similar among treatment and control. USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 27 Figure 10: Cohort II additional EGRA sub-tasks by province and treatment status, endline 3.2.1.4 SPS PROGRAM EFFECTS In this section, we present the estimated effects of the SPS program on the various outcomes used to measure learners’ reading attitudes, behaviors and skills. When estimating these effects, we control separately for all baseline EGRA sub-tasks that were common across the three grades, namely letter sounds, phonemic awareness, familiar word reading, oral reading fluency, reading comprehension and vocabulary. Control variables also include learners’ age and sex, indicators for baseline grade, an indicator that the learners was repeating the grade, an indicator that the learner was absent from school in the last week, an indicator that there are no books at home to read, an indicator of whether the child had breakfast before coming to school, indicators that the child lives with their mother and father, a count of household assets, an indicator that someone at home reads with the learner, an indicator that the learner reads at home on their own daily, an indicator that the learner never reads at home and an indicator that the learner really likes to read. We also include the learner’s height-for-age z-score which is a good measure of long run health and has been shown to be correlated with socio-economic status in early childhood and with cognitive skills in multiple settings. Table 8 shows the effects for all learners. In the first panel of the table we present the effects on reading attitudes and behaviors. The second and third panels show the SPS effects on reading skills in the home language and in English, respectively. We present the results by province, and we include the p-values associated with the effect and the effect size. The estimated effects are all very small and not statistically different from zero for any of the outcomes of interest in either province. Sentence choice Vocabulary EFAL Vocabulary Write a story EC treatment 71% 33% 40% 38% EC control 69% 35% 39% 38% KZN treatment 76% 44% 45% 42% KZNcontrol 76% 44% 46% 44% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Percent correct 28 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV Table 8: Program impact on learner reading attitudes, behaviors and skills EASTERN CAPE KWAZUL U-NATAL ESTIMATE D EFFECT P-VALUE EFFEC T SIZE ESTIMATE D EFFECT P￾VAL UE EFFEC T SIZE READING ATTITUDES AND BEHAVIORS Reading attitude index 0.41 0.57 0.04 0.60 0.27 0.06 Read on own at home 0.01 0.53 0.02 -0.00 0.75 0.01 Read on own at home daily 0.00 0.93 0.00 0.02 0.14 0.05 Read on own at school -0.01 0.50 0.03 0.02 0.13 0.05 Read for fun in holidays 0.02 0.31 0.05 -0.01 0.62 0.02 READING SKILLS IN HOME LANGUAGE Correct letter sounds per minute 0.00 0.99 0.00 1.98 0.07 0.13 Correct words per minute 0.37 0.40 0.02 0.64 0.16 0.05 Oral reading fluency 0.51 0.25 0.03 1.04 0.11 0.06 Productive listening comprehension 0.01 0.89 0.00 0.02 0.76 0.01 Reading comprehension 0.07 0.43 0.03 0.05 0.61 0.02 Written comprehension -0.00 0.99 0.00 0.18 0.20 0.08 READING SKILLS IN ENGLISH Correct letter sounds per minute -0.56 0.68 0.03 1.20 0.22 0.08 Correct words per minute 0.61 0.40 0.03 0.21 0.80 0.01 Oral reading fluency 1.33 0.36 0.04 -0.03 0.99 0.00 Receptive listening comprehension 0.06 0.58 0.04 -0.03 0.69 0.02 Productive listening comprehension -0.02 0.85 0.01 -0.04 0.71 0.02 Reading comprehension 0.05 0.65 0.02 -0.02 0.91 0.01 Written comprehension 0.25 0.21 0.09 0.44 0.09 0.14 Note: controls include all baseline EGRA sub-tasks common across the three grades, learners’ background variables and full set of indicators for the sample strata. Where data is missing, we assign a value to the variable (zero for categorical variables and the sample mean for continuous variables) and include an indicator that the learner is missing data on that variable. Standard errors are clusters at the level of randomization, i.e. the community level. P-values are not adjusted for multiple testing. Effect size computed using pooled standard deviation. The SPS program was initiated in late 2016, with the cohort I schools being the first to receive the program. During the 2017 year, Nal’ibali had the opportunity to learn from the experience of the first cohort and to improve implementation and refine program components. It is therefore possible that USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 29 the SPS program had greater impact among cohort II learners than among cohort I. We estimated the effects of the SPS program separately by cohort and found no impact in either group. We show the estimation details in Table A7 in Annex IV. A program such as the SPS, could also have different impact according to the grade of the learner. The type of differential effect by grade is ambiguous, on the one hand, it is possible that having more resources to read could affect most positively those that are more advanced readers already. On the other hand, it is also plausible that effects on reading attitude and behavior are more pronounced the earlier the child is exposed to the program. We estimated the effects of SPS separately by grade but found no impact in any group. Table A8 in Annex IV shows the details of the estimations by grade. Although, on average, grade should be correlated with the reading skills of the learners, it might not perfectly capture the child’s reading proficiency. Therefore, we also analyzed the effects of SPS on the number of learners with zero scores in the different reading subtasks, and on the bottom, middle, and top performing quartile of learners. We found that there is no impact in reading attitudes, behaviors or in any of the subtasks for any of the subgroups. For example, Table 9 below shows the effects of the program on learners scoring zero in the EGRA subtasks. All the effects are not statistically significant and around zero in magnitude. More details about SPS effects by level of reading performance can be found in Table A9 in Annex IV. Finally, we estimated SPS effects for girls and boys separately. Again there is no evidence of impacts for either group on any of the outcomes of interest. All the details are included in Table A10 in Annex IV. Table 9: Program impact on learners scoring zero on EGRA sub-tasks EASTERN CAPE KWAZULU-NATAL ESTIMATED EFFECT P￾VAL UE EFFECT SIZE ESTIMATED EFFECT P￾VAL UE EFFECT SIZE READING SKILLS IN HOME LANGUAGE Correct letter sounds per minute -0.00 0.67 0.03 -0.03 0.22 0.08 Correct words per minute 0.01 0.51 0.03 0.00 0.61 0.02 Oral reading fluency 0.01 0.54 0.02 0.01 0.23 0.05 Productive Listening Comprehension -0.00 0.74 0.01 -0.00 0.16 0.04 Reading Comprehension -0.00 0.94 0.00 0.02 0.08 0.07 Written Comprehension 0.03 0.04 0.10 -0.02 0.37 0.06 READING SKILLS IN ENGLISH Correct letter sounds per minute -0.01 0.40 0.05 0.01 0.78 0.02 Correct words per minute -0.01 0.72 0.02 -0.01 0.62 0.02 Oral reading fluency 0.01 0.52 0.03 0.00 0.77 0.01 30 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV EASTERN CAPE KWAZULU-NATAL ESTIMATED EFFECT P￾VAL UE EFFECT SIZE ESTIMATED EFFECT P￾VAL UE EFFECT SIZE Receptive listening comprehension -0.00 0.53 0.03 -0.00 0.45 0.03 Productive listening comprehension -0.00 0.96 0.00 0.02 0.43 0.04 Reading comprehension -0.01 0.61 0.02 0.01 0.70 0.02 Written comprehension -0.01 0.36 0.06 -0.02 0.10 0.09 Note: controls include all baseline EGRA sub-tasks common across the three grades, learners’ background variables and full set of indicators for the sample strata. Where data is missing, we assign a value to the variable (zero for categorical variables and the sample mean for continuous variables) and include an indicator that the learner is missing data on that variable. Standard errors are clusters at the level of randomization, i.e. the community level. P-values are not adjusted for multiple testing. Effect size computed using pooled standard deviation. In general, we have found that SPS did not change reading attitudes, behaviors or reading skills of learners in the treatment schools. Annex IV shows all the estimated effects for all outcomes and all subpopulations. There are a handful of positive and negative effects for specific outcomes and specific sub-populations but no more than one would expect to find by chance given that we run 197 regressions11. Furthermore, corrections for multiple testing using conventional approaches (Bonferroni, Holm-Bonferroni and others) eliminate significant effect. 3.2.2 IMPACT OF ATTENDING A NAL’IBALI READING CLUB ON LEARNERS’ READING ATTITUDES, BEHAVIORS AND SKILLS In this section we address evaluation questions 4 and 5. Q4. What is the impact of attending a Nal’ibali reading club (in a SPS) on learners’ reading and writing skills in their mother tongue? And reading and writing skills in English? Around 15 percent of learners in SPSs attend reading clubs. Compared to learners in control schools with similar baseline characteristics, these learners appear to read on their own at home more frequently and for fun in the holidays. In the Eastern Cape, reading club participants also have higher fluency and do better on the oral reading and productive listening comprehension tasks than the matched comparison learners in control schools. The effects range from 0.15 to 0.26 standard deviations. Q5. What is the impact of attending a Nal’ibali reading club (in a SPS) on the time or frequency that students spend reading independently? And students’ motivation to seek out, listen to, read, write, and tell stories? We focus here on a particular group of learners, those that attend a reading club. Among learners in treatment schools, 14 percent attend a reading club in the Eastern Cape and 17 percent attend a reading club in KwaZulu-Natal (Table 10). In control schools, the comparable figures are 9 percent in 11 There are 3 negative and 13 positive effects at the 5% significance level without adjustments for multiple testing. The effects sizes are very modest, with the largest at 0.22 and only two over 0.2. USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 31 the Eastern Cape and 6 percent in KwaZulu-Natal. For the first cohort, we asked learners whether they had ever attended a reading club in their school and when the reading club started. We also asked whether they were still attending a reading club at school. Unfortunately, given that learners could have attended multiple reading clubs in their time at the school, we cannot identify with perfect accuracy those learners who attended reading clubs in the year that the Story Sparker visited the school. For the second cohort, we added a question to enquire specifically whether the learner had attended a reading club at school in the previous year. In line with Nal’ibali’s monitoring data on reading club registrations (see section 3.4 below), we see substantial decreases in the percentage of learners attending reading clubs between the year that the Story Sparker visited the school and the following year. In Eastern Cape cohort II treatment schools 27 percent of learners report attending a reading club in the previous year in contrast to 12 percent that currently attend. Of the learners who attended a reading club in the previous year, 63 percent no longer attend. In KwaZulu-Natal, the decline in reading club attendance in cohort II treatment schools is slightly less steep going from 20 percent in the previous year to 17 percent in the current year. Half of the learners who attended in the previous year are still attending a reading club. Table 10: Percentage of learners attending reading clubs in treatment and control schools EC TREATMENT EC CONTROL KZN TREATMENT KZN CONTROL Currently attend RC (all) 14% 9% 17% 6% Currently attend RC (cohort I) 17% 11% 17% 4% Currently attend RC (cohort II) 12% 9% 17% 8% Attended RC last year (cohort II) 27% 14% 20% 8% Attended RC last year and now (cohort II) 8% 5% 10% 3% It is important to remember that learners that participate in reading clubs are not a random group in the population but rather, they are self-selected into the activity. These learners differ from others in ways we can observe and in ways that we cannot observe. For example, in treatment schools, learners who attend reading clubs tend to have better baseline reading proficiency while reading club attendance does not appear to be related to baseline reading skill in controls schools. The odds of being in a reading club in a treatment school are almost double (1.8 times) for learners in the top quarter of baseline performance as compared to learners in the bottom quarter of baseline performance. Given this, we cannot directly compare learners that choose to attend reading clubs and learners that do not. We use instead statistical techniques (coarsened exact matching) to construct a comparison group of learners from control schools that is very similar to the group of learners who attend reading clubs in all the characteristics that we can observe. The approach rests on a strong assumption, it assumes that the learners are also very similar on terms of other characteristics that we cannot directly observe. Details about the data used and the analysis methods can be found in Annex II and balance on matching characteristics can be seen in Figures A1 to A4 in Annex IV. For the analyses that follow, we focus on learners who report currently attending a reading club for the full sample. It is possible that learners who report currently attending a reading club in treatment schools are attending a club that is not an SPS club in origin or spirit. Assuming attending an SPS reading club has a positive impact, this would bias the measured effect downwards. We therefore also examine results for the second cohort for those learners who attended a reading club in the previous year when the Story Sparker was visiting the school. 32 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV In Table 11, we show the impact of currently attending a reading club on the outcomes of interest by province. Learners that attend reading clubs report that they read on their own at home more frequently than the matched comparison group of learners in control schools. They are also more likely to report reading on their own at school and for fun in the holidays, for example, learners in the Eastern Cape that attend reading clubs are 0.3 standard deviations more likely to read at home than those that do not. In the Eastern Cape, reading club participants show higher word reading fluency in their home language –around 4 words or 0.23 standard deviations. They also score higher on both oral reading and productive listening comprehension, with an effect sizes of 0.26 and 0.17 standard deviations. Aside from productive listening comprehension, currently attending a reading club does not appear to be associated with better reading skills in KwaZulu-Natal. Table 11: Impact of currently attending reading club in SPS school on learner reading attitudes, behaviors and skills EASTERN CAPE KWAZULU-NATAL READING ATTITUDES AND BEHAVIORS EFFECT P￾VALUE EFFECT SIZE EFFECT P￾VALUE EFFECT SIZE Reading attitude index 0.22 0.85 0.02 1.19 0.28 0.12 Read on own at home 0.11*** 0.00 0.30 0.01 0.59 0.04 Read on own at home daily 0.12*** 0.00 0.31 0.13*** 0.00 0.32 Read on own at school 0.06** 0.02 0.16 0.07*** 0.01 0.19 Read for fun in holidays 0.16*** 0.00 0.37 0.08*** 0.01 0.19 READING SKILLS IN HOME LANGUAGE Oral reading fluency 3.68** 0.01 0.23 1.43 0.30 0.08 Reading comprehension 0.74** 0.01 0.26 0.25 0.26 0.09 Productive listening comprehension 0.36** 0.03 0.17 0.31** 0.05 0.14 READING SKILLS IN ENGLISH Correct words per minute 3.17* 0.06 0.15 0.45 0.80 0.02 Receptive listening comprehension 0.08 0.48 0.06 0.09 0.36 0.07 Productive listening comprehension 0.19 0.12 0.13 0.12 0.37 0.07 Note: Coarsened exact matching used to match learners attending reading clubs to learners in control schools with similar baseline characteristics and reading proficiency. Standard errors are adjusted to allow for correlation at the level of randomization, i.e. the community level. P-values are not adjusted for multiple testing. Effect size computed using pooled standard deviation. *p<0.1, **p<0.05, ***p<0.001 USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 33 Results for attending a reading club in the previous year are show in Table 12. These learners report significantly more positive attitudes towards reading than their counterparts in control schools in both the Eastern Cape and KwaZulu-Natal. We no longer see the effects on reported reading behaviors but the impact on reading skills tend to be more pronounced than those in Table 12. Table 12: Impact of attending reading club in SPS school in previous year on learner reading attitudes, behaviors and skills – cohort I only EASTERN CAPE KWAZULU-NATAL READING ATTITUDES AND BEHAVIORS EFFECT P￾VALUE EFFECT SIZE EFFECT P￾VALUE EFFECT SIZE Reading attitude index 2.76** 0.02 0.27 2.05** 0.04 0.22 Read on own at home 0.03 0.24 0.08 0.00 0.91 -0.01 Read on own at home daily 0.06 0.11 0.15 0.09** 0.02 0.23 Read on own at school 0.04* 0.08 0.12 0.05 0.10 0.14 Read for fun in holidays 0.05 0.14 0.11 0.03 0.40 0.07 READING SKILLS IN HOME LANGUAGE Oral reading fluency 3.99*** 0.00 0.25 2.07 0.29 0.11 Reading comprehension 0.54** 0.02 0.18 0.11 0.58 0.04 Productive listening comprehension 0.28* 0.10 0.14 0.35* 0.05 0.17 READING SKILLS IN ENGLISH Correct words per minute 5.17*** 0.01 0.25 2.83 0.14 0.13 Receptive listening comprehension 0.16 0.18 0.12 0.04 0.75 0.03 Productive listening comprehension 0.27 0.18 0.18 0.49** 0.03 0.30 Note: Coarsened exact matching used to match learners attending reading clubs to learners in control schools with similar baseline characteristics and reading proficiency. Standard errors are adjusted to allow for correlation at the level of randomization, i.e. the community level. P-values are not adjusted for multiple testing. Effect size computed using pooled standard deviation. *p<0.1, **p<0.05, ***p<0.001 34 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV 3.3 IMPACT OF SPS ON TEACHERS In this section we address evaluation question 3. Q3. What is the impact of SPS on teachers’ attitudes towards reading for enjoyment in the classroom, storytelling in the classroom, and reading clubs? And on teachers’ language and literacy instruction practices in the classroom (both those trained by Nal’ibali and those not trained by Nal’ibali but teaching in SPS schools)? Teachers trained by Nal’ibali and those not trained but teaching in SPS schools are very similar. Compared to teachers in control schools, they are significantly more likely to report that: learners retell stories that they read, read with the teacher in small ability groups, and use a dictionary; that they allow learners to take readers home and choose their own reader; and to run a reading club at their school. There is no impact of SPS on teachers’ attitudes towards reading for enjoyment or in their pedagogic practice in language/literacy lessons. There is little evidence of pedagogies or activities that foster reading for enjoyment in both SPS and control schools. One aim of the SPS was to shift teachers’ practices through a two-day training activity focused on strategies and activities for implementing reading for enjoyment in classrooms. These included school￾wide activities such as using songs and games and using the Nal’ibali supplement and classroom activities of book-making and book talk and asking questions. Although only two days were allocated to teacher training, the program’s intention was that the Story Sparkers would support teachers into providing reading for enjoyment activities over a longer period of time. Teachers in SPS schools who did not attend training would also be exposed to the Story Sparker, reading materials and Nal’ibali events, competitions and campaigns. While change in teachers’ instructional practices was not a central thrust of the program, the project’s objective is stated as being “To change the attitude, behavior and habits of South Africans towards reading over a period of three years through the creation of at least 2,160 reading clubs in 720 schools, directly reaching at least 32,400 children in two provinces through reading clubs, 108,000 children in Grades R-4 and 64 800 children in Grades 5-7 through classroom-based activities, and an additional 200 000 children through events, campaigns, and the presence of supportive, informed and committed teachers and adults in their lives.” By this measure, the program intended to impact five times as many children through classroom-based activities as through reading clubs. Thus a rigorous consideration of what was happening in classrooms is merited. We use a combination of quantitative and qualitative data to investigate whether Big 5 training or being in a SPS school had any impact on the classroom practices of the teachers and whether reading for enjoyment activities were evident in Home Language reading lessons in SPS schools. We begin with the quantitative data. 3.3.1 IMPACT OF BEING IN A STORY POWERED SCHOOL ON TEACHERS’ READING ATTITUDES AND PRACTICES We investigated the impact of being in a Story Powered School on the literacy and language teaching practices for teachers in our sample. Teachers were asked how frequently a range of activities with learners took place in their class at both baseline and endline. The activities included both those that USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 35 were aligned with Story Powered Schools such as learners playing games related to the literacy/reading lesson and those that are antithetical to the program such as the whole class repeating sentences that the teacher said first. Figure 11 shows the percentage of teachers at endline who report conducting each activity daily, separately for treatment and control teachers. Figure 11: Daily frequency of literacy and language activities with learners by treatment status *p<0.1, **p<0.05, ***p<0.001 Table 13 presents estimates of the effect of being in a Story Powered School on the likelihood of teachers conducting a particular practice on a daily basis. Of the 13 practices, teachers in Story Powered Schools are significantly more likely to report the following practices occurring daily - learners retelling a story that they had read; learners reading with the teacher in small ability groups with the same reader; and learners using a dictionary. Table 13: Program impact on frequency of teacher reading and literacy practices ESTIMATED EFFECT P￾VALUE EFFECT￾SIZE The whole class repeated sentences that you said first. -0.03 0.46 -0.06 Learners retold a story that they read. 0.07* 0.01 0.21 Learners sounded out unfamiliar words. -0.04 0.37 -0.08 Learners learned meanings of new words. -0.00 0.98 -0.00 * ** * 0% 10% 20% 30% 40% 50% 60% The whole class repeated sentences that you said first. Learners retold a story that they read. Learners sounded out unfamiliar words. Learners learned meanings of new words. Learners read aloud (individually) to teacher or to other… Learners read with the teacher in small ability groups with… Learners were assigned reading to do on their own during… Learners were assigned reading to do on their own at home Learners played games related to the literacy/reading lesson You read a story to the learners You discussed a story with the learners Learners used a dictionary Learners answered reading comprehension questions about… Percentage of teachers reporting activity daily Control Treatment 36 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV ESTIMATED EFFECT P￾VALUE EFFECT￾SIZE Learners read aloud (individually) to teacher or to other learners. 0.040 0.35 0.08 Learners read with the teacher in small ability groups with the same reader 0.08** 0.02 0.19 Learners were assigned reading to do on their own during school time. 0.04 0.34 0.09 Learners were assigned reading to do on their own at home -0.03 0.44 -0.07 Learners played games related to the literacy/reading lesson 0.05 0.18 0.16 You read a story to the learners 0.02 0.65 0.04 You discussed a story with the learners 0.01 0.79 0.02 Learners used a dictionary 0.07** 0.02 0.22 Learners answered reading comprehension questions about what they have read 0.05 0.12 0.13 *p<0.1, **p<0.05, ***p<0.001 In Table 14, we show that teachers in Story Powered Schools are 14 percentage points more likely than teachers in control schools to allow learners to take readers home with them and 9 percentage points more likely to let learners choose their own reader. They are 15 percentage points more likely to report a period in the timetable dedicated to reading activities and 27 percentage points more likely to run a reading club at their school. Table 14: Program impact on teachers' reading for enjoyment activities TREATM ENT MEAN CONTR OL MEAN ESTIMATED EFFECT P￾VALU E EFFECT￾SIZE Learners allowed to take readers home 64% 47% 0.14*** 0.00 0.29 Learners allowed to choose own reader 92% 82% 0.09*** 0.00 0.29 Know about Drop Everything and Read campaign 85% 72% 0.15*** 0.00 0.39 Have period in timetable dedicated to reading 74% 61% 0.15*** 0.00 0.31 Have daily dedicated reading period 24% 22% 0.04 0.30 0.09 Run a reading club (Cohort II only) 46% 15% 0.27*** 0.000 0.588 *p<0.1, **p<0.05, ***p<0.001 In terms of attitudes around reading, teachers in Story Powered Schools do not appear to differ from teachers in control schools, except for a higher percentage agreeing with the statement “It is pointless USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 37 reading to children before they know how to read”. Agreeing with this statement does not align with Nal’ibali’s approach to literacy. We show the findings in Table 15. Table 15: Program impact on teachers’ attitude to learning to read TREA TMEN T CON TRO L ESTIMATE D EFFECT P￾VAL UE EFFEC T-SIZE Reading to children before they know how to talk or read helps them learn. 96% 96% -0.00 0.90 -0.01 Children need English for their future so there is no point teaching them to read in an African language. 9% 8% 0.02 0.46 0.06 Reading for enjoyment is nice for children, but it doesn’t help achieve CAPS outcomes. 36% 31% 0.03 0.50 0.07 There’s no point reading to children until they have learned to read themselves. 8% 4% 0.05** 0.02 0.19 Both schools and parents are responsible for teaching children how to read 61% 58% 0.06 0.19 0.12 *p<0.1, **p<0.05, ***p<0.001 In the learner interview, we asked learners whether they were allowed to take readers and library books home and whether they were sometimes allowed to choose their own reader. Figure 12 shows the learner responses by treatment conditional on there being readers in their classroom and a library in their school. The learner reports are very much in line with the teacher reports shown in Table 11 with learners in SPS schools both more likely to be able to take books home and to choose their own reader. Figure 12: Learner reports on taking reading materials home and choosing their own readers by treatment status 39% 61% 33% 46% 71% 44% 0% 10% 20% 30% 40% 50% 60% 70% 80% Allowed to take readers home Allowed to choose reader Allowed to take library books home Learner reports on choosing and taking materials home Control Treatment Learners were also asked how often their teachers read books aloud to the class just for fun. We show the distributions in Figure 13. Just under half of learners in both SPS and control schools report that their teacher reads to them just for fun every day or almost every day. The distributions are 38 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV remarkably similar for learners in SPS and control schools and statistical tests confirm that there are no significant differences. Figure 13: Frequency of teacher reading books aloud to class just for fun by treatment status 47% 49% 40% 38% 9% 8% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Control Treatment Frequency of teacher reading to class just for fun Every day or nearly every day Two to four days a week One day a week Once or twice a month Less than once a month Never 3.3.2 IMPACT OF ATTENDING BIG 5 TRAINING ON TEACHERS’ READING ATTITUDES AND PRACTICES Of the 397 teachers in Story Powered Schools that we re-interviewed at endline, 40 percent attended Nal’ibali’s Big 5 training at the beginning of the year that the Story Powered program began in their school. Similar to the children who choose to attend reading clubs, teachers who attend specific trainings are not a random sub-group and may differ systematically from other teachers. While we cannot control for differences in unobservable or unmeasured characteristics, we can reduce selection bias by matching the teachers in treatment schools who attended the Big 5 training with teachers in control schools with similar baseline characteristics. Balance on the baseline characteristics used for matching can be seen in Table A11 and Figure 5 in Annex IV. Table 16 presents the estimated impact of attending Big 5 training on the same teacher attitudes and practices considered in the previous section. Propensity score matching on a range of baseline characteristics is used to create a comparable group of control teachers against which to measure endline practices and attitudes. Attending training appears to have a similar impact to that of being in a Story Powered School on the likelihood of allowing learners to take readers home; allowing learners to choose their own reader; having a dedicated reading period in the timetable; and of running a reading club at school. Big 5 trained teachers are more likely than similar teachers in control schools to report the following activities occurring on a daily basis - learners sounded out unfamiliar words; learners assigned to do reading on their own at home; and learners using the dictionary. They do not appear to have different attitudes towards reading than teachers with matching baseline characteristics in control schools. USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 39 Table 16: Impact of attending Big 5 training on teachers’ reading practices and attitudes PRACTICES ESTIMATED EFFECT P￾VALUE EFFECT SIZE Learners allowed to take readers home 0.17** 0.02 0.34 Learners allowed to choose own reader 0.10* 0.06 0.29 Know about Drop Everything and Read (DEAR) campaign 0.15** 0.02 0.35 Have period in timetable dedicated to reading 0.17*** 0.01 0.36 Have daily dedicated reading period 0.06 0.42 0.14 Run a reading club 0.25*** 0.00 0.56 The whole class repeated sentences that you said first. -0.04 0.41 -0.08 Learners retold a story that they read. 0.00 0.95 0.01 Learners sounded out unfamiliar words. -0.12** 0.02 -0.24 Learners learned meanings of new words. -0.07 0.17 -0.14 Learners read aloud (individually) to teacher or to other learners. 0.03 0.57 0.07 Learners read with the teacher in small ability groups with the same reader 0.07* 0.08 0.18 Learners assigned reading to do on their own during school time. 0.03 0.62 0.07 Learners were assigned reading to do on their own at home -0.12** 0.04 -0.23 Learners played games related to the literacy/reading lesson 0.07 0.29 0.21 You read a story to the learners 0.01 0.82 0.03 You discussed a story with the learners 0.00 0.97 0.01 Learners used a dictionary 0.10* 0.01 0.33 Learners answered reading comprehension questions about what they have read 0.07 0.19 0.17 ATTITUDES Reading to children before they know how to talk or read helps them learn. 0.01 0.44 0.08 Children need English for their future so there is no point teaching them to read in an African language. -0.00 0.90 -0.02 Reading for enjoyment is nice for children, but it doesn’t help achieve CAPS outcomes. -0.02 0.79 -0.05 There’s no point reading to children until they have learned to read themselves. 0.01 0.63 0.05 Both schools and parents are responsible for teaching children how to read 0.16** 0.02 0.33 *p<0.1, **p<0.05, ***p<0.001 40 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV 3.3.3 Qualitative evidence on impact of teaching in SPS school and attending Big 5 training on language and reading pedagogy Classroom observations were conducted with 20 teachers in control schools,11 SPS trained teachers in and 10 non-SPS trained teachers in SPS schools. Practices were compared across lessons to determine if there were discernable differences in pedagogic practice in language/reading classrooms between SPS schools and non-SPS schools. In addition, the pedagogic practice of trained and non￾trained teachers within SPS schools were also compared. From the SPS documentation and a reading of research literature on the topic, four main conceptual ideas underpinning the SPS conceptualization of reading for enjoyment were derived: role-modelling, motivation, immersion, and choice. In analyzing the data, reading for enjoyment was considered through a range of indicators for each of these concepts (Table 17). The indicators were considered at the level of teaching strategies -including those specified by the SPS program-, learner activities and organizational features of the classroom. Table 17: Concept, level and indicator for analysis of reading for enjoyment in instructional practice LEVEL INDICATOR CONCEPT TEACHER STRATEGY 1 Teacher demonstrated how to handle and care for books Role-modeling 2 Teacher read or told a story and showed enjoyment, excitement or pleasure Motivation 3 Teacher indicated purpose for reading Motivation 4 Teacher read or told a story with intonation and expression Motivation 5 Teacher developed learners’ concept of print and printed material when reading Role-modeling 6 Teacher read out the name of the author, illustrator and/or translator Role-modeling LEARNER ACTIVITY 7 Learners chose a book to read themselves during the lesson Choice 8 Learner/s told a story of their own choosing during the lesson Choice 9 Learners retold, acted out or summarized a story during the lesson Motivation 10 Learners did writing for real purposes Motivation 11 Learners did paired, shared or independent silent reading Immersion 12 Learners engage with a rich range of literary resources Immersion ORGANIZATIONAL FEATURE 13 How print rich the classroom environment was Immersion 14 There was a reading carpet or similar space for use as a shared reading space Motivation The main question framing the analysis was ‘Are there discernible differences in pedagogic practice in Home Language reading lessons between treatment and control schools?’. USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 41 Teacher practices There was some evidence of teachers reading or telling a story and showing enjoyment, excitement or pleasure in about half the treatment and half the control schools. Teachers also read or told a story with intonation and expression in approximately half of treatment and half of control schools. In seven of the Control schools and three of the Treatment schools, teachers indicated a purpose for reading to some or to a large extent. For example, in one treatment school, the teacher read learners a story about a disabled child. Prior to reading the story she told learners that historically people had accused those who bore disabled children of witchcraft. Thus, she created a context for, and purpose in, reading the text. In the remaining lessons this occurred hardly or not at all. Very few lessons addressed concepts of print. Although more suited to the Grade 1 level -particularly in relation to text directionality, the front and back cover of the book, lower case and upper case lettering- aspects like punctuation, paragraphs and chapter headings are relevant across the early grades. In three Treatment and one Control classrooms the teacher drew attention to the author illustrator and/or translator of the book. In summary, the most common strategy teachers used in integrating reading for enjoyment in lessons was reading or telling a story with enjoyment, excitement or pleasure, and with intonation and expression. The incidence of these strategies was relatively evenly spread across treatment and control classrooms, however, suggesting that this strategy was not necessarily a program effect. Learner activity A key element in relation to reading for enjoyment identified above is the exercise of choice. In none of the classrooms were learners observed to choose a book to read themselves. In one of the Control classrooms and three of the Treatment classrooms learners had the opportunity to tell a story of their own choosing. There was also some evidence of songs and games in seven of the Treatment schools, as opposed to four Control schools. Given the dominant activity of the Story Sparkers (see Section 3.5), this may be one area in which the classroom practices of the teachers had been impacted by the SPS program. However, this is also an aspect that does not involve engagement with text. So, while songs and games may function as a motivator in relation to broader literacy practices it doesn’t in any direct way relate to reading. In none of the lessons were learners observed to write for real purposes. There was very little engagement in shared, paired and independent reading, of concern in particular in relation to learners’ immersion in text. Table 18 shows that the dominant form of reading was the whole class reading aloud, either with or without the teacher. Looking at instances of reading (aside from dominant forms), in three classrooms learners read individually aloud to the class or the teacher. In none of the classrooms were learners observed to read independently. Table 18: Dominant type of reading in the classroom CONTROL (N=20) TREATMENT (N=20) The teacher reads aloud to the class 7 7 The whole class reads aloud together with the teacher 2 1 The whole class reads aloud together without the teacher 8 9 The whole class reads portions of text after the teacher 3 1 Learners read aloud together in groups or pairs 0 2 42 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV CONTROL (N=20) TREATMENT (N=20) Learners read individually aloud to the class 0 0 Learners read individually silently independently 0 0 Learners read in pairs silently independently 0 0 Learners read individually to the teacher 0 0 In relation to learners’ engagement with a rich range of literary resources, there was no evidence to suggest that the treatment schools are using a greater range of texts nor different texts to control schools for the purposes of reading instruction. Further, dominant use of the blackboard and the DBE Workbook suggests a restricted range of literary resources, and the lack of use of SPS materials - especially hanging library books- suggests that the program did not succeed in extending the richness of materials used in the classrooms. Classroom organizational resources One of the core principles of Nal’ibali and the SPS program is to create print rich environments that provide learners with exposure to text and promote interest in reading. The classroom walls were observed and photographed by observers. About half of treatment and control schools were judged to be fairly or very reading rich, i.e. the displays were varied, stimulating and inviting in relation to literacy. Five Treatment classrooms and four Control classrooms had a carpet that could be used as a shared reading space. Some classrooms had a reading library in the classroom. Substantial differences between treatment and control classrooms in relation to classroom organizational resources were not found. Overall metrics for reading for enjoyment in the observed lessons We constructed an overall metric for reading for enjoyment that weighted and added the four conceptual dimensions to get a cumulative score for each teacher. The four conceptual dimensions for reading for enjoyment and their indicators are shown in Table 19. Table 19: Reading for enjoyment concepts and indicators CONCEPT INDICATOR ROLE￾MODELING Teacher demonstrated how to handle and care for books Teacher developed learners’ concept of print and printed material when reading Teacher read out the name of the author, illustrator and/or translator MOTIVATION Teacher read or told a story and showed enjoyment, excitement or pleasure Teacher indicated purpose for reading Teacher read or told a story with intonation and expression Learners retold, acted out or summarized a story during the lesson Learners did writing for real purposes There was a reading carpet or similar space for use as a shared reading space USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 43 CONCEPT INDICATOR CHOICE Learners chose a book to read themselves during the lesson Learner/s told a story of their own choosing during the lesson IMMERSION Learners did paired, shared or independent silent reading Learners engage with a rich range of literary resources How print rich the classroom environment was Reading for enjoyment scores12 for teachers in treatment and control schools are shown below in Figures 14 and 15. The graphs show a spread of scores on the reading for enjoyment measure. They also show variation in the incorporation of reading for enjoyment in classrooms. However, only two teachers (two treatment and one control) scored above 50% on the metric. Figure 14: Reading for enjoyment score - teachers in control schools 12 Teachers were assigned a percentage score for each of the four concepts, namely role-modelling, motivation, choice and immersion. The reading for enjoyment score is the average across the four concepts scores with a potential range of 0 to 100 percent. 34 51 33 35 35 26 34 28 6 20 31 25 11 20 23 26 20 47 14 24 0 10 20 30 40 50 60 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Teacher number Reading for enjoyment score - control 44 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV Figure 15: Reading for enjoyment score - teachers in SPS schools 17 29 26 28 25 32 48 46 58 33 20 3 28 54 45 3 9 31 29 23 25 0 10 20 30 40 50 60 70 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 Teacher number Reading for enjoyment score - treatment The figures also show the similarity of practices across treatment and control classrooms. On the metric, the average score in treatment schools was 29% and in control schools 27%. In terms of individual dimensions, lessons overall scored highest on the motivation measure, largely as a result of most teachers reading with intonation and expression and showing enjoyment, excitement or pleasure in reading or storytelling. Lessons overall scored second highest on immersion given that some classrooms were print rich. Lessons overall scored lowest on choice given the paucity of learner opportunity to engage in choice of text, and second lowest on role modelling. The analysis of reading for pleasure in the treatment and control classrooms shows minimal differences between those who received the SPS program treatment and those schools that did not. Although there was some variation, on average across all classrooms there was little evidence of teacher strategies, learner activities and organizational features that contributed to reading for enjoyment. On a metric designed to measure the conceptual dimensions of motivation, role-modelling, immersion and choice, lessons on average scored low (28 percent), with a range across lessons from 3 percent to 5 percent. In summary, no clear pattern emerged in relation to treatment and control schools, where the presence of reading for enjoyment was generally low and evenly distributed. Where the SPS program had a direct effect on instructional practice in relation to reading for enjoyment, we would have expected more of the indicators to be more present in the treatment schools and for the treatment lessons to score higher on the reading for enjoyment metric. The entire analysis reported above was repeated in relation to treatment schools only, comparing SPS-trained teachers in treatment schools with those teachers who had not been trained by the program. No differences in relation the inclusion of reading for enjoyment were found between these two groups of teachers in terms of teacher strategies, learner activity, and classroom organization. USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 45 3.3.4 ASSOCIATION BETWEEN TEACHER PRACTICES AND LEARNERS’ READING ATTITUDES, BEHAVIORS AND SKILLS In this section we address evaluation questions 6 and 7. Q6. What is the association between teacher practices and learners reading and writing skills in their mother tongue? And in English? There is no significant association between learners’ reading and writing skills –in mother tongue or English- and their teacher’s instructional practice in relation to reading for enjoyment. There is no evidence of students seeking out different forms of engagement with stories. Lessons observed were wholly teacher-directed and there were no activities initiated by learners Q7. What is the association between teacher practices and the time or frequency that students spend reading independently? students’ motivation to seek out, listen to, read, write, and tell stories? The classroom observation data also present an opportunity to explore the relationship between teacher literacy and language pedagogy and learner reading attitudes, behaviors and skills. We examine the association between the same learner outcomes that we considered in Section 3.1 and their teacher’s aggregate score across the four reading for enjoyment concepts. Results are shown in Table 20. Other than a small negative and marginally significant association between teachers’ reading for enjoyment score and productive listening comprehension, we do not see any relationship between teachers’ practices and the reading attitudes, behaviors and skills of their learners. As described earlier, there was little evidence of pedagogies or activities that fostered reading for enjoyment in both SPS and control schools. Although we see variation between teachers on the scores, levels are perhaps too low to have any meaningful impact on learners’ literacy practices or performance. Table 20: Association between learners’ reading attitudes, behaviors and skills and teachers’ reading for enjoyment score READING ATTITUDES AND BEHAVIORS COEFFICIENT STANDARD ERROR OBSERVATIONS Reading attitude index -0.014 0.088 446 Read on own at home -0.002 0.002 453 Read on own at home daily 0.001 0.002 453 Read on own at school 0.000 0.003 451 Read for fun in holidays -0.002 0.003 452 READING SKILLS IN HOME LANGUAGE Correct letter sounds per minute 0.011 0.122 453 Correct words per minute 0.032 0.208 453 Oral reading fluency 0.000 0.124 453 Productive listening comprehension -0.012* 0.006 453 Reading comprehension -0.005 0.017 453 Note: controls include gender, age, grade and province. Standard errors are clustered at the level of the teacher. *p<0.1, **p<0.05, ***p<0.001 46 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV No evidence was found in the classroom observation data of students seeking out different forms of engagement with stories. Lessons were wholly teacher-directed. Learners were asked by the teacher on five occasions to tell stories (for example, about a neighbor; their news). As described in the classroom observation report13, learners also had plenty of opportunities to listen to the teacher reading stories and they were also instructed to read stories. None of these activities, however, were initiated by learners. 3.4 UPTAKE AND ADHERENCE TO THE SPS PROGRAM In this section and in the next (section 3.5), we address evaluation question 8. Q8. What is take-up of and adherence to the SPS intervention? On average, the take-up and adherence to SPS is low. Program activities did not take place in the schools as expected. A large number of schools did not fulfill the minimum expectations of the program and very few met the target for all components. The percentage of schools with at least one reading club dropped sharply after the Story Sparker withdrawal. Around 15 percent of learners in SPSs attend reading clubs; neither teachers nor community members are taking on running reading clubs. Even low-effort components of the programs are missing, for example, hanging libraries in display. While we observe some changes in the enabling conditions for reading for enjoyment, we largely see no impact on learner reading skills, attitudes to reading or reading behaviors. In order to understand this lack of impact, we next explore program implementation and take-up through a combination of Nal’ibali monitoring data and the quantitative and qualitative data collected through the evaluation. We begin with a high level overview of program uptake and then consider the details of implementation of the various components through interview data and direct observation of Story Sparkers in Story Powered Schools. The quantitative and qualitative data point to a number of program challenges that resonate with Nal’ibali’s monitoring data. Post randomization, schools that were assigned to treatment were invited to participate in the Story Power Schools program. The invitations were endorsed and coordinated through the provincial education departments and all invited schools signed up to the program. As detailed in the introduction, the SPS program consisted of the following components: – Big Five training – Community training – Hanging libraries and Nal’ibali supplements – Weekly visits from a Story Sparker – Support with registering and running reading clubs – Events, campaigns and competitions Each SPS school was expected to – send 5 teachers to Big 5 training – send 10 community members to community training – establish at least three reading clubs 13 NORC (2019) Impact Evaluation of USAID/South Africa Story Powered School Program – Classroom Observation Report. U. Hoadley USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 47 – participate in competitions and Nal’ibali events, including the reading club showcase – consistently observe the DEAR period – use classroom time to give children meaningful, personally satisfying opportunities to read, write, and hear stories – engage parents and the community around the importance of reading – support their Story Sparker In this section we examine both Nal’ibali monitoring data and quantitative survey data. Findings from the Story Sparker case studies are presented in Section 3.5. 3.4.1 INSIGHTS FROM NAL’IBALI MONITORING DATA Figure 16 presents the percentage of schools by province and implementation year who fulfilled the first four expectations. For each expectation, the figure shows both the percentage of schools fulfilling the requirement at minimum (e.g. registering at least one reading club) and the percentage fulfilling the targeted requirement (e.g. registering at least three reading clubs). Up-take is far from universal and there is considerable variation by province. While uptake of these components improved for the second cohort, a large number of schools were not meeting the expectations. For example, the percentage of schools registering at least three reading clubs increased from below 50 percent to 68 percent but almost a third of cohort II schools did not meet the expectations. Figure 16: Percentage of schools meeting SPS component expectations by year and province Attended Big 5 training At least 5 attended Big 5 training Attended community training At least 10 attended community training Registered a reading club Registered at least 3 reading clubs Reading club show case EC 2017 95% 44% 62% 38% 69% 54% 72% KZN 2017 90% 25% 44% 10% 69% 46% 92% EC 2018 85% 32% 87% 55% 100% 72% 96% KZN 2018 94% 35% 94% 71% 96% 63% 90% 0% 20% 40% 60% 80% 100% 120% Components of the SPS program EC 2017 KZN 2017 EC 2018 KZN 2018 Table 21 shows the percentage of schools fulfilling all these components at minimum and at target, by year and province. While uptake of these components improved for the second cohort, a large number of schools were still not fulfilling the minimum expectations and very few were meeting the 48 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV target for all components. In the Eastern Cape, the percentage of schools receiving the minimum of all these components rose from 23 percent in 2017 to 70 percent in 2018. In KwaZulu-Natal the comparative figures are 25 percent and 84 percent. In the second year of the program, only 15 percent of SPS schools in the Eastern Cape and 22 percent of schools in KwaZulu-Natal were recorded by Nal’ibali as meeting the required expectation for these SPS components. Table 21: Percentage of schools meeting SPS component expectations EASTERN CAPE KWAZULU-NATAL 2017 2018 2017 2018 All four components (minimum) 23% 70% 25% 84% All four components (target) 8% 15% 2% 22% For the second cohort of Story Powered Schools, each Story Sparker was required to record and submit a daily report detailing every school visit. This presents an opportunity for a more detailed view of program up-take. Discussions with Nal’ibali highlighted some challenges with the submission of daily reports, particularly early in the year. It should be borne in mind that the results presented here are based on recorded visits and are likely to be an under-estimate of the actual number of visits. Nevertheless, information on activities conducted and challenges faced should be broadly representative of the Story Sparker’s daily experiences. The average number of reported Story Sparker visits per school in 2018 was around 18 in both provinces. In 2018 there were 206 school days which translates to roughly 41 weeks and schools were meant to receive weekly visits from their Story Sparker. Table 22 shows the percentage of schools by province with less than 10 reported Story Sparker visits, between 10 and 20 reported visits and more than 20 reported visits. If we are extremely conservative and assume that only 50 percent of visits were recorded, then 53 percent of schools in the Eastern Cape and 41 percent of SPS schools in KwaZulu-Natal received less than the targeted number of visits. Figure 17, which presents the distribution of Story Sparker visits in the second term, shows considerable variability between schools in the number of reported visits. Table 22: Number of Story Sparker visits in 2018 EASTERN CAPE KWAZULU￾NATAL Less than 10 visits 13% 12% Between 10 and 20 visits 40% 29% More than 20 visits 47% 59% Average number of visits 18 18 USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 49 Figure 17: Distribution of number of Story Sparker visits in term II of 2018 For each daily report, the Story Sparker indicated which activities had taken place in the school. The percentage of visits during which each of five activities took place are presented separately by province in Figure 18 below. Reading aloud, and songs and games are the most common activities occurring in over 70 percent of visits. Story telling is also fairly common and is reported in just under 60 percent of visits. Interestingly, the least frequent activities are the DEAR period and reading clubs. Story Sparkers are involved in a DEAR period in 37 percent of visits in the Eastern Cape and 28 percent of visits in KwaZulu-Natal. They run, co-facilitate or mentor a reading club in only 15 percent of visits in the Eastern Cape and 28 percent of visits in KwaZulu-Natal. Figure 18: Story Sparker Reported Activities 76% 74% 37% 59% 15% 75% 72% 28% 59% 28% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% % of Reading aloud Songs and games DEAR Storytelling Reading club visits where activity takes place Story Sparker Activites Eastern Cape KwaZulu-Natal In addition to reporting which activities occurred during their school visit, Story Sparkers indicated whether the activity happened before, during or after the school day. Figure 19 shows for each 0 .1 .2 .3 Fraction of schools 0 2 4 6 8 10 Story Sparker visits 50 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV activity, what percentage of visits that the activity occurred during the school day as opposed to before or after school or in the school holiday. Other than reading aloud in KwaZulu-Natal, Story Sparker activities predominately take place during the school day. Figure 19: Percentage of visits activity happens during the school day 61% 65% 62% 45% 84% 57% 79% 88% 36% 73% 78% 75% 91% 62% 70% 71% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Reading aloud Running reading club Cofacilitating reading club Mentoring reading club leader DEAR Songs and games Storytelling Other Percentage of time activity happens during the school day Eastern Cape KwaZulu-Natal Table 23 summarizes additional information from the Story Sparker daily reports. Story Sparkers report frequent use of the Nal’ibali supplements and the hanging library books in their activities. The books are used in over 80 percent of their visits and the supplements are used in half the visits. When supplements are used, the most common use is Cut Out and Keep14 (72 percent) and the Story Corner15 (45 percent). The results in Table 23 also suggests some challenges with up-take. Story Sparkers report receiving no help from the Big 5 in 51 percent of visits and community members only assist in 10 percent of visits. Lack of support is specifically mentioned in 11 percent of Eastern Cape visits and 4 percent of KwaZulu-Natal visits. Overall, 43 percent of Story Sparkers report lack of support as a challenge at least once during the year. Access to learners is also mentioned as a specific challenge. This is somewhat unsurprising given that most Story Sparker activities appear to be scheduled during the school day. We will explore the issues around time for program activities in section 3.5 on the detailed case studies below. Critical to program sustainability is school ownership and engagement with reading for enjoyment activities during the year that the Story Sparker supports the school. The Story Sparker daily reports highlight challenges around school ownership and sustainability. They report that the DEAR period has not happened since their last visit 45 percent of the time in the Eastern Cape and 52 percent of the time in KwaZulu-Natal. They also find that the hanging libraries are not on display when they visit for 48 percent of visits in the Eastern Cape and 40 percent of visits in KwaZulu-Natal. Even more concerning, are the Story Sparker reports that reading clubs have not happened since their last visit in 70 percent of Eastern Cape schools and 67 percent of KwaZulu-Natal schools. It is clear that sustainability after the withdrawal of the Story Sparker is a concern for the program. These daily reports also suggest there is a serious dosage issue with reading clubs during the period of Story 14 Stories and pictures printed to be made into small books (‘Cut Out and Keep’ books) 15 Stories to read aloud (‘Story Corner’ stories) USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 51 Sparker reports. Reading clubs only happen during one in five Story Sparkers’ visits and very seldom between visits. Table 23: Story Sparker daily reports EASTERN CAPE KWAZULU￾NATAL Used the supplements 53% 47% Used the books 81% 83% Able to stick to plan for the visit 72% 83% Received help from Big 5 48% 49% Received help from community 10% 10% Had to substitute for a teacher 5% 11% Hanging library on display when arrived for visit 52% 60% DEAR periods had happened since last visit 55% 48% Reading clubs had happened since last visit 30% 33% Access to learners was a challenge 16% 11% Lack of support was a challenge 11% 4% Challenges with up-take are reflected in the following comments from two Story Sparkers: “They do no run the program if I am not there. Hanging libraries are not displayed they don't use them until I'm at school. Big 5 members are not supportive” “The educators always complain about time and they do not do anything in my absence” The Nal’ibali monitoring data on reading clubs provides a telling indication of the challenges with sustainability after the Story Sparker withdrawal. Figure 20 shows the percentage of schools with at least one and at least three registered reading clubs by cohort and year. For the first cohort, the percentage of schools with at least one reading club drops from 69 percent in the year that the school was supported by the Story Sparker to 47 percent in the following year and 23 percent the year after. While a much higher percentage (98 percent) of schools in cohort II had at least one reading club in the first year, only 53 percent of schools had a reading club in the second year. Nal’ibali expected SPS schools to register at least three reading clubs. Only 49 percent of schools realized this target in the first year of the program for cohort I, increasing to 68 percent of schools in cohort II. However, only 13 percent of cohort II schools registered at least three reading clubs in the second year of the program. By the third year of the program only 2 percent of cohort I schools had managed to sustain at least three reading clubs. 52 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV Figure 20: Percentage of schools with registered reading clubs, by cohort and year 69% 47% 49% 98% 23% 68% 23% 53% 2% 13% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Cohort I (2017) Cohort II (2018) Cohort I (2017) Cohort II (2018) At least one club At least three clubs Percent of school registering 2017 2018 2019 3.4.2 INSIGHTS FROM QUANTITATIVE EVALUATION DATA The endline interviews with learners, teachers and the person most knowledgeable about the SPS program in the school (SPS champion) provide us with further information about program up-take and sustainability. These interview were conducted in the third term of the second year of the SPS program. For the majority of schools, the Story Sparker weekly visits ended at the end of the first year. The SPS champion was asked to list the activities that the Story Sparker had conducted in their school. The results are summarized in Figure 21 separately by province. The figure shows a wide range of activities and considerable variation between the two provinces. The most frequently mentioned activity is leading cutting out and folding of Nal’ibali books. These are the “Cut and Keep” books included in the Nal’ibali supplement. Other activities most commonly mentioned are songs, games and read alouds in the Eastern Cape and read alouds, songs and activities with Nal’ibali supplements and books in KwaZulu-Natal. Figure 21: School reports on Story Sparker Activities, by province 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Led Regular drop everything and read (DEAR/DAAR) Gave ideas for drop everything and read Led Read alouds Led Reading club/s Helped set up reading clubs that teachers ran Led games Led songs Led cutting out and folding Nal’ibali books Led puzzle solving Led plays/drama Led drawing and colouring Did activities with Nal’ibali supplements/books Organized a book borrowing system Organized reading events Organized meetings w/parents about reading Percentage of schools reporting activity School reports on Story Sparker Activities KwaZulu-Natal Eastern Cape USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 53 Although the majority of teachers in all evaluation schools report that they have readers in their classrooms, the quantities tend to be inadequate with learners having to share. Learner reports on access to reading materials give another indicator of how available these materials are and the frequency of their use. Figure 22 shows the percentage of learners who report that there are readers or storybooks in their classroom by cohort and treatment status, separately by province. In the Eastern Cape, only 13 percent of learners in cohort I control schools have readers or storybooks in their classroom. This is in contrast to 44 percent of learners in comparative SPS schools. The difference between SPS and control learners is very similar for the second cohort in the Eastern Cape although the levels have risen for both treatment and control. This is likely due to the provision of the VulaBula readers to all Eastern Cape schools by the Eastern Cape Department of Education in 2018. Access to readers and storybooks in the classroom is almost universal in KwaZulu-Natal with little difference between SPS and control schools or cohorts. Figure 22: Learner reports on readers and storybooks in classrooms, by province 13% 23% 93% 97% 44% 50% 95% 95% 0% 20% 40% 60% 80% 100% Cohort I Cohort II Cohort I Cohort II Eastern Cape KwaZulu-Natal Readers or storybooks in classroom Control Treatment At baseline 58 percent of learners in the Eastern Cape and 44 percent of learners in KwaZulu-Natal reported that they have no books other than schoolbooks to read at home. Table 24 summarizes the impact of the SPS program on increased access to reading materials in the home for learners in SPS schools. In line with teacher reports, learners in SPS schools in both provinces are considerably more likely to report being allowed to take readers and library books home. Around a third of learners in SPS schools in the Eastern Cape and 45 percent of learners in KwaZulu-Natal report that they have some “Cut Out and Keep” books from the Nal’ibali supplement at home. A small number of learners in control schools also report Nal’ibali supplements at home reflecting the broader reach of the campaign beyond SPS schools. Across all these reading materials, learners in SPS schools are 29 and 28 percentage points more likely to have access at home than learners in control schools in the Eastern Cape and KwaZulu-Natal respectively. This increased access to reading materials for learners in SPS schools is unfortunately not accompanied with increased reports of reading at home or reading proficiency measured across a range of dimensions. 54 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV Table 24: Learner access to reading materials at home EASTERN CAPE KWAZULU-NATAL TREATMENT CONTROL TREATMENT CONTROL Readers/storybooks home from class (allowed) 27% 11% 39% 34% Books home from library (allowed) 17% 11% 25% 14% Nal'ibali supplements at home 34% 9% 45% 2% Any of above 51% 22% 70% 42% In section 3.3 above, we saw that teachers in SPS schools were much more likely to report having a timetabled period for reading (DEAR) than teachers in control schools. However, in SPS schools the DEAR period is not universal, frequent or consistent. Among teachers in SPS schools, three terms after the Story Sparker has left the school, 26 percent report no scheduled period for reading, only 24 percent report DEAR being scheduled daily. Furthermore, 26 percent report that the DEAR period is cancelled quite often or nearly always. In line with the monitoring data, we see a decline in the percentage of learners who report attending a reading club after the Story Sparker has left. For cohort II, the percentage of learners attending a reading club in SPS schools dropped 23 percent from in the first year of the program to 14 percent in the third term after the Story Sparker had left. The majority (61 percent in the Eastern Cape and 71 percent in KwaZulu-Natal percent) of SPS schools report no community involvement in reading activities. The following comment by one of the SPS champions resonates with the challenges with up￾take and sustainability raised by the Story Sparkers in their daily reports: “Most activities are not taking place anymore after the Story Sparker left. The teachers were happy to have a Story Sparker as she was helping a lot with reading and other activities.” 3.5 FINDINGS FROM STORY SPARKER CASE STUDIES AND INTERVIEWS This section reports on findings from the midline case studies. We consider the Story Sparker’s role and how it was realized in the case study schools. We also reflect on the successes and failures in relation to the Story Sparker functions, and the likelihood of SPS activities being sustained once they have departed from the school. 3.5.1 STRUCTURE OF STORY SPARKER ACTIVITY IN SCHOOLS In all four schools, children, teachers and the Story Sparker relied on public transport to get to school. The teachers mostly lived far from the school and travelled long distances to and from work. Most of the children and staff arrived at the school as the bell went, and there was a considerable amount of late-coming. This made before school activities difficult. Only at one of the four case study schools did a Story Sparker arrive prior to the bell ringing. All the others were late. Before school activities were therefore not possible. After school activities were not possible for the same reason. Many teachers lived far, and many of the children relied on public transport. There were also no activities during break. The reasons given were that the children wanted to play and eat, and in some schools, school-feeding occurs at this time. Thus the only time available for Story Sparker activities was during assemblies (which were very short and involve the whole school) and in the course of the school day. USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 55 All the Story Sparkers followed a similar routine at most of their schools. They went from class to class, or to classes where the teachers allowed them, and did activities with learners during normal instructional time. During the four observation days at the case study schools, teachers were seen to be interrupted on a number of occasions. Many of the teachers were reported to resent the interruptions. One of the Story Sparkers reported: “Sometimes they [the teachers] don’t want to communicate and when you come in the morning only one teacher will care about you. They say they are very busy in their classes. And they tell you to go after school, but I can’t go after school”. In describing a typical day, another Story Sparker explained: “I go to all the grades, not all the classes because there isn’t time. Some grades have 2 classes. I do different things. Usually I do the supplements in the morning. Sometimes it is difficult for us to take the classes. The teachers don’t want it. At 1 o’clock I run the reading club. The rest of the school cleans at this time”. There was variation across schools in relation to how much instructional time is ‘given’ to the Story Sparkers. In one of Story Sparker B’s schools, she had spent the whole year with the Grade 2 class as there was no teacher for that class. The Story Sparkers said that in some of their schools they were often asked to substitute. On other days, the amount of time they got varied with the school and with what was going on in the school at the time. There were time constraints in the day to day work of the Story Sparkers and difficulty in establishing routine. There were also time constraints in relation to the structure of activity across the year. Three of the four Story Sparkers complained about the amount of time they had to work in schools. They said that the mandated once a week was too infrequent to do their work. Story Sparker A said “I wish from Monday to Thursday I could work with ONE school, so that I can see the improvement. It’s hard to see improvement. Then another week I will go to another school”. In addition, reliance on public transport in areas with poor road infrastructure meant that there were often times the Story Sparkers were unable to visit certain schools, especially in the rainy season. Time was also lost to meeting the requirements of the ‘Milestones’ at district level. As the provincial coordinator pointed out: “The Story Sparkers are only implementing fully for about seven months. January is the training. Sept to Oct showcase”. Given that the Story Sparkers in this time were required to focus on the reading club (of 30 learners), and preparing displays and activities for the Showcase, other activities in the schools ceased. One of the Story Sparkers said, “Basically, after the Showcase things slow down”. 3.5.2 OBSERVATIONS OF FOUR STORY SPARKERS IN CASE STUDY SCHOOLS Each Story Sparker was shadowed for a whole day, with detailed notes taken of their activity. An interview was also conducted that probed some of the activities observed. Table 25 shows the activities undertaken by the four Story Sparkers across the four schools. It includes all engagements between the Story Sparker and the learners in four schools over the course of four days. All time that was spent by the Story Sparker managing resources, settling classes down and making arrangements with the teacher was removed. This totaled about 42 minutes. A breakdown of how the remaining 469 minutes (close to eight hours) was spent is shown in the table below. 56 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV Table 25: Time Breakdown of Story Sparker Activity over Four Days of Observation STORY SPARK ER (SS) ACTIVITIES DURATI ON TIME OF DAY GRADE 1 SS A School assembly – two learners read aloud part of two books (6 min.) 6 min. 7:58-8:04 Various 2 SS A Supplement cut out and keep (23 min.) Learner read aloud (7 min.) 30 min. 8:04-8:50 Grade 3 3 SS A SS reads aloud a story to learners (8 min.) Discussion & Questions on the story (4 min.) Three learners select books from hanging library and learners read aloud to class (7 min.). 27 min. 8:50-9:13 Grade 1 4 SS A Learners come up and read aloud poems (6 min.) Learner reads aloud from a book from hanging library while other learners act out the story (2 min.) 8 min. 9:18-9:26 Grade 4 5 SS A Play two games (16 min.) Cut out and keep bookmaking (27 min.). 57 min. 11:12-12:09 Various 6 SS B Plays game (5 min.) Cut out and keep bookmaking (26 min.). Learners cut and paste pictures of Nal’ibali characters from supplement onto paper (11 min.). Learners write sentence about what character is thinking (7 min.) 57 min. 7:52-8:55 Grade 2 7 SS B SS plays game with learners (10 min.) Learners listen to half a story read aloud on Nal’ibali radio (6 min.). SS discusses what happened in the story (5 min.). 21 min. 9:13-9:41 Grade R 8 SS B SS dances and sings with learners. Then plays a game (7 min.). Learners read aloud a supplement story at the front of the class (about 5 participate) (7 min.). The SS tries to ‘spark a story’ but learners don’t respond, become noisy. (5 min.) Learners draw (the cover of a book) (23 min.) 42 min. 10:48-12:00 Grade 3 9 SS B Reading club The learners sing. (9 min.) Learners cut and paste to make animal masks out of paper (31 min.). 52 min. 1:00 – 2:16 Various 10 SS C Sing song (2 min.). SS reads aloud supplement story to learners. (4 min.) Asks questions. Discusses story with learners (4 min.). 35 min. 8:10 – 8:44 Grade 1 USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 57 STORY SPARK ER (SS) ACTIVITIES DURATI ON TIME OF DAY GRADE Learners draw and color in a fish (23 min.). Sing (2 min.). 11 SS C About 6 learners try to read aloud story from supplement (14 min.). SS talks about story and asks a few questions (4 min.) Learners sing a number of different songs (7 min.). 25 min. 8:49-9:13 Grade 1 12 SS C Sing song. (3 min.) SS reads aloud story to learners and asks questions (9 min.). Learners draw fruit. (23 min.) 25 min. 9:20 – 10:12 Grade R 13 SS C Asks questions about a story in the supplement (4 min.) Learners cut and paste a hat out of paper (21 Min.) 25 min. 10:44-11:05 Grade 3 14 SS C Reading club. Learners sing (13 min.). Cut out and keep bookmaking (14 min.). 27 min. Various 15 SS D Learners sing songs (3 min.). A few learners go to the front and read aloud and operate a hand puppet at the same time (16 min.). Learners make a book (45 min.). Cut and paste to make a paper animal mask (15 min.). Some learners read aloud poem with mask in front of them (7 min.) Learners sing (2 min.). 88 min. 8:39-10:39 Grade 1 Story Sparkers’ activity was categorized according to four different types of activity: • Reading and engagement with text (generally discussion or questions and answers) • Cut, paste and drawing activities, which included making the supplement Cut Out and Keep • Songs and games • Writing Figure 23 below shows the percentage of total time spent on each activity. The graph shows that the majority of the time, 261 minutes (56 percent) was spent with learners doing cut, paste and drawing activities. This was followed by activities where there was engagement with text (reading or discussion), which took up 25 percent (120 minutes) of the time. Songs and games took up 81 minutes (17 percent) of the time and only one writing task was observed (representing 1 percent, or 7 minutes, of the total time). Below, we discuss in detail the issue of reading activity and the structure and management of all activities. 58 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV Figure 23: Time Allocation to Different Activities by Four Story Sparkers Over Four Days 3.5.2.1 READING ACTIVITY Overall, across the four Story Sparkers, around 25 percent of their time was spent engaging with texts, either in read alouds or discussion and question and answer sessions around a text. Notably, this engagement with text occurred only during normal instructional time with whole classes and not in the Reading Clubs. In the three Reading Clubs observed there was no reading. Activities in all three entailed singing and cut, paste and draw activities. Most of the read alouds consisted of learners reading aloud to the class, in much the same fashion they would in their conventional classes (perhaps not surprising given that they are in their conventional classes). The Story Sparkers themselves only read to children on three occasions, for a total of 21 minutes across the four Story Sparkers across four days. In other words, Story Sparkers read aloud to learners for 4 percent of the total time. Only two of the four Story Sparkers read aloud; Story Sparker B and Story Sparker D were not observed to read aloud at all for the entire duration of the observation. No story-telling was observed. On a number of occasions, the stories that were read were not completed. In Activity 15, for example, several learners repeatedly read the first page of the story, enacting it using a hand puppet. The story was never completed. Many of the learners struggled to read, and most of the class did not attend to their attempts. This was quite different to the listening of the Nal’ibali story on the radio and the occasions where the Story Sparkers did read. Here, learners listened intently and with enthusiasm. It was clear, therefore, that the learners were eager to hear stories, but the very weak reading by fellow learners did not manage to engage them. Part of problem of the learners’ reading we know – reading levels in these schools are very low. But the problem also stemmed from the fact that the Story Sparkers struggled to judge text difficulty and match the appropriate text to children. In Activity 15, the Story Sparker gave a group of Grade 1 learners an English poem to read that was far too difficult. She tried to get eight learners to read it, all the time gently cajoling and encouraging them. The text should have been abandoned and replaced with one that would have make the reading of poetry (and listening to it) enjoyable rather than impossible. In Activity 10 the Story Sparker repeatedly moved learners into the light when they failed to read a passage she had given them. It seemed she mistook their inability to grasp a too-difficult text 26% 56% 17% 1% 0% 10% 20% 30% 40% 50% 60% Reading and text engagement Cut, paste and draw Songs and games Writing USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 59 with an inability to see in the dark classroom. Matching an appropriate text to a learner’s ability is a relatively complex and specialized skill. Leaving the question of what texts should be used open to a Story Sparker, particularly one who does not know the learners very well, is problematic. Where there were read alouds, many of the strategies that might have been used to develop reading for enjoyment were absent in a familiar, recitation type reading by learners. This probably was largely due to the fact that the reading activities were taking place in normal classes, with large numbers of learners. This significantly constrained the SPS aims of developing reading for enjoyment: overall the observed reading was not fun, participation (in reading or listening) was not voluntary and it did not appear to engage the students’ interests or imagination. Only on one occasion (Activity 3) were children given the opportunity to select books. Here, for a period of seven minutes, three learners selected books from the Hanging Library and read aloud to the class. Although the activity was short, and the learners were slightly puzzled by being asked to do this, they enjoyed it immensely. In short, there was very little reading by the Story Sparkers; learners struggled to read themselves and did not provide engaging reading experiences for other learners; there was a mismatch at times between text level and learner ability; and there was little evidence of reading for enjoyment or its development. 3.5.2.2 STRUCTURING AND MANAGING PEDAGOGIC ACTIVITY The idea for SPS was derived from reading clubs that were conducted outside of conventional classroom spaces and times. A relaxed, intimate and enjoyable space, outside of the formality and strictures of the normal timetable was meant to provide a conducive context for pleasurable reading activity. The assumption regarding Story Sparkers, it seems, is that they do not require specialized pedagogical preparation for their tasks. However, the Story Sparkers are in a school environment and often in large, conventional classes, and their inability to manage the behavioral demands and instructional demands of the context were apparent. In particular, it was clear that the Story Sparkers had no preparation in how to manage behavior, how to structure an activity, how to support learners in doing an activity and how to appropriately pace an activity. Behavior Management In terms of behavior management, in most of the classes observed the class teacher was present during the session with the Story Sparker. Generally, the teachers were responsible for keeping order in the classes, but where they didn’t the Story Sparkers’ behavior management was similar to that of the teacher. For example, in Activity 11, while learners stood in the front of 43 learners reading the supplement aloud, the Story Sparker walked around sternly disciplining the learners and trying to keep them quiet. On two other occasions the Story Sparkers shouted at the learners. The context was not conducive to the kinds of social relations envisaged by the project and the Story Sparker fell back on the rather severe disciplining strategies of the teachers. Once in a less formal arena, in the Reading Clubs, the Story Sparkers struggled to maintain control. In Activity 14 it took the Story Sparker 13 minutes to settle the learners before she could begin, and it was only with several attempts at singing that she managed to focus the learner’s attention. Scaffolding, Pacing and Feedback The Story Sparkers have not been prepared for understanding how to scaffold activities, in other words provide learners with assistance and feedback as they undertake an activity. They also have no sense of appropriate pacing, or how much time should be allocated to activities. One example was Activity 12, where after a brief reading of a story a class of 98 Grade R learners were given a piece of paper and crayons and told to ‘draw a fruit’. No examples were provided and there was no discussion. Although the children clearly struggled, and their drawings were extremely rudimentary, no assistance 60 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV was given. The Story Sparker stood at the side of the classroom and watched the learners for 23 minutes, before collecting the completed drawings. In Activity 8, the Story Sparker handed out an A3 piece of paper and instructed learners to discuss in groups and “make a book”. No further scaffolding or instruction was given. The activity ended in chaos, with only one or two learners involved in drawing. When one learner showed the Story Sparker the picture she had drawn the Story Sparker looked but said nothing, shouting at the class to be quiet. Another learner had written a paragraph and when he approached the Story Sparker she offered no comment or question. The learner sat down again. In each case, the Story Sparker was unable to scaffold tasks, pace activities appropriately and feedback on what learners produced. These are specialized functions that teachers are trained to undertake. The Story Sparkers have not been trained to effectively manage the pedagogic demands of the activities set. Especially not in the context of large classes. Selected Purposeful Activity The Story Sparkers also clearly found difficulty in structuring purposeful activities themselves. In Activity 15, learners were instructed to continue making a book they had started within the prior lesson. Constructing the book entailed learners tracing pictures from a Cut Out and Keep, and then copying out the text next to the pictures. In other words, they were required to reproduce the Cut Out and Keep. The pedagogical value of this exercise is difficult to discern, as is any idea of how this will foster reading for enjoyment. This task takes 45 minutes to complete. Similarly, we see a lack of pedagogical purpose, especially one linked to developing reading for enjoyment in the cut, paste and draw activities. These take up most of the time that the Story Sparkers are engaged with learners. While learners seem to find the activities enjoyable, they are not linked to any reading activity. The books that are made are not read (or at least not in the same session, though there is often time) and the mask-making and hat making is not linked to a reading or story-telling activity. 3.5.2.3 DISCUSSION – STORY SPARKER OBSERVATIONS In the absence of pedagogical expertise, coupled with a great deal of autonomy in deciding what to do, with whom and when, the Story Sparkers resort to very basic, concrete activities which they themselves are clear on – cut and paste, games and songs. Reading, and in particular the discussion of stories and activities based on readings, is rarely seen. This is an issue that was acknowledged by the Provincial Coordinator who said that “Most of the activities are songs and games. So I think we need to limit the songs and games and explain to the Story Sparkers that the program is not just about this. It is also about reading and writing”. When asked why she thought there was such a strong focus on songs and games, she said: “Maybe they think it’s what is infusing joy. Sometimes I think it’s a way of getting out of things”. But looking at the contexts into which the Story Sparkers have been inserted, the chosen activities are possibly a response to very difficult teaching conditions. It is easier to manage a class all singing well-known songs than leading them in a group discussion of a story text. The Cut Out and Keep is an engaging and manageable task for learners which functions well in controlling their behavior. One implication of the foregoing discussion is the possibility that the Story Sparker’s work should be much more directed and structured, allowing for the possibility of scripting (say, of questions to ask after the reading of a text). The paucity of read alouds conducted by the Story Sparker also suggests that a minimum number of read alouds by the Story Sparker per week per school should be stipulated. Arguably, it is mainly the read aloud activities and text-based discussion that directly have the potential to develop reading for enjoyment. While the learners clearly enjoy the other activities immensely, they were about enjoyment without the reading. In order to create a carefree, personalized environment very careful attention needs to be paid to the structure and organization of USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 61 that environment. There is a need for clarity around how much time is to be spent on what games, texts, etcetera, in order for purposeful activity to happen. Whether the SPS activities take place within conventional instructional time or outside of that, the Story Sparkers need some training in classroom management skills. The structure, pacing and scaffolding of activities should be stipulated and where necessary scripted, at least until the Story Sparkers develop competence of their own. The fact that the Story Sparkers are using conventional classes raises a number of critical questions for the program: • Is it possible to realize the aims of the program when Story Sparkers work in regular, large classes? • Is the program comfortable with disrupting instructional time? Would the DBE condone the way in which time is being used in the school? Given that the schools do not have a timetabled DEAR period, de facto instructional time is being eroded. • Are the Story Sparkers equipped to deal with large groups of learners and structure meaningful reading for enjoyment activities under these conditions? 3.5.3 FACTORS IMPACTING ON THE UPTAKE OF THE PROGRAM IN SCHOOLS It is clear from the descriptions above that the program isn’t being implemented as intended. From the observations and interviews with the Story Sparkers, Heads of Department (HOD), Literacy Mentors and the Provincial Coordinator, a number of factors emerged that impacted on the uptake of the program in the schools. Each of these is discussed below drawing on the interview and observation data. 3.5.3.1 THE STRUCTURE OF THE SCHOOL DAY What is clear from the discussion above is that there is no time available for Story Sparkers to do their intended work – before and after school activities are not possible and break is short and taken up by other activities (in some cases school-feeding). As a result, the Story Sparkers, with uneven success, appeal to teachers to use some of their instructional time for SPS activities. This is done with the sanction and often under the direction of the Literacy Mentors. The notion of a ‘DEAR’ period is fudged in order to justify this. Few schools are running DEAR periods, however, and the work of the Story Sparkers is often disruptive and the SPS program is interrupting regular instructional activity. If the Story Sparkers were continuing to use class time there would need to be a clear agreement with the school about a DEAR period and it would be during this 30-minute period that the Story Sparker would be able to engage with learners, not interrupting other class time. The current situation is as frustrating to teachers as to the Story Sparkers who are often made to feel like a hindrance to teacher’s work. The program would also need to resolve when Reading Clubs happen, given that they require about an hour and cannot run before or after school. In some schools they currently use ‘extracurricular time’; in others they occur during normal school time. It is likely that in many schools they are not occurring at all due to the lack of a clear and stipulated time period in which they can take place. 3.5.3.2 SCHOOL AND TEACHER ACCEPTANCE The success of the Story Sparkers’ work in schools and thus the functioning of the program is dependent on the support offered by the school, especially the principal, HOD and teachers. The central challenge identified by the Story Sparkers was the lack of support from teachers in the schools. Reading Clubs were regarded as an additional burden, for which they had too little time. Time to 62 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV engage with and support the work of the Story Sparker was limited by the teachers’ obligations to other programs 16. The idea behind the Story Sparker is that they offer mentorship to teachers in reading for enjoyment. Given their youth and lack of experience in a school environment, in addition to their status in relation to teachers, it is unrealistic to expect that the Story Sparker is able to provide mentorship. There was no evidence of Story Sparkers taking on this role. School and or teacher resistance and apathy towards the program is a significant factor impacting on uptake. It also has serious consequences in terms of the persistence of the program once the Story Sparker leaves the school, which is low as we showed above. The idea of a committed and motivated group of Big 5 educators taking over the program is very unlikely to be realized. 3.5.3.3 NUMBER OF FUNCTIONS AND ACTIVITIES At the two levels of the program – the school and the district - the program has multiple activities. The range of activity expected of the Story Sparkers was described in section 1.1. When we consider that the Story Sparker spends one day in each school per week, the expectations are unrealistic. Both teachers and Story Sparkers indicated that the amount of time spent at a school was insufficient, especially with respect to certain activities. Establishing good relationships with community members and including them in the program’s work was identified as one aspect requiring time. In developing a culture of reading in general, one of the Literacy Mentor’s argued that there was too much time between visits, “it’s hard to get things going and keep it going”. The Story Sparkers also found the milestones relentless. Story Sparker C commented: “The Milestones too much. All these competitions (see section 1.1). We don’t have time to prepare. We have all these events and competitions”. Two of the Story Sparkers also complained that the dates for various events were often not clear, that these shifted multiple times and some events were announced very last minute. This made planning and establishing a routine difficult. The Literacy Mentors also experienced the program as having too many different activities: “The milestones are congested. There is no time to reflect. Now, we’ve just finished the Holiday program. Now it’s the reading showcase. There are too many things going on. There’s no time to reflect and learn”. The provincial coordinator concurred: “We are just going from one milestone to the next. There is no breather. No reflection. It’s too clenched up together, There’s no de-brief and no time for ironing out problems”. Reading Club Showcase In the course of interviews and observations it became clear that the Reading Club Showcase occupies a very central place in the program for the year. This is a two-day event where schools come together to ‘showcase’ what they have been doing. On the first day, three learners are invited to the event and the make a presentation – drama, songs or readings – that is judged. On the second day, 15 learners per school are invited to the event where they join in various games, songs and storytelling activities. The event is held during school time. In 2018, the event changed and involved four days for the winning learners who first participated in a circuit event and then in a provincial event. The Reading Club Showcase consumes a great deal of resources, given the transport and catering costs. The logistical work, which includes politicized negotiations with local providers, including taxis around transport, takes up much of the Literacy Mentors’ time. The event also entails weeks of work by the Story Sparkers in preparing the Reading Club members who will attend the event. The benefits for those who attend are articulated by one of the Literacy Mentors: “At that time of the year the children must reap what they sew. They have a chance to share their opportunities and show what 16 Jika Mfundo and NECT, for example USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 63 you can do. It is encouragement for everyone and that is important because in these communities there is no inspirations”. While the motivational aspect of the Showcase is clear, this would pertain to a relatively small number of children. Given that it consumes a considerable amount of financial and time and resources, the question is whether it benefits sufficient learners in a productive way. Further, there is cause for concern in the way in which reading -and reading for enjoyment- is constructed. Reading is about developing consistent, sustained, habitual practice. It is not an ‘event’. In some ways, the showcase (and preparation for it) models reading as a performance rather than practice. This was also seen in the assembly readings in the schools. They had little to do with a story (as the readings were seldom completed) and more about learner performance of reading. When the Story Sparkers were questioned about what they did at the showcase they all mentioned the displays they prepared and the ‘performance’. Story Sparker D described her different schools: “At School X we did acting and a puppet show. At School Y we did acting and a song. School Z we did a puppet show and a song. And at School D we did a puppet show and a song. The puppet show was the thing. Shooo. We had only 10 minutes to present.” In none of the descriptions did any of the Story Sparkers mention reading or books. 3.5.3.4 PROGRAM FOCUS AND SUPPORT From the observations and interviews it emerges that the central aims, energy and channeling of resources within the SPS program has become bifurcated. While increasingly more effort appears to be expended towards ‘milestones’, much less is harnessed for supporting the practice of the Story Sparkers at the level of the school – arguably the heart of the program. Literacy Mentors and the Provincial Coordinator’s time is consumed with the logistical and administrative requirements of issuing and responding to briefs concerning various training activities and the Reading Showcase. In the case studies. it was clear that the Story Sparkers lack the capacity to implement the program in a way that is likely to develop reading for enjoyment. The focus on games and songs, and the paucity of reading aloud and storytelling activities renders the program very enjoyable for learners, but unlikely to realize the central SPS aim. There are two key areas where Story Sparkers require greater support. The first is in garnering support and time at the level of the school for the program. Although an SPS champion was referred to in a couple of the 16 schools overseen by the Story Sparkers, meaningful support was rare. In particular, teachers have not taken on the activity of running reading clubs. The reasons for lack of support vary across schools, but predominantly teachers complain of additional work and the already significant demands of programs like NECT and Jika Mfundo. The second area in which Story Sparkers require support and development is around reading for enjoyment activities and how to develop reading cultures. Literacy Mentors also expressed a need for this kind of support. 3.5.3.5 UNDERSTANDING THE DEVELOPMENT OF READING HABITS The SPS program entails a comprehensive understanding of reading for enjoyment, and numerous activities and ideas around how to try and instill this in learners. The Story Sparkers clearly understand and embody the notion of ‘infusing joy’, and one of the most positive aspects of the program is the enthusiasm of the learners, who were seen, and reported to thoroughly enjoy the program. The Story Sparkers themselves were enthusiastic and positive. The observations and interviews do, however, call into question how and the extent to which the program is developing reading habits in the learners. Mathewson (2004) argues that simply having a positive attitude to reading does not lead to a reading habit. He argues that there are a number of mediating factors influencing a person’s decision to read. 64 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV These include external motivators like incentives, purposes, norms around reading of others and the settings and its literate practice outside of readers (Mathewson 2004:1436). The question is whether a very light touch book flood and a very light touch mentoring scheme for developing a reading culture in schools, can shift the external environment. This is particularly so when the central agent for change -the Story Sparkers- visits infrequently and is constrained in activity. We have also argued that through some SPS activities reading is constituted as a performance rather than a habit. In the observations there is very little evidence of learners seeing others read and doing so with enjoyment. A change in reading habit according to Mathewson (2004) also has internal mediating factors, in particular, the internal emotional state of the reader which stems from their ability to concentrate on a text and become fully engaged in the writer’s meaning and emotions. Where reading proficiency is very low, it is difficult to imagine how one might develop a learners’ sense of enjoyment of their own reading. Careful consideration needs to be taken of what texts are introduced and how they are introduced to learners. While the SPS program can certainly convey an enjoyment of stories, games, songs and books there is no clear way in which it will lead to a change in learners’ reading habits without them engaging in a positive way with texts. This is the nettle that the program needs to grasp: how to develop reading for enjoyment with learners who struggle to decode text, who have limited letter and word recognition skills and limited vocabulary. USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 65 4. COSTS The implementing partner, Nal’ibali, provided us with cost information collected following the USAID Education Cost Reporting Guidelines17. We received information about the expenses incurred by the program and also about the in-kind contributions they received from Government and non￾Government donors. Table 26 provides a summary of the costs and in-kind contributions. Annex V details all the costs by years and categories and the value that we assigned to the in-kind contributions, and the corresponding assumptions used. Table 26: SPS expenses and in-kind contributions, in US dollars TOTAL COST % OF TOTAL COST PER SCHOOL COST PER LEARNER COST Expenses: General Management and Operations $1,613,453 21% $2,241 $9.10 M&E and Reporting $248,935 3% $3456 $1.40 Teacher training $910,005 12% $1,264 $5.13 Teaching and Learning Materials $1,971,076 26% $2,738 $11.12 Parents/Community Engagement $1,211,987 16% $1,683 $6.83 Total Expenses $5,955,456 79% $8,271 $33.58 In-kind contributions: Teacher training $704,263 9% $978 $3.97 Teacher SPS activities time $821,188 11% $1,141 $4.63 Volunteer training $52,841 1% $73 $0.30 Volunteer SPS activities time $28,755 0.4% $40 $0.16 Total in-kind contributions $1,607,047 21% $2,232 $9.06 Total 7,562,503 100% $10,503 $42.65 Note: see details in Annex V Nal’ibali estimated that the SPS program reached 177,333 learners from grade R to grade 4 in the 720 schools treated. The last two columns of Table 26 present the cost per school and per learner using those estimates. The total amount spent by the SPS program per learner is $42.65. Of this total, approximately $11 per learner corresponds to the provision of teaching and learning materials. During the project, 2362 teachers were trained by SPS at a cost of $ 298.16 per teacher. In this cost we include expenses incurred by SPS and in-kind contributions that include teachers’ time, training venues, transportation costs, and office space. As we showed, there is no impact on SPS learners’ motivation to read, reading behavior, attitudes towards reading or reading proficiency, and therefore, we do not estimate cost-effectiveness. We found some positive impact for learners that chose to attend an SPS reading club, particularly in the 17 NORC did not directly verify the expenditures associated with the program. 66 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV Eastern Cape. Taking into account only the 23 percent of learners that attended a reading club in the year that the Story Sparker visited the school, the cost of SPS goes up to $ 186 per learner. Given the general lack of impact, the evaluation team decided not to conduct an estimation of the costs if the program were to be scaled up. USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 67 5. CONCLUSIONS AND RECOMMENDATIONS We have shown that the SPS program has generated some positive changes in the schools where it is present. SPS increased access to reading materials at school and in the home for learners in SPS schools. The SPS program also affected teachers positively in some dimensions. Although we do not see significant changes in attitudes or behaviors towards reading for enjoyment, they report an increase in conducting certain good practices in their classroom on a daily basis. Of the 13 practices we measured, teachers in SPSs are significantly more likely to report that learners retell stories that they had read, read with the teacher in small ability groups with the same reader and use a dictionary. Teachers trained by SPS are also more likely than teachers in control schools to allow learners to take readers home and to let learners choose their own reader. They are more likely to report a period in the timetable dedicated to reading activities and more likely to run a reading club at their school. However, these improvements did not translate into changes of learners’ reading proficiency, attitudes or behaviors towards reading, such as the time that they spent reading independently at home. Around 2315 percent of learners in SPSs attend reading clubs. When compared to learners in control schools with similar baseline characteristics, these learners appear to read on their own at home more frequently and for fun in the holidays. Reading club participants in the Eastern Cape also have higher fluency and better oral reading and productive listening comprehension than the matched comparison learners in control schools. The effects are however modest ranging from 0.15 to 0.26 standard deviations. We explored the potential reasons behind these results, making use of all the quantitative, qualitative and program monitoring data available. Taken together these data point towards four main problems: 1) Partial implementation. Story Sparkers visits to schools were less frequent than anticipated. Although teachers tend to make very positive comments about the program, the central challenge identified by the Story Sparkers was the lack of support from teachers in the schools. Story Sparkers’ reports suggest that there was a serious dosage issue with reading clubs during the period they visited SPS schools. Reading clubs only happen during one in five Story Sparkers’ visits and very seldom between visits. 2) Low take-up by the schools. Despite universal sign up amongst schools invited to participate in SPS, the program activities do not take place in the schools as expected. Nal’ibali monitoring data shows that a large number of schools did not fulfill the minimum expectations of the program and very few were meeting the target for all components. In 2018 –the best year of the program in terms of activities taking place at schools- only 15 percent of SPS schools in the Eastern Cape and 22 percent of schools in KwaZulu-Natal were recorded by Nal’ibali as meeting the required expectation for SPS components. Story Sparkers reported lack of support in the SPS schools, and even low-effort components of the programs are missing, for example, hanging libraries were not on displayed in almost half of the schools when the Story Sparkers arrived. Reading clubs require at least half-hour (although Nal’ibali recommends one hour) and are very unlikely to run before or after school for multiple reasons. Neither teachers nor community members have taken on the activity of running reading clubs. We saw a sharp decline in the percentage of learners who report attending a reading club after the Story Sparker has left and the majority of SPS schools report no community involvement in reading activities. The lack of commitment and apathy of the schools’ staff towards the program is a significant factor impacting on uptake and seriously compromises its sustainability. 3) Program limitations. There are areas of the program that need improvement. Although the program was expected to hold activities before and after the school day or during breaks, this was 68 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV clearly not possible. Story Sparker activities predominately took place during the school day, crowding out other activities and generating certain resentment from teachers. This is a central problem that needs to be addressed. Story Sparkers do not have pedagogical expertise and they have a great deal of autonomy. The case studies found that many SPS activities are very well received by the learners however they included very little storytelling, reading, or writing. There is very little reading by the Story Sparkers and learners struggle to read themselves and do not provide engaging reading experiences for other learners. Story Sparkers only spend a quarter of the time engaging with texts, either in read alouds or discussions, and text-related questions and answers sessions. Moreover, this only occurred during normal instructional time with whole classes and not in the Reading Clubs. In the three Reading Clubs observed there was no reading. Activities in all three entailed singing and cut, paste and draw activities. In general, there was little evidence of reading for enjoyment or its development. We recommend that the SPS program is reviewed taking into account the following considerations: • The feasibility of reading clubs needs to be revisited given the impossibility of before and after school activity • Increase the amount of reading the Story Sparker does. Consider making the Story Sparker’s work more structured and directed, specifying a minimum number of texts to be read and allowing for the possibility of scripting. For example, preparing the set of questions to ask after the reading of a text. • Develop a clearer concept of how to engender reading for enjoyment in learners who have very low reading proficiency • Revisit the issue of sustainability of the program once the Story Sparkers depart from the school. Acknowledge that Story Sparker’s are not in a position to mentor teachers. Are there ways other than Reading Clubs that teachers can take the program up long term? Are there other ways to gain teacher commitment to the program? • Weigh up costs and benefits of the Reading Showcase competition –or at least its frequency- in terms of program aims • More support needs to be provided for Story Sparkers in negotiating relationships with schools and in delivering reading for enjoyment activities. • In training on the use of Nal’ibali supplements focus on practical ways to use sections other than “Cut Out and Keep”. • Review the role of Literacy Mentors as literacy mentors to the Story Sparkers, providing the latter with support and development in their primary task – developing reading for enjoyment. Also, reinstate the literacy mentorship that was previously provided to Literacy Mentors. USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 69 4) Existing policies, programs and practices. The classroom observations show some significant changes in the nature of instruction in reading classrooms from existing research understandings. These are attributed to some large-scale, comprehensive reform efforts, including curriculum change, materials distribution and structured pedagogy reading interventions. The question arises as to how the SPS program was designed in relation to these reforms. While it possibly attempts to fill a neglected area of the CAPS curriculum – reading for enjoyment and for choice, this is undermined by the fact that the NECT and Jika iMfundo programs have greater levels of accountability and monitoring mechanisms attached to them. SPS and programs like it need to try and align their interventions more carefully with those that are already in place. The classroom observation study shows how the reading proficiency approach of the Jika iMfundo and NECT trumps reading for enjoyment activity as skills-related activities are specified and written into lesson plans for teachers. The broad instructional interventions, especially the CAPS, NECT and Jika iMfundo have strong pacing requirements built into their program, and make full use of curriculum time. This is a crucial aspect to take into account in trying to shift practice: whether there is time available for teachers to innovate or extend their practice beyond official requirements. While the goals of the SPS program are critical and laudable, the program did not take sufficient cognizance of existing policies, program and practices of teaching reading in order that their program could gain traction in the school lives of the pupils who so desperately need to develop a love of reading. 70 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV REFERENCES Ardington, C. and A. Menendez (2018) Impact Evaluation of USAID/South Africa Story Powered School Program – Baseline. NORC at The University of Chicago Bruhn M. and D. McKenzie, “In Pursuit of Balance: Randomization in Practice in Development Field Experiments”, American Economic Journal: Applied Economics, 2009, vol. 1, issue 4, 200-232 Case, A., Paxson C. and M. Islam, Making sense of the labor market height premium: evidence from the British household panel survey, Economics Letters 102 (2009), pp. 174–176. Deaton 2007 Height, health, and development. Proceedings of the National Academy of Sciences Aug 2007, 104 (33) 13232-13237; DOI:10.1073/pnas.0611500104 Draper, K., and S. Spaull, (2015). Examining Oral Reading Fluency among Grade 5 Rural English Second Language (ESL) Learners in South Africa: Analysis of NEEDU 2013. South African Journal of Childhood Education 5(2) pp.44-77 Gambell, T and D. Hunter (1999) Rethinking Gender Difference in Literacy. Canadian Journal of Education, 24 (1) pp.1-16 Hoadley, U. (2019) Impact Evaluation of USAID/South Africa Story Powered School Program – Classroom Observation Report. NORC at The University of Chicago Hoadley, U. (2019) Qualitative Study of the Story Powered Schools Programme: Case Studies of four Story Sparkers and their Schools. NORC at The University of Chicago. Hoadley, U. (2019) Impact Evaluation of USAID/South Africa Story Powered School Program – Classroom Observation Report. NORC at The University of Chicago Krashen, S. (2004). The Power of Reading: Insights from the Research. Libraries Unlimited: Westport; Evans, M.D.R. et al. (2010) Family scholarly culture and educational success: books and schooling in 27 nations. Research in Social Stratification and Mobility 28: 171-197; UK Department of Education (2012). Research evidence for reading for pleasure (available from www.gov.uk). Mathewson, G.C., 2004, ‘Model of attitude influence upon reading and learning to read’, in R.B. Ruddell & N.J. Unrau (eds.), Theoretical Models and Processes of Reading, pp. 1431–1461, International Reading Association, Newark, DE. Mullis et al., 2012 PIRLS 2011 International Results in Reading TIMSS & PIRLS International Study Centre, Chestnut Hill (2012) Needu (2013). NEEDU reading study 2013: the state of reading in Grade 5 in selected rural primary schools. National Education Evaluation and Development Unit. Available at https://nicspaull.files.wordpress.com/2014/12/needu-reading-study-2013_28oct14.pdf. PIRLS Literacy 2016: South African Highlights Report http://www.up.ac.za/media/shared/164/ZP_Files/pirls-literacy-2016-hl-report-3.zp136320.pdf Riordan, C. (1999) The Silent Gender Gap. Education Week, 19 (12) pp.46-47 Room to Read (2018) Data-Driven Methods for Setting Reading Proficiency Benchmarks. Prepared by Matthew Jukes, Chris Cummiskey, Melissa Gargano, and Peggy Dubeck. https://www.roomtoread.org/media/984466/room-to-read_fluency-benchmarking￾analysis_report_may-2018.pdf RTI International (2017) Setting and Using Benchmarks for Reading Performance. https://ierc￾publicfiles.s3.amazonaws.com/public/resources/10746_Benchmarks_brief_02-16-17_r6.pdf USAID.GOV SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM | 71 Spaull, N. Pretorius, E., & Mohohlwane, N. (2018). Investigating the comprehension iceberg: Developing empirical benchmarks for early grade reading in agglutinating African languages. RESEP Working Paper Series: 01/2018 Taylor D (2004) Not Just Boring Stories: Reconsidering the Gender Gap for Boys. Journal of Adolescent and Adult Literacy, 48 (4), pp. 290-298 Zuze and Reddy 2014, School Resources and Gender Reading Literacy Gap in South African Schools, International Journal of Educational Development, Volume 36, May 2014, Pages 100-107 72 | SOUTH AFRICA IMPACT EVALUATION FOR STORY POWERED SCHOOL PROGRAM USAID.GOV U.S. AGENCY FOR INTERNATIONAL DEVELOPMENT 1300 Pennsylvania Avenue, NW Washington, DC 20523