1 BASELINE REPORT April 2016 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by NORC at the University of Chicago. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. USAID IMPACT EVALUATION OF THE MAKHALIDWE ATHU PROJECT (ZAMBIA) GC-10F-0033M/AID-OAA-M-13-00010 2 USAID IMPACT EVALUATION OF THE MAKHALIDWE ATHU PROJECT (ZAMBIA) BASELINE REPORT April 2016 PN 7617.006.02 GC-10F-0033M/AID-OAA-M-13-00010 3 EXECUTIVE SUMMARY The Makhalidwe Athu project (MA) is an 18-month intervention aimed at improving the reading skills of 1,200 students in 2nd and 3rd grade in the Chipata and Lundazi districts of Zambia’s Eastern province. The project, funded by the All Children Reading Partners (USAID, World Vision and the Australian Government), and implemented by Creative Associates, will provide reading materials in ChiNyanja (the predominant local language) and support reading activities through SMS text messaging. The objective of MA is to provide short stories for 2nd and 3rd graders at low cost. Over a 9-month period, participant households will receive three text messages on their mobile phones each week. These three messages comprise a short story (e.g. 160 characters each) for children to read with their families. Children will be provided a notebook and encouraged to transcribe the stories. In addition, participants can call in for a pre-paid recorded voice message (IVR), which includes comprehension questions, as well as a recording of the story itself. Creative will also conduct periodic meetings with participant parents to talk about the structure of the program and how to read and listen to the stories with their children. Each month there will be community meetings to answer to any questions about the program, address problems with the SMS messages, and get feedback from participant parents about the program. NORC is evaluating the impact of the program on parent and student attitudes towards reading, time students spend reading at home alone, time students spend reading supported by other family members, and student reading test scores. For this evaluation NORC is conducting an experiment where school communities are assigned randomly to treatment and control groups. Baseline data was collected between November 2015 and January 2016. Specifically, NORC fielded a caregiver survey, a student survey, and an Early Grade Reading Assessment (EGRA). In January 2017, the same caregivers and students will be surveyed. The collected data from baseline and endline will be used to evaluate the program impact on children’s reading attitudes and EGRA scores.1 This report presents the results of the baseline data collection effort. We found that out-of-school reading resources are quite limited in these communities. The majority of children (58 percent) have no access to children’s books in ChiNyanja at home, a reader’s book, or reading activities in the community. The data also indicates that students’ reading skills are relatively low. Forty percent of the sample could not correctly pronounce a single letter, and two thirds could not read a single word. Finally, we also found that, despite having been randomly assigned, the treatment group tends to have better average values for indicators than the control group. This was observed in the analysis of the sociodemographic characteristics but more critically in the EGRA scores. Although these results suggest that the sample may be slightly unbalanced across some key variables, collecting longitudinal data will still allow us to properly approximate the estimation of causal parameters. 1 In addition, an uptake survey was fielded in June 2016. 1 TABLE OF CONTENTS EXECUTIVE SUMMARY.........................................................................................................3 LIST OF TABLES .....................................................................................................................2 LIST OF FIGURES ...................................................................................................................3 LIST OF ACRONYMS ..............................................................................................................4 A. PROJECT BACKGROUND ................................................................................................5 B. EVALUATION PURPOSE & EVALUATION QUESTIONS..............................................6 B1. EVALUATION PURPOSE............................................................................................. 6 B2. EVALUATION QUESTIONS AND INDICATORS ....................................................... 6 C. EVALUATION DESIGN AND BASELINE DATA COLLECTION ...................................8 C1. EVALUATION DESIGN OVERVIEW.......................................................................... 8 C2. SCHOOL AND STUDENT SAMPLE ............................................................................ 8 C3. BASELINE DATA COLLECTION.............................................................................. 10 C4. MIDLINE AND ENDLINE DATA COLLECTION ..................................................... 15 D. BASELINE RESULTS .......................................................................................................15 D1. HOUSEHOLD CHARACTERISTICS ......................................................................... 15 D2. OUT-OF-SCHOOL READING RESOURCES............................................................. 18 D3. STUDENTS’ READING HABITS AS REPORTED BY CAREGIVERS...................... 20 D4. SELF-REPORTED STUDENT READING HABITS ................................................... 22 D5. EGRA SCORES ........................................................................................................... 25 E. BALANCE..........................................................................................................................29 F. FINAL COMMENTS .........................................................................................................31 ANNEX I. DATA COLLECTION ...........................................................................................32 I1. ADVANCE TEAM ACTIVITIES.................................................................................. 32 I2. PARENT/CAREGIVER HOUSEHOLD SURVEY........................................................ 32 I3. EGRA AND STUDENT SURVEY................................................................................. 34 ANNEX II.FURTHER RESEARCH: INTRAHOUSEHOLD INVESTMENTS IN HUMAN CAPITAL Error! Bookmark not defined. ANNEX III. INSTRUMENTS..................................................................................................38 III1. ADVANCE TEAM EXCEL INSTRUMENT .............................................................. 38 III2. PARENT/CAREGIVER HOUSEHOLD SURVEY INSTRUMENT........................... 40 III3. EGRA/STUDENT SURVEY INSTRUMENT............................................................. 52 2 LIST OF TABLES Table 1. Summary statistics.......................................................................................................................................17 Table 2. Available reading resources......................................................................................................................18 Table 3. Students’ reading attitudes........................................................................................................................23 Table 4. Students’ reading habits................................................................................................... 23 Table 5. Child reads on their own at home ...........................................................................................................25 Table 6. Child reads with someone at home .........................................................................................................25 Table 7. EGRA scores and percent with zero correct answers ......................................................................26 Table 8. EGRA scores regressions – 2 nd grade.....................................................................................................27 Table 9. EGRA scores regressions – 3 rd grade.....................................................................................................28 Table 10. Household characteristics by treatment status.................................................................................29 Table 11. Available reading resources by treatment status.............................................................................30 Table 12. EGRA scores and percent with zero correct answers by treatment status.............................30 3 LIST OF FIGURES Figure 1. Household books by language (%)........................................................................................................19 Figure 2. Reasons why caregivers find it difficult or very difficult helping their children learn how to read (% of 499 households)...................................................................................................................................20 Figure 3. How often child reads on his own at home by having reading materials from school (% households) ......................................................................................................................................................................21 Figure 4. How often households members read with child at home (% households)..............................22 4 LIST OF ACRONYMS AT Advanced Team DEBS District Education Board Secretary EGRA Early Grade Reading Assessment IE Impact Evaluation INESOR Institute of Economic and Social Research IRR Inter-rater reliability ITT Intent-to-Treat MA Makhalidwe Athu NORC NORC at the University of Chicago RCT Randomized Control Trial RTS Read To Success project SMS Short Message Service TOT Treatment-on-the-Treated USAID United States Agency for International Development VAM Value-Added Model 5 A. PROJECT BACKGROUND The Makhalidwe Athu project (MA) is an 18-month intervention aimed at improving the reading skills of 1,200 students in 2nd and 3rd grade in the Chipata and Lundazi districts of Zambia’s Eastern province. The project, funded by the All Children Reading Partners (USAID, World Vision and the Australian Government), and implemented by Creative Associates, will provide reading materials in ChiNyanja (the predominant local language) and support reading activities through SMS text messaging. This intervention will be fielded in the context of the Read To Succeed project (RTS), a large-scale activity that provides teacher training and other services to improve reading outcomes in the country. The objective of MA is to provide short stories for 2nd and 3rd graders at low cost. Over a 9- month period, participant households will receive three text messages on their mobile phones each week. These three messages comprise a short story (e.g. 160 characters each) for children to read with their families. Children will be provided a notebook and encouraged to transcribe the stories. In addition, participants can call in for a pre-paid recorded voice message (IVR), which includes comprehension questions, as well as a recording of the story itself. The ultimate goal of the program is to improve reading skills of 2nd and 3rd graders in the treatment communities. Having access to age-appropriate and culturally-relevant reading materials is expected to positively affect children’s attitudes towards reading, increase the amount of time children spend reading at home, as well as increase the amount of time parents and other household members support children reading. A key feature of MA is that the stories will be constructed to reflect local culture and language. Through promotional campaigns (meetings, community events, and radio announcements) Creative staff will mobilize and inform communities about the project. They will target local authors, teachers, and other members of the community to submit local stories, folktales, and original content. After crowdsourcing stories, Creative will conduct writers’ workshops, adapting stories to the SMS format and leveling stories to children’s age profiles to provide relatable and engaging content. Furthermore, Creative’s literacy specialists will develop comprehension questions, which will be transmitted at the end of each story so parents and children spend time discussing the content and what they thought about the story. Creative will also conduct periodic meetings with participant parents to talk about the structure of the program and how to read and listen to the stories with their children. Each month there will be community meetings to answer to any questions about the program, address problems with the SMS messages, and get feedback from participant parents about the program. Submitting stories via SMS is intended to provide a low-cost channel to distribute a large number of stories, addressing the shortage of reading materials for early grade students that is prevalent in these communities. 6 B. EVALUATION PURPOSE & EVALUATION QUESTIONS B1. EVALUATION PURPOSE The purpose of the evaluation is to measure the impact of MA on the reading habits and skills of students in treatment communities. NORC will evaluate the impact of the program on parent and student attitudes towards reading, time students spend reading at home alone, time students spend reading supported by other family members, and student reading test scores. While the concept of employing information communication technology (ICT) for education is gaining popularity, the evidence around the effectiveness of mobile for reading (M4R) interventions is scarce2 . As a result, it is particularly important to properly document the impact of this program; if this intervention renders its expected impact, it could serve as a leading example of a low cost strategy to distribute reading materials in low income communities. B2. EVALUATION QUESTIONS AND INDICATORS The impact evaluation of MA seeks to address the following research questions: 1. Have the reading skills of Grade 2 and 3 students receiving the treatment improved as a result of the MA intervention? What are the magnitudes of these improvements, and in which reading domains have skills improved? 2. How and to what extent have student attitudes toward reading changed as a result of MA? Are students enjoying reading at home? Are they more likely to participate in out of school reading activities in the community as a result of MA? 3. Do parents spend more time supporting their children’s reading activities as a result of MA? How much time are they spending on reading activities as a result of MA? 4. Do students spend more time reading at home on their own as a result of MA? How much time are they spending on reading activities? 5. Are there any spillover effects of MA? Are other children in the household participating in the MA/SMS reading activities? Table 1 presents indicators, data sources and the analysis methodology associated with each research question. 2 Wagner, Daniel, et al. Mobiles for Reading: A Landscape Research Review. USAID: Washington, DC, 2014 June. 7 Table 1. Evaluation Design Matrix Research Question Indicator / measurement question Expected Outcomes Instruments Have the reading skills of Grade 2 and 3 students receiving the treatment improved as a result of MA? What are the magnitudes of these improvements, and in which reading domains have skills improved? Reading assessment scores (EGRA) Higher scores Baseline and Endline Early Grade Reading Assessments (EGRA) How and to what extent have student attitudes toward reading changed as a result of MA? Are students enjoying reading at home? Are they more likely to participate in out-of-school reading activities in the community? - Does [child] like to read? - Does [child] like to listen to stories? - Does [child] participate in reading activities outside home after school? How often? Increase in reported motivation to read, listen to stories and participation in after school reading activities Student questionnaire and Caregiver questionnaire Do parents spend more time supporting their children’s reading activities as a result of MA? How much time are they spending on reading activities? - How often do parents read with [child] and for how long? Increase in frequency/durati on of time spent by parents reading with child Caregiver questionnaire Do students spend more time reading at home on their own as a result of MA? How much time are they spending on reading activities as a result of MA? - How often does [child] read on his/her own at home and for how long? Increased frequency/durati on of time spent reading independently Student questionnaire Caregiver questionnaire Are there any spillover effects of MA? Are other children in the household participating in the MA/SMS reading activities? - Other than [child], have any of his/her siblings participated in reading MA/SMS? How often? Siblings participate in reading MA/SMS activities Caregiver questionnaire 8 C. EVALUATION DESIGN AND BASELINE DATA COLLECTION C1. EVALUATION DESIGN OVERVIEW To evaluate the impact of MA we conducted an experiment where school communities were assigned randomly to treatment and control groups. Randomization ensures that, on average, characteristics of the treatment and control groups are statistically identical, with the only difference being their participation in the intervention. Therefore, any measured difference in outcomes between the groups over time can be attributed to the program. Collecting baseline data is important even in the context of randomized controlled trials because it allows the evaluator to check if randomization was properly implemented. Testing for proper randomization involves conducting balance tests between treatment and control households for their main characteristics, and in particular for the EGRA scores. If randomization was successful, and no contamination of the treatment has taken place before baseline data is collected, no statistically significant differences should be observed between treatment and control groups in terms of EGRA scores at baseline (or any other characteristic). With a balanced baseline sample, any differences that are observed at endline can be attributed to the intervention. Collecting longitudinal data (that is, we will observed the same students at baseline and endline) is equally important, because it can be used to control for any differences that treatment and control groups may observe, even if treatment was randomized. C2. SCHOOL AND STUDENT SAMPLE Sample Size Calculation During the evaluation design stage, we conducted a power analysis to determine an adequate sample size for estimating impact. We assumed an alpha of .05, a one-tailed test, an intracluster correlation coefficient of 0.1, and that the baseline EGRA score explains roughly 30% of the variation in the endline score. With a sample size of 80 schools and 30 students selected from each school, we have a power level of .80 to detect a standardized effect size of 0.18 (i.e., EGRA score is 0.18 standard deviations higher among the treatment than among the control students). Assuming 20% attrition at the student level (30*80%=24), we have a power level of .80 to detect a standardized effect size of 0.19. We have enough power to detect any difference larger than this scenario. School Selection NORC matched schools in pairs so their background characteristics (namely the number of students and the distance to the District Education Board Secretary - DEBS) are observationally equivalent. We randomly chose 40 of these school pairs. In each pair one school was randomly 9 assigned to the treatment and the other to the control group. Note that although the program is not school-centered, to participate in the program children need to be enrolled in a school. The school sample frame is the list of all public schools in the districts of Lundazi and Chipata. The list for each district was provided by the corresponding District Education Board Secretaries (DEBS). These lists contain information on the number of students by grade. There are 210 schools in Chipata and 140 in Lundazi, for a total of 350 schools. We matched school pairs using the number of students and distance to DEB for each school to make sure that, at least across these two characteristics, schools in the treatment and control groups were observationally equivalent. To construct the pairs we followed these steps: 1. Calculate the quartiles of the number of students and terciles of the distance to DEBS, and use these variables to divide the sample in twelve blocks or cells. 2. Randomly select 40 school pairs in proportion to the total number of schools in the cell. 3. Randomly assign one school from each of the 40 pairs is to the treatment and the other to the control group, resulting in a sample of 40 schools in the treatment group and 40 schools in the control group. Student Sample Given that the MA intervention centers on the delivery of text messages and voice recordings to parents and caregivers, participation in the MA intervention requires access to a working cell phone. Because Zambian school rosters do not contain information on the cell phone ownership of parents, the student sample frame (that is, the list of all eligible students from which to sample students from each treatment and control schools) was not readily available and needed to be constructed as part of the data collection effort. To construct the sample frame, Advanced Teams (AT) of enumerators were sent to the field to survey all students in grades 1 and 23 across all 80 sampled schools and ask if anyone in their home has a cell phone. In total, the ATs attempted interviewing 8,681 students, out of which 4,910 (56.5 percent) reported having a cell phone at home. 2,354 students (27.1 percent) were absent during the AT visit, so the cell phone ownership rate of 56.5 percent can be considered a lower bound of the true cell phone ownership rate.4 From the students that reported having a cell phone at home, AT randomly selected 30 students (15 1 st graders and 15 2 nd graders) as well as 10 additional students from each grade as 3 Because the sample is constructed at the end of the school year (November 2015), we interview students in grades 1 and 2 so most of them will be in grades 2 and 3 in 2016. 4 The presented cell phone ownership rate corresponds to the rate of the sample of interviewed students, not the population they represent. In fact, as it’s explain in more detail in the data collection section, in some schools not all students were surveyed by the AT but at a sample of them. Specifically, if in a given grade there were more than 90 students, AT enumerators randomly selected and interviewed only 60 students, so covering large schools was feasible in the time allotted. Along these lines, if the data is weighted incorporating this sampling design the cell phone ownership rate is 58%, so the difference is not really major. In this report all the presented analyses correspond to unweighted figures. Working with unweighted data makes it easier to track the sample construction process. 10 replacements in case some of the originally selected students' households were not eligible or declined participation. However, it was not always possible to select 30 students from each school because there were not always enough students who reported that their parents had a cell phone; in those cases, all students that reported their parents had a cell phone were included in the sample. In total 2,362 students were sampled – 98.6 % of the planned 2,400. The team planned to sample 1,600 replacements; however, given that several schools did not have 30 students who reported having parents with cell phones, enumerators selected 1,283 students – 80% of the planned 1,600. Note that during the construction of the sample frame and replacement list, it is possible that two students from the same household were sampled because they were siblings. C3. BASELINE DATA COLLECTION To collect data on the indicators outlined in Table 1, we deployed baseline data collection in three phases: 1. Advanced Team phase (October 2015), the purpose of which was to construct the sample and replacement frame. 2. Parent/Caregiver Household survey data collection (November 2015), the purpose of which was to collect data on socio-demographic characteristics of the household members (age and education level, reading habits), as well as some information on household wealth (assets). We also collected data on how much time parents spend reading with their child, how much time the child reads on her/his own, whether s/he participates in reading activities after class in the community, and if other children of the household participate in the SMS/MA activities (at endline). The data collected with this instrument inform research questions 2 to 5. 3. Early Grade Reading Assessment (EGRA) and student questionnaire data collection (January 2016)5 , the purpose of which was to evaluate students’ reading skills; in particular, letter sound identification, oral reading, reading comprehension and listening comprehension (an input into research question 1), as well as ask children about their reading habits (e.g. if they like to read, how often they like to read, and for how long they read on their own)— inputs into research questions 1, 2, and 4. All data collection activities were carried out in partnership with the Institute of Economic and Social Research (INESOR) at the University of Zambia. For EGRA data collection, our partner School-to-School International conducted the enumerator training, data quality oversight, and a data quality review. 5 We were unable to conduct baseline data collection earlier than October/November 2015. However, Grade 7 and Grade 9 school examinations in Zambia can occur as early as November; during these periods, lower grade students are usually not in attendance at school, making it difficult to conduct EGRA data collection in November. However, Creative’s implementation timeline involved having orientation interviews on the MA intervention with parents in December. To ensure we did not delay Creative’s implementation, and to accommodate the end of the school year, we split baseline data collection between the parent/caregivers in November, and students in January at the start of the school year. This was done with the understanding that exposure to one orientation meeting would be unlikely to influence baseline EGRA scores. 11 Advanced Team Data Collection Only parents/caregivers with cell phones and their children were eligible to participate in the study. To construct the list of students eligible for the intervention (e.g. the student sample frame), INESOR sent out one AT to each of the selected 80 schools for the study, with one enumerator covering one school in a day. To construct the student sample frame, enumerators first asked head teachers for the Grade 1 and Grade 2 rosters, and digitized these rosters using excel. For schools with under 90 students in a grade, the AT asked all 1st and 2nd graders listed on the roster if anyone in their household had a phone. To be able to cover large schools (defined as schools where there are more than 90 students in a grade) in the time allotted, 60 students were randomly selected from the grade roster and interviewed. Upon visiting the classroom, the AT would call out names, and mark whether a student was absent on the day. For students that were not absent, AT members pulled students aside individually to: (a) ask for consent, using a script approved by the NORC Institutional Review Board; (b) ask for the student’s first and last name (confirming this was the name on the roster); (c) ask for the parent’s name; and (d) ask a simple pre-screening question: “Does anyone in your house have a cell phone?”. After the sample frame was constructed, enumerators randomly selected the sample of students (15 from each grade and 10 replacements from each grade—for a total of 50 students per school). Enumerators left notes with the head teacher to distribute to the parents of the selected students, informing them about the possibility of being contacted in the following weeks regarding the survey and for participation in the program. Parents were asked by the head teacher to return the notes within a week of receiving them. Follow-up cars visited all schools a week after notes were distributed to pick up returned notes. Returning the note was not required to participate in the survey (or the program). For the caregiver survey, enumerators had to locate and interview parents of sampled children even if they had not returned the note. More information on the training, fieldwork, and data quality review process for this phase of data collection can be found in Annex I. E1. The consent statement, parent notes, and screenshot of the excel instrument that teams used can be found in Annex 2. F1. Parent/Caregiver Household Data Collection The parent/caregiver household survey was developed by NORC and INESOR. The household survey includes a series of screener questions, intended to determine the respondent’s eligibility to participate in the study. This screening criteria was agreed upon by NORC, Creative, and USAID, and included questions about the respondents’ interest in participating in the intervention and the respondents’ access to a working cell phone number. The survey was pre-tested with 15 to 20 ChiNyanja-speaking respondents that closely match the respondent profile to test both the content, skip patterns, phrasing of questions, and the appropriateness of the translation. The sections of the parent/caregiver household survey include the following: 12 Section Title Content A Introduction and Consent Consent to participate in the study Screener/eligibility questions Consent to participate in MA Consent to administer EGRA B Home Literacy Environment Household roster: education, age, reading level, time and frequency spent reading with child Child demographics (gender, age, grade) Reading materials at home Child reading practices out of school Parent attitudes towards reading at home C Household Assets Household assets including: livestock and agricultural land, floor material, household items, child items, electricity, cell phone reception D Follow-Up Information Moving and relocation information, enumerator and supervisor comments The baseline parent/caregiver household data collection phase occurred between mid￾November and early December 2015, over the course of 2-3 weeks, with enumerators visiting each school-community over the course of a day. The parent/caregiver household survey was intended to take approximately 30 minutes to administer; on average, it took approximately 40 minutes. In the case that multiple children from the same household were sampled, household-level questions were only asked once, and only child-specific questions were asked twice. To ensure high quality data, NORC deployed tablet data collection. NORC programmed the survey in SODA, testing the instrument extensively on tablets prior to its deployment in the field. Training occurred over the course of one week, familiarizing enumerators with the study and intervention, informed consent, interviewing techniques, and the content of the questionnaire. The pilot occurred in one school in Chipata and one school in Lundazi over the course of one day, with a debrief occurring after the pilot on lessons learned, and a brief refresher the day after. Teams left for the field one day after training ended. 2,223 household interviews were completed, corresponding to a total of 2,398 students.6 In the cases where multiple children from a household were sampled, the household was only interviewed once, with child-specific questions (e.g. those related to a child’s reading habits) asked separately for each child and questions common to the household (e.g. household assets) asked once. During the caregiver survey, the replacement rate, calculated as the number of replacement households interviewed as a fraction of the total number of interviewed households, was 14.4 percent. The main reason why households could not be interviewed was because members of the household were not home or available to interview after repeated visits (a total of 149 households). It is important to mention that only 68 households were screened out because they did not have a working cell phone, equivalent to 3 percent of the 6 9 extra interviews were conducted but they were dropped because they were not in the sample (or replacement) list. 13 households originally sampled. This implies that the 56.5 percent lower bound of the cell phone ownership rate that was estimated above using students’ responses, could be actually lower. However it is difficult to estimate the prevalence of cell phone ownership using this data because we do not observe actual cell phone ownership in households of students that were screened out by ATs. More details about training, the data quality control process, and fieldwork can be found in Annex I. The instruments can be found in Annex 2. EGRA and Student Data Collection The Early Grade Reading Assessment (EGRA) aims to evaluate foundational reading skills that a child must have to read fluently with comprehension. To assess the reading skills of sampled students at baseline, NORC employed the 2014 Zambia EGRA, developed by RTI and used by several USAID-funded programs in Zambia including Read to Succeed and Time to Learn. The 2014 EGRA, which was also used for the 2014 National Assessment, contains 5 subtasks in ChiNyanja and 2 subtasks in English. Because this intervention is aimed at improving ChiNyanja reading skills, and also due to feedback provided by those involved in administering EGRA in 2014 regarding the quality of the English subtasks, USAID and NORC agreed to only administer the 5 ChiNyanja subtasks: Orientation to Print, Letter Sound Identification, Non-Word Decoding, Oral Reading Passage/Reading Comprehension, and Listening Comprehension. Other slight modifications were made to the EGRA upon its review; those are outlined in more detail in Annex I E3. Appended to the EGRA was a short student questionnaire to capture the child’s self-reported out-of-school reading activities. The questionnaire was developed by NORC/INESOR with feedback from USAID and Creative; INESOR translated the questionnaire using double translation and back translation. It was pre-tested with 15-20 students, with adjustments made to the instrument as necessary, and was designed to not take more than 10-15 minutes to administer, with the administration of EGRA and the student questionnaire not to exceed 30 minutes. The structure of the EGRA + Student Questionnaire instrument is presented below: Subtask Title Content 0 Introduction and Consent Assent to participate in study 1 Orientation to Print (Untimed) 3 questions. Indicate where one begins reading printed text on a passage and the direction one would read text. 2 Letter Sound Identification (Timed) Produce sounds of 100 letters presented in written form. Presented in a grid of 10 rows and 10 columns. 3 Non-Word Decoding (Timed) Sound out, or decode, unfamiliar words. Asked to read out 50 words without meaning. 4a Oral Reading Fluency (Timed) Read a passage of narrative text of ~60 words in length. 4b Reading Comprehension (Untimed) 5 questions. Respond to five questions asked about above passage. 5 Listening Comprehension (Untimed) 5 questions. Oral response to listening comprehension questions. 6 Student questionnaire Reading practices in the classroom, at home: both alone and 14 Subtask Title Content with others. Frequency of reading and reading preference. Student attendance at school and teacher attendance at school. The launch of the impact evaluation and timing of the parent/caregiver household data collection effort coincided with the end of the school year in Zambia. After discussions with local staff, Creative, and the Ministry of Education, USAID and NORC determined that because students often do not attend school starting in mid-December during Grade 7 and Grade 9 National Examinations, the EGRA and student questionnaire would be administered in the first two weeks of the school year, in January 2016. NORC’s subcontractor, School-to-School International (STS), programmed all tools into Tangerine. NORC/STS carried out the training of enumerators and supervisors in the administration of EGRA in collaboration with INESOR. Full enumerator training lasted 6 days, followed by a pilot test, debriefing, and an additional day of practice for enumerators that needed it. In January 2016, enumerators visited schools to conduct the EGRA, examining 2,260 students. If children were not located at their school, enumerators visited the children’s households to interview them at home (this occurred in about 5% of all interviews completed). Six percent of children were unable to be interviewed because their family had moved, the child was not locatable at home, the child’s parent did not permit the interview, or the child was too ill or had a disability that prevented them from participating in the assessment. Additionally, 26 students from one of the control schools, whose caregivers were surveyed in December, did not take the EGRA exam because it was not possible to visit the area due to flooding. Therefore, while we have baseline caregiver survey data for 2,398 students, baseline EGRA data is available for 2,260 students. Note that instead of dropping the 138 students from the caregiver data that do not have EGRA data in all analyses, we include them in the descriptive statistics on caregiver data. Planned and observed samples Planned Observed Schools 80 79 43 in Chipata and 36 in Lundazi Students 2,400 2,398 with household survey but no EGRA data 2,260 with household survey and EGRA data, 52 percent in 2nd grade and 48% in 3rd grade in 2016 More details about training, the data quality control process, and fieldwork can be found in Annex I: E3. The instrument can be found in Annex 2: F3. 15 C4. MIDLINE AND ENDLINE DATA COLLECTION In January 2017, the same caregivers and students will be surveyed7 . The collected data from baseline and endline will be used to evaluate the program impact on children’s reading attitudes and EGRA scores. The caregiver instrument will be complemented with take-up questions (e.g. how often parents/caregivers opened the SMS and read them with the child, how often they attended the community meetings organized by the program implementer). This information will help document not only the average effect of offering the services, but also the effect of families actually using these materials. In addition to an endline assessment, a midline assessment on take-up will also be conducted on a small subsample of parents (approximately 280). The purpose of this midline assessment will be two-fold: to better understand take-up of the intervention among parents and to provide information to Creative on potential improvements to implementation. D. BASELINE RESULTS Below, we present the results from the Parent/Caregiver household survey and Early Grade Reading Assessment/Student Questionnaire and review the main conclusions. D1. HOUSEHOLD CHARACTERISTICS 7 Note that students from households who did not take EGRA during baseline will, if they are able and willing, be administered EGRA at endline. These students could serve as replacements for students surveyed at baseline but that are not found at endline. 16 Table 1 shows summary statistics for the household sample. Fifty-three percent of respondents are male. While literacy seems relatively low in these communities, with only 53 percent of mothers8 able to read and 79 percent of fathers able to read, average years of education appears surprisingly high. On average, mothers have 5.6 years of education while fathers have 7 years of education. It is possible respondents tend to overstate their education level; it is also possible that these figures accurately reflect their actual years of education but that the quality of the education they received was not very high. On average, households have 5.3 members in this sample. 8 Father is defined as the male parent and mother as the female parent. 17 Table 1. Summary statistics Household characteristics Respondent is male 53% Mother knows how to read 57% Father knows how to read 79% Average years of education (mother) 5.6 Average years of education (father) 7.0 Average age (mother) 34.3 Average age (father) 40.7 Household size 5.3 More than one child is surveyed (siblings) 7.9% Household assets Average number of plots 2.2 HH owns television 28% HH owns bed 54% HH owns bicycle 75% HH has electricity 6% HH owns radio 61% N 2,221 Source: MA Baseline data. Sample sizes for some variables are lower due to item-specific missing data. The descriptive statistics of selected household assets are also displayed in 18 Table 1. On average, households have 2.2 plots. Twenty-eight percent of households own a television, 54 percent own a bed, and 75 percent own a bicycle. Access to electricity is very rare in these communities (6 percent). 9 Approximately 61 percent own a radio. D2. OUT-OF-SCHOOL READING RESOURCES One of the main channels through which MA aims to improve students’ reading skills is by providing students with reading resources that are properly leveled, engaging, and accessible outside of school. Along these lines, it is important to determine to what extent children in this community already have access to adequate reading materials, or if the stories that MA will send can be considered a relative scarce resource. In this subsection we review the availability of reading materials as reported by households. Table 2 shows whether students have school reading materials (e.g. reader’s book, reading cards, or reading passages) and if they participate in afterschool reading activities; these include reading clubs, activities at the community center, church activities (involving reading), and reading with friends, among other options. Only a third of caregivers report that their child has a reader’s book or other reading materials from school. Moreover, only 12 percent of caregivers say that there are afterschool reading activities in their communities. In the few cases where afterschool reading activities are provided, 60 percent of students in the control group participate in them, while the figure for the treatment group is 63 percent. Table 2. Available reading resources Child has reading materials 36% Afterschool reading activities are available 12% -Child participate in afterschool reading activities 61% N 2,390 Source: MA Baseline data. Figure 1 shows the fraction of households that have no books, one to five books, six to ten books, and more than ten books in English and ChiNyanja. Panel A corresponds to books for adults. In 55 percent of households there are no books for adults in English, and in 39 percent of households there are no books in ChiNyanja. In 31 percent of households there are between one and five books in English, while in 48 percent of households there are between one and five books in ChiNyanja. Very small fractions of households have more than five books in any language. Panel B presents the results for children’s’ books. Sixty-nine percent of households report not having any children’s books in English, with 62 percent reporting no children’s books in ChiNyanja. Twenty-four percent of households have between one and five books in English and 9 It may be puzzling that the fraction that owns a working television is so much higher than the fraction that has electricity. The specific question on electricity referred to whether households were connected to the electric grid. Our field team explained that while most households are not collected to the electric grid, many may have access to ‘local’ forms of electricity, like solar power. 19 in 33 percent of households have between one and five books in ChiNyanja. Again, very few households have more than five books in either language. Figure 1. Household books by language (%) A. Adult books (% of households) B. Children books (% of households) Source: MA Parent Questionnaire data. Sample sizes varies across category/language of books due to item-specific No response. Overall, the scarcity of reading resources in these communities is widespread. Moreover, 58 percent of students do not have access to any of the three surveyed resources (reader’s book, books at home, and available resources at the community). In addition to these objective measures of reading material availability, caregivers were also asked if they thought that the lack of these types of resources makes it difficult for them to help their children learn how to read. They were asked first if they found that helping their children learn how to read was very difficult, difficult, neutral, easy or very easy. Twenty-two percent of the sample (499 households) report that they find it difficult or very difficult to help their children. Figure 2 displays the reasons why parents find helping their children learn how to read difficult or very difficult. 55 31 5 10 39 48 5 7 0 10 20 30 40 50 60 70 80 None 1-5 6-10 More than 10 English (N=2385) Nyanja (N=2385) 69 24 3 3 62 33 3 2 0 10 20 30 40 50 60 70 80 None 1-5 6-10 More than 10 English (N=2386) Nyanja (N=2388) 20 Figure 2. Reasons why caregivers find it difficult or very difficult helping their children learn how to read (% of 499 households) Source: MA Parent Questionnaire data The results indicate that the lack of proper reading materials is an important reason for why caregivers find it difficult to help their children learn how to read, second only to not being able to read well enough to teach their child. The third most frequent reason is that parents do not know how to help their child. Other reasons with lower prevalence point to lack of time (caregiver’s or child’s) and that the child is not interested. In summary, the data showing a scarcity of out-of-school reading resources indicate that the stories provided by MA may fill an important gap. The majority of children (58 percent) have no access to children’s books in ChiNyanja at home, a reader’s book, or reading activities in the community. Moreover, among caregivers that find it difficult or very difficult to help their child learn how to read, 37 percent say that the lack of reading resources is one of the reasons why. D3. STUDENTS’ READING HABITS AS REPORTED BY CAREGIVERS Parents were surveyed on their children’s reading habits at home. Not surprisingly, the frequency at which caregivers report children read at home on their own is correlated to whether children have reading materials from school (e.g. reader’s book), and to whether there are children books in ChiNyanja at home. Figure 3 shows the frequency at which children read at home on their own by reading resources availability. In Panel A results are displayed by whether or not the child has reading materials from school, and in Panel B by whether or not there are children books in ChiNyanja at home. The figure in Panel A shows that roughly half of the children that do not have reading materials from school never read at home on their own according to the caregiver, while only 30 percent of the children that have reading materials never read at home. A similar pattern can be detected when we look at the results by availability of children books in ChiNyanja (Panel B). 1 3 7 16 23 28 37 39 0 5 10 15 20 25 30 35 40 45 Do not think that helping child read outside of school will make a difference Other Child does not have enough time (e.g. he/she has to work) No interest from child Do not have enough time Do not know how to help child read Do not have access to appropriate reading materials Cannot read well enough to teach child 21 Figure 3. How often child reads on his own at home by having reading materials from school (% households) A. By availability of reading materials from school B. By availability of children books in ChiNyanja Source: MA Parent Questionnaire data. Figure 4 shows the frequency at which household members read with the focal child at home, by availability of reading resources. The caregiver survey collected data on how often each household member reads with the child, so we present data for the household member that most frequently engages in this activity.10 There is a positive correlation between resource availability and frequency at which household members read with children. In Panel A we can see that among the children that do not have reading materials from school, their household members never read with them in 19 percent of the cases, while in the case of students that do have reading resources from school, that fraction is only 10 percent. Also, 24 percent of children that do not have reading materials from school have a household member that reads with them four days a week or more, while in the case of children that do have reading resources from school, the fraction of children that have a household member that reads with them four days a week or more is 36 percent. A similar pattern can be detected in Panel B, where the results can be observed by availability of children books in ChiNyanja. 10 A more detailed analysis on the data that was collected on how much time each household member spends reading with the focal child is presented in Annex II. 22 Figure 4. How often households members read with child at home (% households) A. By availability of reading materials from school B. By availability of children books in ChiNyanja Source: MA Parent Questionnaire data. D4. SELF-REPORTED STUDENT READING HABITS Children were surveyed on their attitudes towards reading and their reading habits. Table 3 shows children’s responses on a series of questions related to their reading preferences. Roughly half of the students report that they like listening to stories, a quarter answer that they are indifferent, and the rest report that they do not like it. A number of questions were asked on whether children like reading or practicing reading. Sixty-seven percent report that they like reading or practicing reading at home on their own, a quarter report that they are indifferent and the rest say they do not like it. Students report similar answers when asked whether they like to read with someone at home. Regarding children’s attitudes towards reading at school, 70 percent report that they like reading or trying to read in class alone; 23 percent report they are indifferent and the rest report they do not like it. Similar results were found for whether students like to read in class out loud. 23 Table 3. Students’ reading attitudes Do you like to listen to stories? I like it 56% 1,217 I do not like or dislike it 25% 537 I do not like it 19% 417 Do you like to read or try to read on your own at home? I like it 67% 1,026 I do not like or dislike it 25% 392 I do not like it 8% 124 Do you like to read or try to read with someone at home? I like it 69% 1,195 I do not like or dislike it 23% 396 I do not like it 8% 139 Do you like to read or try to read in class alone? I like it 70% 1,010 I do not like or dislike it 23% 336 I do not like it 7% 101 Do you like to read or try to read in class out loud? I like it 70% 950 I do not like or dislike it 21% 287 I do not like it 8% 114 Notes: N=2,195. Sample sizes are lower for some questions due to item-specific missing data. Overall, these results show that the majority of children report enjoying reading activities. At the same time, there is still an important fraction of students (between 30 and 35 percent) that report either indifference or a dislike for this type of activity. Clearly, there is still room for children to positively change their attitudes towards reading – an outcome MA aims to affect. Students were also surveyed on their reading habits and self-assessed ability to read. Table 4 shows results for these questions. Roughly half of the students report that they know how to read. Perhaps surprisingly, it is not the case that 1 st graders cannot read and 2 nd graders can. In fact, knowing how to read and grade-level are only weakly correlated: 40 percent of 1st graders reported they know how to read, while the figure for 2nd graders is 55 percent. Table 4. Students’ reading habits Do you know how to read? 49% Do you read or try to read on your own at home? 70% Number of days read at home last week 2.4 Does anyone read or help you try to read with you at home? 78% Number of days read with someone at home last week 2.3 Do you read or try to read in class alone? 66% Number of days read in class alone last week 1.8 Do you read or try to read in class out loud? 61% Number of days read in class out loud last week 1.7 Number of days attended school in last week 4.4 N 2,249 24 Source: MA Baseline data. Sample sizes for some variables are lower due to item￾specific missing data. Regarding students’ self-reported reading habits, the results are similar to what was found for self-reported reading preferences. Seventy percent of students said they read or try to read at home. On average, children report reading at home 2.4 days a week. Students were also asked if someone helps them to read at home, and 78 percent report receiving such help. On average, 66 percent of students report reading or trying to read in class alone last week, and 61 percent report reading out loud during class. Finally, the number of days that students attended school on average was 4.4 days a week, indicating an absenteeism rate for this sample of 12 percent. These figures suggest that sampled households provide suitable conditions for an intervention like MA to yield expected results. First, the majority of students report that they receive help from someone at home to read or try to read, which is important because for MA to work children need assistance from their parents to learn how the stories can be read, and overall support so children can spend time with the cell phones reading the stories. Second, students seem to have some exposure and familiarity with reading activities, so MA activities will not be completely foreign for them. It is worth highlighting that there seem to be some discrepancies between caregivers’ and children’s answers regarding children’s reading habits. For example, 70 percent of children report reading or trying to read at home, which is higher than what could be inferred from the data reported by the caregiver. According to caregivers, as many as 44 percent of children never practice reading at home on their own. To analyze this more systematically, Table 5 and Table 6 show crossed tabulations on reading habits at home as reported by the child and her caregiver. Specifically, Table 5 shows how students and their caregivers reported whether the student in question reads at home alone. The figures show the number of households in each cell and below, in parentheses, the corresponding row percentages. Among the students whose parents say the child never reads at home on her own, 36.7 percent of the children say they do not read at home on their own, but 63.3 percent say they do. Among the students whose parents say their children read at home on their own, 25.4 of the children say they do not read at home and 74.7 percent say they do it. While there is a strong correlation between caregivers’ and children’s responses, the two ‘sources’ on reading habits do not coincide perfectly. 25 Table 5. Child reads on their own at home According to child No Yes Total According to Caregiver No 344 594 938 (36.7) (63.3) (100.0) Yes 305 898 1,203 (25.4) (74.7) (100.0) Source: MA Baseline data. Table 6 display results for reading at home with someone. While the pattern is relatively similar to what was observed for children reading on their own, the correlation between caregivers’ and children’s responses is less strong. Among the students whose parents say the child never reads at home with someone, only 25.9 percent of the children say they do not read at home with someone, and 74.1 percent say they do. Among students whose parents say their children read at home with someone, 20.6 percent of the children say they do not read at home with someone, and 79.4 percent say they do. It is possible that, for the caregiver, it is hard to track every time their children read with a household member, which could explain the low correlation that is observed between caregivers’ and children’s responses for this activity. Table 6. Child reads with someone at home According to child No Yes Total According to Caregiver No 92 263 355 (25.9) (74.1) (100.0) Yes 385 1,481 1,866 (20.6) (79.4) (100.0) Source: MA Baseline data. D5. EGRA SCORES Table 7 presents EGRA results as average scores and fractions of students with zero correct answers, by reading skill. In the first row the results for “Orientation to Print” are displayed. In this subtask students are asked three questions on the mechanics of how to read a passage. On average students answered 1.4 of these questions correctly. However, based on feedback from the enumerators who were administering the EGRA, there is reason to believe that the dialect of ChiNyanja used to explain instructions for this subtask was not necessarily appropriate, rendering lower than expected scores, especially because the language used to describe instructions for where to point one's finger was in “deep ChiNyanja” (see Annex I E3 for more information). The rest of the instrument is divided in five sections. The first of these sections is called Letter sound identification, where students are asked to identify a list of letters. In total, students are 26 supposed to identify 100 letters. The next section is called Non-word reading, where students need to read a list of 50 made-up words. Next, for the Oral reading subtask, students are asked to read a short passage out loud that has 41 words. Students are also asked 5 comprehension questions on this passage. It is important to highlight that the number of questions each child is asked varied depending on how much of the text they were able to read. Students that are not able to read anything are automatically assigned a zero in the reading comprehension skill. Finally, for the listening comprehension section, students are asked five comprehension questions about a text the interviewer reads for them. The main results can be summarized as follows:  Thirty-nine percent of students did not correctly identify any letter.  Seventy-five percent of students read zero ‘made-up’ words correctly.  Sixty-eight percent of students read no ‘real’ words correctly,  The average score in the reading comprehension question was 0.3 out of 5 possible questions  The average score in the listening comprehension question was 2.8 out of 5 possible questions Table 7. EGRA scores and percent with zero correct answers Task Measure Score/ % Zero correct Orientation to print Number of correct answers (0 to 3) 1.4 Letter sound identification Percentage with zero letters identified 39% Number of letters identified (0 to 100) 7.7 Non-word reading Percentage with zero words read 75% Number of correct words read (0 to 50) 2.7 Oral reading passage Percentage with zero words read 68% Number of correct words read (0 to 41) 4.2 Reading comprehension Number of correct answers (0 to 5) 0.3 Listening comprehension Number of correct answers (0 to 5) 2.8 N 2,264 Source: MA Baseline data. Sample sizes for some variables are lower due to item￾specific missing data. The EGRA scores indicate that students in these communities do not perform well in the evaluated reading skills. More than two thirds of the surveyed children could not read any word, and 39 percent could not identify the sound accompanying a given letter. This indicates that there is a lot of room for improvement, and in particular for a program like MA to have an impact on reading skills; however, it also suggests that this low skill level in the absence of treatment may act as an important restriction for access to reading materials to have any impact. 27 In order to examine correlations between sociodemographic characteristics and EGRA scores, we estimated regressions for the six subtasks described in Table 7, as a function of key demographics, including parental education and household assets. Table 8 presents results for students in 2nd grade (in 2016). Overall, we can see that not many sociodemographic characteristics are correlated with EGRA scores. Whether the student is female is negatively correlated with 4 of the 6 analyzed scores, but the coefficient is pretty small (smaller than the standard error in most cases) and never significant. Student age and household size are positively correlated with all six scores, but are never significant. Having a mother that knows how to read is positively correlated with EGRA scores in 5 of 6 cases, but is significant only for the “Orientation to Print” score. The parameter indicates that having a mother that knows how to read would increase the Orientation to print score by 0.17 points (note that the maximum score for this subtask is 3). Whether the father knows how to read is positively correlated with 5 of 6 scores, but is only significant for Oral Reading Passage. The coefficient implies that having a father that knows how to read is associated with an increase of one point (i.e. one word) in the “Oral Reading Passage.” In terms of assets, there does not seem to be an obvious correlation with any of the included assets, except having a bike, which is negatively correlated with scores, and is significant in four of six cases. The rest of the assets have positive coefficients in all cases, but are only statistically significant for having a TV and a radio, and only for the “Listening Comprehension” score. However, F-tests for joint significance show that owning these assets (TV, bed, bike and radio) have a significant effect for all subtasks except for “Orientation to print” and “Non-word reading.” Finally, being in the treatment group is positively correlated with all 6 scores, but it is not significant. Table 8. EGRA scores regressions – 2 nd grade Orientation to print Letter sound identification Non-word reading Oral reading passage Reading comprehension Listening comprehension Student is female -0.056 0.211 -0.07 -0.018 0.017 -0.089 (0.062) (0.514) (0.231) (0.353) (0.029) (0.068) Student age 0.046 0.246 0.093 0.137 0.017 0.036 (0.024) (0.163) (0.098) (0.152) (0.012) (0.025) Household size 0.018 0.185 0.081 0.054 0.000 0.03 (0.015) (0.129) (0.061) (0.086) (0.008) (0.016) Mother can read 0.172* -0.041 0.012 0.02 0.04 0.006 (0.076) (0.541) (0.240) (0.365) (0.034) (0.085) Father can read -0.105 0.687 0.489 0.999* 0.038 0.037 (0.090) (0.567) (0.280) (0.395) (0.029) (0.095) HH has TV 0.01 1.043 0.624 0.749 0.053 0.268** (0.099) (0.760) (0.456) (0.618) (0.057) (0.087) HH has bed 0.093 0.22 0.22 0.325 0.041 0.02 (0.080) (0.518) (0.232) (0.348) (0.039) (0.074) HH has bike -0.156* -1.933** -0.459 -0.895* -0.068 -0.221* (0.077) (0.671) (0.305) (0.383) (0.036) (0.085) HH has radio 0.093 0.557 0.138 0.566 0.049 0.205* (0.080) (0.608) (0.313) (0.427) (0.039) (0.093) 28 HH in treatment sample 0.156 1.161 0.407 0.522 0.051 0.083 (0.089) (0.934) (0.391) (0.586) (0.049) (0.122) N 1181 1179 1180 1180 1181 1181 Source: MA Baseline data. Standard errors clustered at the school level. * p<0.05 ** p<0.01 *** p<0.001 Finally, Table 9 presents results for 3rd graders. The estimates are relatively similar to what is found for 2nd grade. Being a female is negatively correlated with five of six of the evaluated subtasks, and is significant for “Orientation to print” and “Listening Comprehension.” Students’ age is positively correlated with five of six scores, but none of the coefficients is significant. The same is observed for household size. Whether the mother knows how to read is positively correlated with all six scores, and is significant for five of them. Whether the father can read is positively correlated with five of six scores, but in no case is the correlation significant. Having a television and having a bed is positively correlated with all scores, but the coefficients are not statistically significant. As was observed for 2nd graders, having a bike is negatively correlated with all scores, but in this case no coefficient is significant. No coefficient for having a radio is significant, and there is not a pattern in the coefficients’ signs. In effect, no asset is significant in any case; however, F-tests for joint significance show that these assets have a significant effect for all subtasks except “Oral reading passage.” Being in the treatment group is positively correlated with all scores, but the results are only significant for “Orientation to print.” Table 9. EGRA scores regressions – 3 rd grade Orientation to print Letter sound identification Non-word reading Oral reading passage Reading comprehension Listening comprehension Student is female -0.187* -0.470 -0.328 -0.448 0.021 -0.180** (0.072) (0.664) (0.408) (0.657) (0.057) (0.064) Student age 0.045 0.159 0.019 0.083 -0.002 0.018 (0.023) (0.217) (0.155) (0.236) (0.023) (0.023) Household size 0.020 -0.083 0.113 0.140 0.020 0.000 (0.014) (0.176) (0.133) (0.201) (0.017) (0.017) Mother can read 0.016 1.885** 1.465** 2.068** 0.226*** 0.152* (0.083) (0.636) (0.495) (0.745) (0.061) (0.071) Father can read 0.121 0.657 0.895 1.368 0.054 -0.078 (0.118) (0.901) (0.606) (0.915) (0.083) (0.094) HH has TV 0.056 1.388 0.762 1.444 0.136 0.116 (0.089) (0.726) (0.579) (0.884) (0.085) (0.075) HH has bed 0.045 1.153 0.136 0.495 0.057 0.123 (0.077) (0.727) (0.457) (0.690) (0.065) (0.070) HH has bike -0.026 -1.009 -0.903 -1.330 -0.108 -0.102 (0.078) (0.718) (0.562) (0.863) (0.088) (0.105) HH has radio -0.017 -0.530 0.151 0.089 -0.061 0.061 29 (0.082) (0.680) (0.485) (0.720) (0.069) (0.073) HH in treatment sample 0.286* 1.769 1.356 1.988 0.147 0.108 (0.118) (1.149) (0.792) (1.195) (0.106) (0.102) 1082 1081 1081 1081 1082 1082 Source: MA Baseline data. Standard errors clustered at the school level. * p<0.05 ** p<0.01 *** p<0.001 E. BALANCE In this section we review to what extent households and other characteristics are balanced between treatment and control groups. Checking balance is important because it is a straightforward way to check that treatment and control groups are observationally equivalent, so any differences observed at baseline can be attributed to the program. Table 10 shows households characteristics by treatment status. In general, it appears households in treatment schools observe better sociodemographic characteristics than households in the control group. In effect, parents of sampled students have more years of education in the treatment group than in the control group, and asset ownership is more prevalent in the treatment than in the control group. However, most of these differences are not statistically significant at standard levels of confidence; in fact only one indicator – owning a bed – is significant at the 10 percent level. Table 10. Household characteristics by treatment status Control Treatment p-value Household members' characteristics Respondent is male 53.3% 53.4% 0.978 Mother knows how to read 54% 59% 0.126 Father knows how to read 79% 80% 0.713 Average years of education (mother) 5.5 5.6 0.678 Average years of education (father) 6.9 7.1 0.549 Average age (mother) 34.4 34.1 0.524 Average age (father) 40.5 40.9 0.462 Household size 5.4 5.2 0.309 Household assets Avg number of plots 2.2 2.3 0.553 HH owns television 25.5% 30.9% 0.134 HH owns bed 50.8% 57.0% 0.079 HH owns bicycle 75.1% 74.0% 0.663 HH has electricity 5.9% 5.3% 0.865 HH owns radio 60.5% 61.6% 0.660 N 1,121 1,100 30 Source: MA Baseline data. Standard errors clustered at the school level. * p<0.05 ** p<0.01 *** p<0.001 Similar results are observed for availability of reading resources. Table 11 shows data on reading resources by treatment status. It can be seen that although for some indicators the fractions are higher for the treatment than for the control group, the differences are not statistically significant. Table 11. Available reading resources by treatment status Control Treatment p-value Child has reading materials 35% 37% 0.47 Afterschool reading activities are available 12% 12% 0.81 -Child participate in afterschool reading activities 60% 63% 0.68 N 1,208 1,182 Source: MA Baseline data. Standard errors clustered at the school level. On the other hand, the imbalance seems more apparent when we look at EGRA results. Table 12 shows results for EGRA scores by treatment status. Students in the treatment school outperform control across all five reading skills. Although the difference is significant at the five percent level only for “Orientation to Print,” differences in “Letter Sound Identification,” “Non￾Word Reading,” and “Oral Reading Passage” are significant at the 10 percent level. This indicates that, although treatment was assigned randomly, by chance this particular ‘sample draw’ seems to be relatively unbalanced. Table 12. EGRA scores and percent with zero correct answers by treatment status Task Measure Control Treatment p-value Orientation to print Number of correct answers (0 to 3) 1.3 1.6 0.011* Letter sound identification Percentage with zero letters identified 42% 37% 0.217 Number of letters identified (0 to 100) 6.9 8.6 0.082 Non-word reading Percentage with zero words read 79% 72% 0.064 Number of correct words read (0 to 50) 2.2 3.2 0.076 Oral reading passage Percentage with zero words read 71% 66% 0.165 Number of correct words read (0 to 41) 3.5 4.9 0.082 Reading comprehension Number of correct answers (0 to 5) 0.3 0.4 0.107 Listening comprehension Number of correct answers (0 to 5) 2.8 2.9 0.250 N 1,133 1,131 Source: MA Baseline data. Standard errors clustered at the school level. * p<0.05 ** p<0.01 *** p<0.001 The fact that baseline data could not be collected before treatment assignment prevented randomization from being conducted between similar students or similar schools. The pre￾randomization matching that was conducted was based only on school size and distance to the DEBS. Unfortunately, the particular draw that was used to assign treatment and control schools ultimately produced a relatively unbalanced sample. The main risk of having an unbalanced panel at baseline is that differences at endline may not be attributable to the program impact. 31 However, because we have baseline data and a rich set of household covariates, causal parameters will still be possible to estimate, using Value-Added models in particular. F. FINAL COMMENTS In this report we presented the main results from the baseline data for the MA project. We reviewed the research questions, described the data collection process and presented summary statistics by treatment group for key sociodemographic indicators and outcomes of interest. A few important conclusions are worth highlighting:  First, cell phone ownership is not universal but it is relatively high in these communities. We found that at least 57 percent of households have a working cell phone. This percentage could be higher because we did not observe cell phone ownership for the 27 percent of students that were absent the day AT visited schools. This demonstrates that cell phones can constitute a powerful channel to distribute reading materials, which is required for MA to be scaled up.  Second, out-of-school reading resources are quite limited in these communities. We surveyed whether students had a reader’s book or other school reading materials, children’s books in ChiNyanja at home, and if there were reading activities in the community. We found that 58 percent of students do not have access to any of these resources, which highlights the importance of providing reading materials and a channel for students to practice reading outside of school.  Third, students’ reading skills are relatively low. Forty percent of the sample could not correctly pronounce a single letter, and two thirds could not read a single word. This shows that it may be particularly important and useful to provide reading resources in these communities, but it also suggests that children’s backgrounds and/or education quality may be major restrictions for MA to have the expected impact.  Finally, we also found that, despite having been randomly assigned, the treatment group tends to have better average values for indicators than the control group. This was observed in the analysis of the sociodemographic characteristics but more critically in the EGRA scores. Although these results suggest that the sample may be slightly unbalanced across some key variables, collecting longitudinal data will still allow us to properly approximate the estimation of causal parameters. 32 ANNEX I. DATA COLLECTION We partnered with INESOR at the University of Zambia, a Zambian data collection firm, to collect the baseline data. The baseline data collection effort was conducted in three different stages. First, “Advance Teams” (AT) were sent to the field to construct the student sample frame and select sample. Second, enumerators were sent to households of the selected students to conduct the caregiver survey. Third, enumerators visited each school to run the EGRA and the student survey. Below we describe how each of these data collection efforts was conducted. I1. ADVANCE TEAM ACTIVITIES Training Enumerators attended a day long training administered by NORC staff. NORC administered standard exercises with enumerators to familiarize them with a simple data collection template and procedures they will be using in the field. Throughout the training, these exercises were scored to identify where additional training was needed, as well as identify the highest performing candidates for the AT. Fieldwork and quality control Prior to visiting each school, INESOR scheduled visits with the head teacher of each school. Four teams composed of three enumerators and a supervisor were deployed daily to a cluster of four schools. Each AT member was asked to cover one school each day, so the AT data collection lasted one week (five days). Each district had one to two staff from INESOR and NORC to oversee the quality of data collection in the field. Data quality review The final dataset from each school was reviewed by NORC to ensure its accuracy. Given that the objective of the AT data collection was to properly select the sample of eligible students (and the list of replacements), the review focused on checking that the right number of students was selected, and that the selection of these students was properly conducted; in particular, that selection was random. I2. PARENT/CAREGIVER HOUSEHOLD SURVEY Training Training of enumerators for the household survey occurred over the course of a week. The training was designed to familiarize enumerators with the study and the intervention, the questionnaire, informed consent, survey best practices, interviewing techniques, sampling and replacement procedures, and how to troubleshoot issues that arise in the field. The training was led by NORC staff, with support from INESOR. The training placed a strong emphasis on participatory and demonstration activities during training that honed the skills of 33 participants and enabled them to put theory into practice. To this end, the training included activities such as: role-play exercises, inter-rater reliability (IRR) exercises, and a one day pilot and debrief, where enumerators will administer the survey to at least two or three respondents. Quality control For the training, NORC created a data quality control checklist used by trainers and supervisors to assess the level of skills of enumerators. A larger number of enumerators than necessary were invited to training, such that only the best were selected for the actual field period. This selection was based on an assessment during the classroom training using the IRR tests, as well as during the pilot test, classroom and pilot observations, and a training quiz developed by NORC. Supervisors were selected and finalized towards the end of the training. Supervisors were the primary liaison between INESOR/NORC, schools, and each enumerator team. They were responsible for coordinating field logistics on a daily basis as necessary, overseeing sampling/replacement lists, implementing quality control procedures, and liaising with INESOR/NORC when issues arose. They were not expected to conduct parent interviews, although some did during the course of the field period. Supervisors were provided a separate half day training to outline their roles and responsibilities, the field quality procedures they were responsible for (including validation/back-checks, spot-checks, and observations), and what to do in case of unexpected circumstances. Supervisors were required to review all questionnaires at the end of each day. In addition, they were required to administer a validation/back-check survey to 10% of responses. NORC constructed a series of 7-10 questions to verify that the interview took place and that the enumerator selected responses appropriately in the tablet. Supervisors conducted these back￾checks by either visiting the household later, or calling the respondent on the cell phone provided by the respondent. When discrepancies arose, supervisors worked with enumerators to review responses and come to a resolution on the issue. Supervisors were also asked to observe 10% of interviews using an observation checklist to ensure they are being carried out appropriately, and to provide feedback to enumerators on what went well in interviews and how they could improve. During the first few days of data collection, NORC staff were in the field to observe data collection, meet with data collection teams to address major issues that arise, and make modifications to field procedures as necessary. INESOR staff were in the field during the entire field period in order to observe interviews, ensure that all protocols are adhered to, and provide support to teams as necessary. At the end of each day, data was uploaded to the secure NORC SODA server. NORC reviewed the data on an ongoing basis to gauge whether or not interviews were happening in an appropriate amount of time, appropriate values were entered for open ended questions, cell phone numbers were the appropriate length, and other potential areas to verify that interviews were carried out appropriately. Field Work Summary and Challenges 34 The bulk of data collection for the caregiver survey took place in the last three weeks in November. Where notes were returned with phone numbers for sample parents, teams scheduled an appointment with parents to conduct the interview; where notes were not returned, teams called schools the day before their visit and asked head teachers to relay a message that the data collection team would be visiting the school on a particular day. For any parent that could not be contacted in advance to arrange an interview, teams travelled to parents homes to interview them. One challenge facing teams within the first few days of data collection was that some schools had no notes returned from parents. In those cases, teams had to directly visit all parents at their homes. When notes were returned, enumerators reported that in certain schools, cell phone reception was poor or phones were shut off. Another challenge faced throughout the field period was the inability to get in touch with certain head teachers before the day of visit; while protocol required teams to contact schools one day in advance to let them know the enumerator teams were visiting and to understand if there were any major events that were happening that would preclude parents from participating in an interview that day (e.g. a village funeral), some schools were unresponsive. Therefore, upon arrival at the school, several schools had to be revisited due to funerals occurring in that school community. Supervisors and enumerators reported that the initial reaction of parents was generally enthusiastic, with parents eager to know when the intervention would start if their school was selected. In two schools, supervisors reported that several parents lied about having a cell phone; but subsequent screener questions revealed that parents did not have a cell phone and these cases were replaced. There were also several schools where parents were not selected for an interview but requested to be interviewed by enumerators. Supervisors reported two schools where teachers received teams negatively. In one school, the head teacher was not happy that their own child was not selected for the survey. In another school, teachers tried to interfere with the team’s activities to include their own students in the survey. In six schools, supervisors reported that several respondents were afraid data collection teams were associated with Satanic practices. Most notably, in one school, the majority of parents wrote down fake phone numbers on the advanced team notes. Despite these cases, the supervisors and enumerators worked to socialize the project with parents. As a result, the replacement rates in these schools are not high, with an average of one refusal per school. Upon the final completion check for the caregiver survey, it appeared that not all of the required interviews were completed at nine schools; five of these schools had been assigned to treatment and four to control. Because Creative was planning on meeting with parents in December to introduce the MA intervention, in the interest of not delaying implementation of MA, INESOR revisited five of these schools in December 2015 to complete the interviews before Creative entered these communities. INESOR revisited the four remaining communities, which were control schools, in January 2016 during the EGRA data collection period. I3. EGRA AND STUDENT SURVEY Changes to EGRA/Tangerine from 2014 Questionnaire 35 The following changes were made to the RTS EGRA instrument for the MA impact evaluation:  Wording of consent was changed in order to change the focus of the student questionnaire, to remove reference to numeracy exercises, and update the language regarding stopwatches in favor of “phone” which is reference to the tablet that is more easily understood by pupils in the rural areas;  The order of the subtasks followed the RTS EGRA assessment. While the National EGRA placed the orientation to print subtask in fourth place, it was determined that following the RTS order with orientation to print leading the assessment it would allow for a more accurate assessment of students’ knowledge, as all subsequent subtasks would potentially provide the pupil with answers to the orientation to print subtask;  The RTS EGRA assessment used the Oral Reading Fluency (ORF) subtask text. In order to not show the ORF text to pupils prior to ORF subtask a text from the G2 Term 2 ChiNyanja text book was used;  In the non-word reading sub-task the word “annuli” was changed to “anuli” in order to match ChiNyanja orthographic rules. Additionally, the national EGRA non-word reading subtask had “anuli” and it is believed the “nn” is a typo resulting from Microsoft Word autocorrect as this was experienced by STS staff; and  In the Oral reading fluency sub-tasks the word “kumsika” in the first line of the passage was changed to “ku msika” following feedback from teachers present at the training. Due to the unavailability of a ChiNyanja linguist it was not possible to further explore the issue. Training NORC’s subcontractor, School-to-School International (STS), programmed all tools into Tangerine. NORC/STS carried out the training of enumerators and supervisors in the administration of EGRA in collaboration with INESOR. Full enumerator training lasted six days, followed by a pilot test, debriefing, and an additional day of practice for enumerators that needed it. During the training, assessors were trained in EGRA and the accompanying student questionnaire. For the training of EGRA subtasks, the various subtasks goals and rules were presented in English to the group and then practice was conducted in the local language (ChiNyanja). Practice of EGRA and the questionnaire continued throughout the week including demonstrations by pairs with review following each subtask. Three IRR exercises were conducted during the training to determine the participant’s inter-rater reliability. For the first two IRR exercises the mode response was used to calculate scores. For the third IRR, a script was use to ensure enumerators were not just marking similarly but marking correctly. A quiz and review of the quiz took place on day three of the training. During training the following marking conventions for EGRA were adopted:  In the non-word reading and the oral reading fluency sub-tasks assessors were prompted to consider syllabicated words as wrong; 36  “L” and “R” letter substitution was marked as correct in non-word reading and oral reading fluency sub-task. After discussions with Dr. Beatrice Matafwali and ChiNyanja language teachers who participated in the enumerator training it was determined that in ChiNyanja the “R” is often substituted for the “L” and while it is not a word in ChiNyanja it does not affect comprehension;  In the oral-reading comprehension and listening comprehension sub-task assessors were prompted to consider as wrong, answers given in a language different than ChiNyanja if the pupil did not self-correct after being prompted to answer in ChiNyanja; and  Acceptable answers for Comprehension questions – enumerators were prompted to consider as correct answers provided between brackets as well as answers which were to be considered “similar” to the ones in brackets. Multiple examples of answers to be considered correct were provided during training as well as during feedback sessions. Quality control For the training, NORC and STS created a data quality control checklist used by trainers and supervisors to assess the level of skills of enumerators. A larger number of enumerators than necessary were invited to training, such that only the best were selected for the actual field period. This selection was based on an assessment during the classroom training using the IRR tests and during the pilot test, as well as their score on the training exit quiz developed by NORC and STS. In the field, supervisors remained in close contact with all their enumerators and conducted spot-checks and observations, ensuring that EGRA and other survey instruments were administered correctly. Furthermore, a field coordinator for each region was in the field during the entire field period in order to conduct similar spot-checks and observations and ensure that all protocols are adhered to. During the first week of data collection, STS staff were in the field (one in each district) to observe the quality of data collection and provide feedback to supervisors and teams based on their observations. STS staff also reviewed the data on an ongoing basis to make sure that procedures outlined in the training for administering EGRA were followed. Field Work Summary and Challenges Enumerators were required to locate a specific list of students at the school, and if necessary, attempt to find the student at home to interview the student. Enumerators were carefully trained how to locate a quiet, comfortable setting in which to administer EGRA at both the school and at home. Of the 2260 interviews conducted, approximately five percent of interviews were conducted at home. One of the major challenges of this data collection effort was navigating the rainy season. While the 2015-2016 rainy season was delayed, teams were unable to visit one control school because rains made the roads impassable. Given the delayed onset of the rainy season and the location of the school in a region of Lundazi that is currently receiving heavy rains, this school will not be able to be accessed until May 2016. 37 During school practice and the initial days of data collection STS staff noted the following observations regarding the EGRA tool:  Orientation to Print Subtask- The second question posed in the orientation to print subtask did not seem to be clear. Even pupils who could answer the first and third question did not seem to successfully answer the second question;  ChiNyanja instruction to point your finger – The instruction to pupils to point their finger to where they will start the subtasks was often not clear. Enumerators indicated that the instruction was in a more formal ChiNyanja that may be difficult for more rural populations to understand and that is not necessarily used colloquially. Enumerators were instructed to read instructions as is and not to deviate;  Listening Comprehension Subtask – Many students understood the story’s character Mangani was going to school on a Monday. However, when asked what day it was when Mangani was going to school they could only respond in English. When asked if they could say it in Chinanja, students often remained silent resulting in incorrect answers; and  Student Questionnaire –Students seemed to have difficulties in understanding the meaning of the “smiley faces” and providing an answer by pointing to them. Using the script in the survey instruments, enumerators where instructed to stop and re-explain the meaning of the smiley faces as necessary. Future data collection efforts, especially with the potential development of the 2016 Zambia EGRA to be used across several grantees and organizations in Zambia, should keep these issues in mind. 38 ANNEX II. INSTRUMENTS II1. ADVANCE TEAM EXCEL INSTRUMENT “Let me tell you why I am here today. I work with the University of Zambia and we are trying to understand whether your parent has a cell phone. We are trying to understand how children like you learn to read, and may want to call your parent to ask them some questions. We would like your help in this, but you do not have to answer my questions if you do not want to. Do you want to continue?” STUDENT FIRST NAME STUDENT LAST NAME ABSENT Refuse? CELL PHONE? PARENT NAME ask? Type in first name from roster Type in last name from roster If the student is not present today, mark X. Otherwise, leave blank. Read out consent. If child says no, select X Ask student: "Does someone in your house have a cell phone?" If response is yes, select YES If response is no, select NO If response is do not know, select DO NOT KNOW Ask student "What is your parent's name?" if student has cell phone. Write down SAMPLE REPLACEMENT PARENT NAME Roster HOME ADDRESS/ DIRECTIONS PHONE NUMBER LISTED NOTE RETURNED? Step 1: Filter only students that answered "YES" for "CELL PHONE?" Step 2: Calculate the sampling interval using Step 1: Filter only students that answered "YES" for "CELL PHONE?" Step 2: Filter OUT students that have an X under "SAMPLE" Check roster. If available, type. Check roster. If available, type. Check roster. If available, type. SUPERVISOR ONLY: FILL OUT UPON SECOND VISIT TO SCHOOL 39 the Sampling Interval Calculator tab Step 3: Determine your starting point by finding the name of the kid that is in the position of the starting point Step 4: Select 15 students. Step 5: If a student is selected, type X, otherwise leave blank Step 2: Calculate the sampling interval Step 3: Determine your starting point by finding the school's name on the first tab Step 4: Select 15 students. Step 5: If a student is selected, type X, otherwise leave blank ENUMERATOR RESERVE CODES: -9=REFUSED; -8=DON’T KNOW; -7=NOT APPLICABLE Page 40 II2. PARENT/CAREGIVER HOUSEHOLD SURVEY INSTRUMENT BASELINE QUESTIONNAIRE - Makhalidwe Athu Project COVER SHEET INFORMATION Enter enumerator name [enu_name]: _______________________________________ Enumerator no [enu_num]: |__|__| Date [enu_date]: DD: |__|__| MM: |__|__|YYYY: |__|__|__|__| School ID [school_code]: |__|__|__|__|__|__| Questionnaire No [quex_id]: |__|__|__|__|__|__| GPS location of household [gps]: LATITUDE(N/S) |__| – DEGREES:|__|__|__| MINUTES: |__|__| SECONDS |__|__|.|__|__|__| LONGITUDE (E/W) |__| – DEGREES: |__|__|__| MINUTES: |__|__| SECONDS |__|__|.|__|__|__| SECTION A: INTRODUCTION AND CONSENT [section_a_intro] Good day. My name is [ENUMERATOR NAME] and I represent INESOR, a research organization with the University of Zambia. 1A. Is this the home of/are you the parent or guardian of READ OUT NAME OF CHILD FROM FIELD CONTROL SHEET? [child_home] 01 Yes 02 No  [ASK FOR DIRECTIONS TO HOUSE, AND END INTERVIEW] |__|__| 1B NOTE TO ENUMERATOR: ASK THIS QUESTION ONLY IF YOUR SAMPLE SHEET INDICATES THERE ARE SIBLINGS. Is this also the home of/are you the parent or guardian of READ OUT NAME OF CHILD FROM FIELD CONTROL SHEET? [child_home_sib] 01 Yes 02 No |__|__| 1C ENTER NAME OF CHILD 1 [child_id]: ___________________________________ 1D ENTER PUPIL ID OF CHILD 1 [student_id]: |__|__|__|__|__|__|__|__| 1E* ENTER NAME OF CHILD 2 [child_id_sib]: [SKIP IF NO SIBLING] ___________________________________ 1F* ENTER PUPIL ID OF CHILD 2 [student_id_sib]: [SKIP IF NO SIBLING] |__|__|__|__|__|__|__|__| 2. May I please talk to the parent or guardian of [CHILD’S NAME]’s? [ask_parent] We are working with NORC at the University of Chicago – a research organization. We are conducting a survey in this area to better understand how we can use cell phones to improve children’s reading practices in Zambia, in the context of a study approved by the Ministry of General Education. We would like to ask you a few questions about this. The survey will take approximately 30 minutes and we will ask you questions about your household’s basic characteristics, including whether you have a working cell phone to participate in the study, and your child’s reading practices. You and your child were randomly selected to participate in this study from a pool of pupils at his/her school who indicated their parent may have a cell phone. This survey will be repeated in January 2017. You may also receive a short follow-up survey in April 2016. Whatever information you will provide will be strictly confidential and not be shown to any other persons. Participation in the survey is completely voluntary. If we should come to any question that you do not want to answer, just let me know and I will go on to the next question. You are also free to stop the interview at any time or withdraw altogether. [introduction] ENUMERATOR RESERVE CODES: -9=REFUSED; -8=DON’T KNOW; -7=NOT APPLICABLE Page 41 Should have any queries about the survey when I am gone you can contact the following: [HAND CONTACT INFORMATION TO RESPONDENT] 3. Do you wish to participate in this survey? May we start now? [consent] IF YES, CHECK HERE IF RESPONDENT CONSENTS TO INTERVIEW |__|. ASK RESPONDENT TO SIGN CONSENT FORM. IF NO, END INTERVIEW. 4. What is your name? [name_resp] ____________________________________________ 5. Do you or someone in your household have a working cellphone that can be used three times a week to receive free SMS messages? [working_phone] 01 Yes 02 No  [END INTERVIEW, THANK PARTICIPANT FOR THEIR TIME] |__|__| 6. If the call is free, can you call out on your phone? [call_out] 01 Yes 02 No  [END INTERVIEW, THANK PARTICIPANT FOR THEIR TIME] |__|__| 7. Can you receive calls on your phone? [rec_calls] 01 Yes 02 No  [END INTERVIEW, THANK PARTICIPANT FOR THEIR TIME] |__|__| 8. Do you have access to any means of charging your phone on a regular basis, at home or elsewhere? [charge_phone] 01 Yes 02 No  [END INTERVIEW, THANK PARTICIPANT FOR THEIR TIME] |__|__| Thank you. Now we would like to know if you would be interested in participating in Makhalidwe Athu, a program aimed at improving the reading skills of pupils in the Chipata and Lundazi districts. [CHILD’S NAME]’s school may be selected to participate in this program. If [CHILD’S NAME]’s school is selected, you can choose to participate in the program. In this program, participants will receive short stories on their mobile phones for their children to read. The stories will be sent using text messages. These messages will be free; no fees will be charged to you. You will also receive phone calls with voice recordings of the stories and questions about the stories to discuss with [CHILD’S NAME]. These phone calls will be automatically triggered by you calling a number and hanging up after ringing once or twice, so you are not charged. In addition, every month there will be a meeting in your community to answer to any questions you may have about the program, address problems with the SMS messages and get your feedback about the program. Participants will be expected to spend 30 minutes, 3 times per week, every week for a year, listening to their child read these text messages, discussing and answering questions about them, as well as attending the monthly meetings. 9. If [CHILD’S NAME]’s school is selected, would you be interested in participating in the program? [consent] 01 Yes 02 No  [END INTERVIEW, THANK PARTICIPANT FOR THEIR TIME] |__|__| 10. Thank you. To be able to send these messages, we will need your phone number. Now, think about the phone you use most often. If this phone uses more than one line, think about the line that is used the most. What is the number of this line? [phone_num] |__||__||__||__||__||__||__||__||__||__| Refuse  [END INTERVIEW, THANK PARTICIPANT FOR THEIR TIME] Don’t know  ENUMERATOR RESERVE CODES: -9=REFUSED; -8=DON’T KNOW; -7=NOT APPLICABLE Page 42 [CHECK IF PHONE NUMBER OF PARTICIPANT IS AVAILABLE IN DIGITIZED PUPIL LIST, IF YES CONTINUE. IF NOT ASK IF PARTICIPANT CAN CALL ENUMERATOR’S PHONE TO DISPLAY THEIR NUMBER. IF THERE IS NO PHONE NUMEBR AVAILABLE AND PARTICIPANT CANNOT CALL ENUMERATOR, END INTERVIEW AND THANK PARTICIPANT FOR THEIR TIME] We will also visit [CHILD’S NAME]’s school and ask [CHILD’S NAME] to take a short, 20 minute reading assessment and questionnaire on reading habits at home. Prior to asking [CHILD’S NAME] to participate, we will explain the purpose of the questionnaire and only proceed if s/he wants to participate. We will also not interview [CHILD’S NAME] if you do not want us to. The results of this assessment will have no effect on [CHILD’S NAME]’s grades at school. 11*. Do you authorize [CHILD’S NAME] to participate in the assessment and answer the questionnaire? [assessment_auth] 01 Yes 02 No  [END INTERVIEW, THANK PARTICIPANT FOR THEIR TIME] |__|__| ENUMERATOR RESERVE CODES: -9=REFUSED; -8=DON’T KNOW; -7=NOT APPLICABLE Page 43 SECTION B: HOME LITERACY ENVIRONMENT 12A* What is the gender of [NAME OF CHILD]? [child_age] 01 MALE 02 FEMALE |__|__| 12B* How old is [NAME OF CHILD]? [child_gender] [IN YEARS] |__|__| YEARS 12C* What is the grade of [NAME OF CHILD]? [child_grade] |__|__| GRADE ENUMERATOR RESERVE CODES: -9=REFUSED; -8=DON’T KNOW; -7=NOT APPLICABLE Page 44 First, I would like to ask you some basic questions about your household and reading practices with your children. You are a household member if: (i) You have lived under this "roof" or within the same compound/homestead/stand at least 15 days during the last 12 months OR you arrived here in the last 15 days and this is now your usual residence; (ii) when you are together you share food from a common source with other household members; and (iii) you contribute to or share in a common resource pool. I am going to ask you some questions about the members of this household. ENUMERATORS: IF SIBLINGS ARE SAMPLED, TELL PARENT THE FOLLOWING: I am first going to ask about [NAME OF CHILD 1] and then I will ask about [NAME OF CHILD 2]. First, I will ask for the names of all members, then I will ask a series of questions about each. If you do not feel comfortable providing the name of the household member, provide us with the initials. ENUMERATORS: IF THEY DO NOT WANT TO GIVE NAMES, PLEASE ASK FOR INITIALS OR SOME OTHER WAY FOR US TO REFER TO THEM. COLLECT ALL NAMES FIRST IN 13A AND THEN ASK 13B-13I FOR EACH PERSON BEFORE MOVING ON TO THE NEXT HOUSEHOLD MEMBER. IF THEY DO NOT WANT TO PROVIDE THE INITIALS, TYPE IN RELATIONSHIP TO CHILD. E.G. FATHER, MOTHER, UNCLE.” Make sure to include everyone. [roster_msg] 13A [roster_name] 13B [roster_g ender] 13C [roster_rel] 13D [roster_age] 13E [roster_edu] 13F [roster_read ] 13G* [roster_readfr eq] 13H* [roster_read len] 13I* [roster_readdesc] Please tell me the name of each of the members of this household, starting with yourself, followed by your spouse if you have one, and then the other members from oldest to youngest Please note that no names will be entered in our reports in order to protect your privacy. What is [NAME]’ s GENDER ? 01 MALE 02 FEMALE What is [NAME]’s relationshi p to [NAME OF CHILD]? 11 PARENT 12 GRAND￾PARENT 13 AUNT/ UNCLE 14 SISTER/ BROTHER 15 COUSIN 99 OTHER How old is [NAME]? YEARS 98 ADULT, BUT DOES NOT KNOW AGE 99 CHILD, BUT DOES NOT KNOW AGE [IF HOUSEHOLD MEMBER IS 5 YEARS OR YOUNGER, SKIP TO NEXT MEMBER SKIP TO 13B] What is the highest level of education completed by [NAME] currently? 01 NONE 02 PRE-SCHOOL 03 GRADE 1 04 GRADE 2 05 GRADE 3 06 GRADE 4 07 GRADE 5 08 GRADE 6 09 GRADE 7 10 GRADE 8 11 GRADE 9 12 GRADE 10 13 GRADE 11 14 GRADE 12 15 VOCATIONAL / TECHNICAL 16 UNIVERSITY 17 OTHER POST￾SECONDARY 18 ADULT LITERACY ONLY (NO FORMAL EDUCATION) 19 KORANIC/RELIGIOUS ONLY (NO FORMAL EDUCATION) 99 OTHER Does [NAME] know how to read in ChiNyanja? 1 YES 0 DOES NOT KNOW HOW TO READ In a typical week, how often does [NAME] read with [NAME OF CHILD]? 01 Four Days A Week Or More 02 Two Or Three Days A Week 03 One Day A Week 04 Once Or Twice A Month 05 Less Than Once A Month 06 Never [SKIP TO NEXT HOUSEHOLD MEMBER] Each time [NAME] reads with [NAME OF CHILD], on average how long does he/she spend reading to [NAME OF CHILD]? MINUTES [IF 13H=0 SKIP TO NEXT HOUSEHOL D MEMBER] On average, when [NAME] and [NAME OF CHILD read, situation better describes these sessions? 01 [NAME] Reads And [NAME OF CHILD] Listens Most Of The Time 02 [NAME] Reads Half The TIME And [NAME OF CHILD] Reads The Other Half 03 [NAME OF CHILD] Reads And [NAME] Listens Most Of The Time 01 ________ |_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| 02 ________ |_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| 03 ________ |_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| 04 ________ |_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| ENUMERATOR RESERVE CODES: -9=REFUSED; -8=DON’T KNOW; -7=NOT APPLICABLE Page 45 05 ________ |_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| 06 ________ |_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| 07 ________ |_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| 08 ________ |_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| 09 ________ |_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| 10 ________ |_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| 11 ________ |_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| 12 ________ |_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| 13 ________ |_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| 14 ________ |_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| 15 ________ |_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| 16 ________ |_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| 17 ________ |_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| 18 ________ |_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| 19 ________ |_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| 20 ________ |_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| |_|_| ENUMERATOR RESERVE CODES: -9=REFUSED; -8=DON’T KNOW; -7=NOT APPLICABLE Page 46 14. What language do you use at your household most often to communicate with each other? [language] 01 CHINYANJA 02 NGONI 03 TUMBUKA 04 ENGLISH 05 BEMBA 06 CHICHEWE 07 SENGA 08 NSENGA 09 OTHER __________________ |_|_| I would now like to ask you some questions about the reading materials available to your household and in your community. 15. How many of the following are in your household: English? 01 None 02 1-5 03 6-10 04 More than 10 ChiNyanja? 01 None 02 1-5 03 6-10 04 More than 10 Other language? 01 None 02 1-5 03 6-10 04 More than 10 a Adult books (including religious materials) in [LANGUAGE]? 0 1 |_|_| 2 |_|_| 3 |_|_| b Children’s books (including learner’s text books) [LANGUAGE]? 4 |_|_| 5 |_|_| 6 |_|_| c Newspapers [LANGUAGE]? 7 |_|_| 8 |_|_| 9 |_|_| d Magazines [LANGUAGE]? 10 |_|_| 11 |_|_| 12 |_|_| e Posters [LANGUAGE]?13 |_|_| 14 |_|_| 15 |_|_| 16. A.* Does [CHILD’S NAME] have a reading materials (e.g. reader’s book, reading cards, or reading passages) from school? [readersbook] 01 Yes 02 No [SKIP TO Q07] |_|_| 16B*. Does [CHILD’S NAME] ever bring his/her reader’s book home from school? [bringhomebook] 01 Yes 02 No [SKIP TO Q07] |_|_| 16C*. If yes, on average, how many days a week did [CHILD’S NAME] bring it home in the last month? [bringhomebook_days] ENTER A NUMBER 0-7: |_| days/week |_| DAYS/WEEK 17. *How often does [CHILD’S NAME] read on his/her own at home? [readaloneathome] 01 Four Days A Week Or More 02 Two Or Three Days A Week 03 One Day A Week |_|_| ENUMERATOR RESERVE CODES: -9=REFUSED; -8=DON’T KNOW; -7=NOT APPLICABLE Page 47 04 Once Or Twice A Month 05 Less Than Once A Month 06 Never  [SKIP TO Q199] 18. *When [CHILD’S NAME] reads alone at home, how long does he/she usually read? [readaloneathome_mins] |_|_| MINUTES |_|_| 19. *Is there a place where [CHILD’S NAME] can participate in reading activities in your community? [readingactivities] 01 Yes 02 No  [SKIP to Q24; DON’T KNOW SKIP TO Q24] |_|_| 20. *Does [CHILD’S NAME] participate in any reading activities outside home after school, like reading clubs, visiting the school library, etc.? [readingactivitiespart] 01 Yes 02 No  [SKIP to Q244] |_|_| 21. *In which of these outside home reading activities does [CHILD’S NAME] participate? [CHECK ALL THE APPLY] [readingactivities_type] 01 READING CLUBS 02 READING ACTIVITIES AT THE COMMUNITY CENTER 03 READING THROUGH CHURCH ACTIVITIES 04 VISIT THE SCHOOL LIBRARY 05 READING WITH FRIENDS 06 READING WITH A MENTOR/OLDER PUPIL IN THE COMMUNITY 07 STORY-TELLING COMPETITIONS 08 READING COMPETITIONS/READ-A-THONS 09 WRITING CLUBS 10 OTHER:_________________________ |_|_| |_|_| |_|_| |_|_| |_|_| 22. *In a typical week, how often did [CHILD’S NAME] participate in any of these outside home activities (if [CHILD’S NAME] participated in more than one of these activities, consider the number of times he participated in total)? [readingactivities_often] 01 Four Days Or More A Week 02 Two Or Three Days A Week 03 Once A Week 04 Once Or Twice A Month 05 Less Than Once A Month |_|_| 23. *How long does [CHILD’S NAME] spend on these reading activities each time he/she participates, on average? [readingactivities_long] |_|_| MINUTES ENUMERATOR RESERVE CODES: -9=REFUSED; -8=DON’T KNOW; -7=NOT APPLICABLE Page 48 I would now like to ask you for your opinion on your child’s reading practices and progress. 24. *By the end of what grade do you expect [CHILD’S NAME] should be able to read? [expectread] |_|_| GRADE |_|_| 25. *Do you think [CHILD’S NAME] can read as well as a child his/her age is supposed to? [readatage] 01 Yes 02 No |_|_| 26. *Do you feel confident you can help [CHILD’S NAME] to learn how to read? [confident] 01 Yes 02 No |_|_| 27. *Would you say that helping your child learn how to read is … [helpingread_difficulty] 01 Very difficult 02 Difficult 03 Neutral  [SKIP TO SECTION C] 04 Easy  [SKIP TO SECTION C] 05 Very easy  [SKIP TO SECTION C] |_|_| 28. *Why is it difficult? [CHECK ALL THAT APPLY] [difficulty_why] 01 CANNOT READ WELL ENOUGH TO TEACH CHILD 02 DO NOT KNOW HOW TO HELP CHILD READ 03 DO NOT HAVE ACCESS TO APPROPRIATE READING MATERIALS 04 DO NOT HAVE ENOUGH TIME 05 DO NOT THINK THAT HELPING CHILD READ OUTSIDE OF SCHOOL WILL MAKE A DIFFERENCE 06 NO INTEREST FROM CHILD 07 CHILD DOES NOT HAVE ENOUGH TIME (E.G. HE/SHE HAS TO WORK) 08 OTHER: _____________________________________ |_|_| |_|_| |_|_| |_|_| |_|_| 29. Do you think that the responsibility of teaching a child to learn how to read is [responsibility]: 01 Primarily a responsibility of the teacher 02 Equally a responsibility of the parent and the teacher 03 Primarily a responsibility of the parent 04 Other: __________________________ |_|_| 30. Do you think it is important for children to read outside of school? [readoutside_imp] 01 Yes 02 No |_|_| SECTION C: HOUSEHOLD ASSETS An important part of this study is understanding how the effectiveness of the intervention changes with socioeconomic status- I would like to ask you a couple quick questions about your household’s assets. ENUMERATOR RESERVE CODES: -9=REFUSED; -8=DON’T KNOW; -7=NOT APPLICABLE Page 49 31. How many… Enter a number. If none, enter « 0 » 31A large livestock (e.g. oxen, cattle) does your household own? [livestock_large] |_|_|_|_|_| 31B small livestock (e.g. goats, pigs, sheep) does your household own? [livestock_small] |_|_|_|_|_| 31C plots of agricultural land does your household own? [plots] |_|_|_|_|_| 32. What type of material is your floor made of? [floor] 01 EARTH/MUD 02 CONCRETE/FLAG STONE/CEMENT 03 TILE/BRICKS 04 WOOD 05 OTHER: Specify____________ |_|_| 33. Does this household have… 1 YES 2 NO A a chair in good condition? [chair] |_| B a bed in good condition? [bed] |_| C a clock in good working condition? [clock] |_| D a radio in good working condition? [radio] |_| E a television in good working condition? [television] |_| F a computer in good working condition? [computer] |_| G a bicycle in good working condition? [bicycle] |_| H a motorcycle in good working condition? [motorcycle] |_| I a car, truck, or boat with engine in good working condition? [car] |_| J a refrigerator in good working condition? [fridge] |_| K a stove in good working condition? [stove] |_| 34. *Does [CHILD’S NAME] have… 1 YES 2 NO A at least two sets of clothes? [clothes] |_| B at least one pair of shoes? [shoes] |_| C A bed to sleep on? [mat] |_| 35. Is this household connected to the electric grid? [electricity] 01 Yes 02 No |_|_| 36. What is the main type of energy used for cooking in your household? [main_energy] 01 COLLECTED FIREWOOD |_|_| ENUMERATOR RESERVE CODES: -9=REFUSED; -8=DON’T KNOW; -7=NOT APPLICABLE Page 50 02 PURCHASED FIREWOOD 03 CHARCOAL OWN PRODUCED 04 CHARCOAL PURCHASED 05 COAL 06 KEROSENE/PARAFFIN 07 GAS 08 ELECTRICITY 09 SOLAR 10 CROP/LIVESTOCK RESIDUES 11 OTHER: SPECIFY ________________ 37. Who is the owner of the cell phone number [PIPED TEXT FROM Q10]? [owner] 01 RESPONDENT 02 RESPONDENT’S SPOUSE 03 RESPONDENT’S SIBLING 04 RESPONDENT’S PARENT 05 OTHER: SPECIFY____________ |_|_| 38. How often would you say you have a clear cell phone reception at home? [reception] 01 Always 02 Usually 03 About half the time 04 Seldom 05 Never |_|_| SECTION D: FOLLOW UP INFORMATION Thank you. As I said, we will come back to interview you in January 2017, and may visit you in April 2016. We want to make sure we can find you in case you move. 39. Do you plan to move to a new home before January 2017? [move] 01 Yes 02 No  [END INTERVIEW] |_|_| 40. What is address and telephone number where you plan to move? If you do not know where you will move, is there someone who we could contact who would know where you moved? If so, could you provide their contact information? [adult_newaddress] ___________________________________________________ 41. Will [CHILD’S NAME] move with you? [child_move] 01 Yes 02 No  [Skip to Q42] |_|_| 42. If so, what school will they be attending? [child_newschool] ____________________________________________________ 43. Do you plan to move to a new home before April 2016?[move_midline] 01 Yes 02 No  [END INTERVIEW] |_|_| 44. What is address and telephone number where you plan to move? [adult_newaddressmid] ___________________________________________________ ENUMERATOR RESERVE CODES: -9=REFUSED; -8=DON’T KNOW; -7=NOT APPLICABLE Page 51 We have now come to the end of interview. Do you have any immediate question/s about the interview? Thank you very much for participating in this survey! Please reach out to the number listed on the information brochure if you have any questions or concerns. Also, please reach out to the number if you do move. [end] 45. Household address/direction to household [full_address]: ______________________ 46. ENUMERATOR: WHERE DID YOU CONDUCT THIS INTERVIEW? [location] 01 School 02 Home 03 Other: Specify ______________________________ |_|_| Enumerator comments [comments]: ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ Supervisor comments [comments_sup]: ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ Supervisor ID [sup_no]: |_|_| Supervisor date [sup_date]: DD: |__|__| MM: |__|__|YYYY: |__|__|__|__| 52 II3. EGRA/STUDENT SURVEY INSTRUMENT General instructions Establish a playful and relaxed rapport with the child through a short conversation (see example topics below). The child should perceive the assessment almost as a game to be enjoyed rather than a test. Use this time to identify in what language the child is most comfortable communicating. Read aloud slowly and clearly ONLY the sections in boxes. Uli bwanji. Dzina langa ndine…………….. ndipo ndikhala ku………….... Ndingakonde kukuuza za moyo wanga. Good morning. My name is ____ and I live in _____. I’d like to tell you a little bit about myself. [Number and ages of children; favourite sport, radio or television program, etc.] 1. Kodi umakonda kucita ciani ngati siuli mu sukulu? What do you like to do when you are not in school? [Wait for response; if pupil is reluctant, ask question 2, but if they seem comfortable continue to verbal consent]. 2. Kodi ndi masewera otani amene umakonda kusewera? What games do you like to play? Verbal Consent: Read the text in the box clearly to the child.  Ndifuna kukuuza cifukwa cake ndabwera kuno lero. Ndigwira nchito mu unduna wa maphunziro mu Zambia ndipo tikufuna kumvetsetsa mmene ana amaphunzirira kuwerenga . Iwe wasankhidwa mwamwai. Let me tell you why I am here today. I work with the University of Zambia and we are trying to understand how children learn to read. You were picked by chance.  Ndifuna thandizo lako pa nkhaniyi. Koma suyenera kutengako mbali ngati sufuna. We would like your help in this. But you do not have to take part if you do not want to.  Ife tizachita sewero la kuwerenga. Ine ndizakufunsa kuwerenga malembo, mau ndi ka nthano kakafupi mokweza mau. Ndizakufunsanso kuzindikira ndi kuyankha mafunso ocepa. We are going to play a reading game. I am going to ask you to read letters, words and a short story out loud.  Mwakugwiritsa nchito nkoloko iyi, ndizaona nthawi imene utenga kuwerenga . Using this stopwatch/device/gadget, I will see how long it takes you to read.  Zimene tizachita pano si mayeso ndipo sizidzakhudza maphunzilo ako pasukulu lino. This is NOT a test and it will not affect your grade at school.  Ndizakufunsanso mafunso ena monga kumene umayeselera kuwerenga ndiponso ngati ukonda kuwerenga. . I will also ask you other questions about where you practice reading and whether you like it.  Sindizalemba dzina lako ndipo palibe aliyense adzadziwa za mayankho ako. I will NOT write down your name so no one will know these are your answers.  Kaciwirinso, sungatengeko mbali ngati sufuna kutero. Tikayamba kufunsa mafunso, ngati siufuna kuyankha funso ungakhale cete, zilibwino cabe. Once again, you do not have to participate if you do not wish to. Once we begin, if you would rather not answer a question, that’s all right.  Kodi uli ndi mafunso alionse? Do you have any questions?  Kodi wakonzeka kuti tiyambe? Are you ready to get started? Check box if verbal consent is obtained: *ii* YES (If verbal consent is not obtained, thank the child and move on to the next child, using this same form) IF CHILD AGREES, ASK HIM/HER TO WRITE NAME TO THE RIGHT 53 Demographics Date of assessment Assessor’s Name District: School Name: School EMIS Number: Pupil ID Sex:  Boy  Girl Grade:  G2  G3 Assessor ID: Date: / / / Disposition code Complete 01 Complete- school 02 Complete- home Will interview later 02 Absent (not at school) No interview 03 Not locatable (the child was not locatable at home) 04 Disability prevents taking the exam 05 Child too ill to take exam 06 Parent refuses interview 54 TASK 1: ORIENTATION TO PRINT  Page X  X Show the child a story passage in the pupil stimuli packet. Read the instructions in the gray boxes below, provide the child 10 seconds to respond, recording the child’s response before moving to the next instruction. If the child doesn't respond in the 10 seconds, mark as no response and move on. Materials: a passage from the pupil stimuli packet Sindifuna kuti uwerenge tsopano. Pa pepala iri, ungayambire kuti kuwerenga? Ndionetse ndi cala cako. I don’t want you to read this now. On this page, where would you begin to read? Show me with your finger. 1. (Child puts finger on the top row, left￾most word) O Correct O Incorrect O No Response Tsopano ndionetse mbali imene udzawerenga motsatira. Now show me in which direction you would read next. 2. (Child moves finger from left to right) O Correct O Incorrect O No Response Ukafika kotsirizira kwa mzere, udzawerenga kuti motsatira? When you get to the end of the line, where would you read next? 3. (Child moves finger to left-most word of second line) O Correct O Incorrect O No Response Total Correct /3 55 TASK 2: LETTER SOUND IDENTIFICATION  Page 1  60 seconds  Pano ndili ndi tsamba limene liri ndi malembo a alifabeti ya muchinyanja. Coonde ndiuze MAMVEKERO a malembo a alifabeti amene ungathe kuwerenga. Usanene maina ake. Koma mvekero zake. Here is a page full of letters of the ChiNyanja alphabet. Please tell me the SOUNDS of as many letters of the alphabet as you can. Not their names, but their sounds. [point to the letter A] Mwacitsanzo, mvekero la lembo ili ndi /a/. For example, the sound of this letter is /a/. [point to the letter p] Tiye tiyese: ndiuze mvekero la lembo ili: Let’s practice: Tell me the sound of this letter.  Cabwino, mvekero la lembo ili ndi /p/ Good, the sound of this letter is /p/.  Mvekero la lembo ili ndi /p/ The sound of this letter is /p/. [point to the letter L] Tsopano tiye tiyese lembo lina. Ndiuze mvekero la lembo ili. Now let us try another one. Tell me the sound of this letter.  Cabwino, mvekero la lembo ili ndi /l/. Good, the sound of this letter is /l/.  Mvekero la lembo ili ndi /l/. The sound of this letter is /l/. [point to first letter] Ndikanena kuti “yamba”, uyambire apa ndi kupitiriza mopingasa tsamba ili. Lata pa lembo lirilonse ndipo ndiuze mvekero la lembo limenelo mmau okweza. Uwerenge mwamsanga ndiponso modekha. Ngati wafika pa lembo limene sudziwa, pitiriza kupita ku lembo lotsatira. Ika cala cako pa lembo loyamba. Wakonzeka? Yamba. When I say “Begin,” start here and go across the page. Point to each letter and tell me the sound of that letter in a loud voice. Read as quickly and carefully as you can. If you come to a letter you do not know, go on to the next letter. Put your finger on the first letter. Ready? Begin. Start the timer when the child reads the first letter.  If a child hesitates or stops on a letter for 3 SECONDS, point to the next letter and say “Go on”  When the timer reaches 0, say “stop.”  If the child does not provide a single correct response on the first line (10 items), say “Thank you!”, discontinue this subtask, check the box at the bottom, and go on to the next subtask.  ( / ) Mark any incorrect letters with a slash ( Ø ) Circle self-corrections if you already marked the letter incorrect ( ] ) Mark the final letter read with a bracket Examples: A p L 1 2 3 4 5 6 7 8 9 10 m N K I d k A J m u (10) C d b o L I U K A w (20) G n a e s E A D I g (30) l r A a v f A T W i (40) D a t L N a A M i Y (50) t u z N i I N k e O (60) u Z P i U N i M i l (70) A p A a B W T k c M (80) a w N m E R a A h a (90) n A o l O n a U T S (100)  Time remaining on stopwatch at completion (number of SECONDS)  Exercise discontinued because the child had no correct answers in the first line Wacita bwino! Tiye tipitirize patsamba lotsatira Good effort! Let’s go on to the next section. 99 Don’t know/no response 56 Task3: Non-word Reading  Apa pali mau opangidwa mcinyanja. Ndifuna kuti uwerenge mau amene ungakwanitse kuwerenga, Uwerenge mau awa osati masipelingi. Here are some made-up words in ChiNyanja. I would like you to read as many as you can. Do not spell the words, but read them. [point to the word “oli”] Mwacitsanzo, liu lopangidwa ili ndi: “ola” For example, this made-up word is: “ola”. [point to the word “koki”] Tiye tiyese: conde werenga liu ili. Let’s practice: Please read this word.  wacita bwino. Liu ili ndi “koki” Good, This made-up word is “koki.”  Liu lopangidwa ili ndi “koki” This made-up word is “koki.” [point to the word “cota”] Tsopano tiye tiyese liu lina: conde werenga liu ili: Now let us try another one. Please read this word.  “wacita bwino, liu lopangidwa ili ndi “cota” Good, This made-up word is “cota.”  Liu lopangidwa ili ndi “cota” This made-up word is “cota.” [point to first word] Ndikanena kuti “yamba” uyambire apa ndipo uwerenge mopingasa patsamba ili. Lata liu lirilonse ndipo uliwerenge mokweza mau. Uwerenge mofulumira ndi mosamala mmene ungakwanitsire. Ngati wapeza liu limene sudziwa, pita ku liu lotsatira. Ika cala cako pa liu loyamba. Wakonzeka? Yamba. When I say “Begin,” start here [point to first word] and read across the page [point]. Point to each word and read it in a loud voice. Read as quickly and carefully as you can. If you come to a word you do not know, go on to the next word. Put your finger on the first word. Ready? Begin. Start the timer when the child reads the first word.  If a child hesitates or stops on a letter for 3 SECONDS, point to the next word and say “Go on”  When the timer reaches 0, say “stop.” 1.  If the child does not provide a single correct response on the first line (5 items), say “Thank you!”, discontinue this subtask, check the box at the bottom, and go on to the next subtask.  ( / ) Mark any incorrect words with a slash ( Ø ) Circle self-corrections if you already marked the word incorrect ( ] ) Mark the final word read with a bracket Examples: ola koki cota 1 2 3 4 5 nipe atapi gelu kelo mdzimu (10) ninane wondi umbe rizi ninda (20) ledesi fikiraku tomo ngalo zirama (30) yu ane mwane mukudi dzimo (40) liraku ia anuli wekusera dzimoli (50) cofukwa udi kubu anauna mtisinaka (60) wera eka diko amoi kasuci (70) ateta lia nacho komi labo (80) menepa ncheto ndaako nthua balo (90) mtanyama mtutu ndokonda mtingi ko (100)  Time remaining on stopwatch at completion (number of SECONDS)  Exercise discontinued because the child had no correct answers in the first line Wacita bwino! Tiye tipitirize patsamba lotsatira Good effort! Let’s go on to the next section. 99 Don’t know/no response 57 TASK 4a: ORAL READING PASSAGE (2014)  60 seconds TASK 4b: READING COMPREHENSION Show the child the sheet in the pupil stimulus booklet as you read the instructions.  If a child hesitates or stops on a letter for 3 SECONDS, say “Go on” 1.  If the child does not provide a single correct word on the first line of text. Do not ask any comprehension questions. 2. After the child is finished reading, REMOVE the passage from in front of the child. Ask the child only the questions related to the text read. A child must read all the text that corresponds with a given question. If the child does not provide a response to a question after 15 seconds, mark “no response” and continue to the next question. Do not repeat the question.  Apa pali ka nthano kakafupi. Ndifuna kuti uwerenge mokweza, mofulumira komanso mosamala. Ukatsiriza kuwerenga, ndizakufunsa mafunso onena za nkhani imene wawerenga. Ndikanena kuti “ yamba,” uwerenge bwino kwambiri mmene ungakwanisire. Ngati wapeza liu limene sudziwa, pita ku liu lotsatira. Ika cala cako pa liu loyamba. Wakonzeka? Yamba. Here is a short story. I want you to read it aloud, quickly but carefully. When you finish, I will ask you some questions about what you have read. When I say “Begin,” read the story as best as you can. If you come to a word you do not know, go on to the next word. Put your finger on the first word. Ready? Begin.  Tsopano ndidzakufunsa mafunso ocepa onena za nthano imene wawerenga. Yesa kuyankha mafunso mmene ungakwanisire. Now I am going to ask you a few questions about the story you just read. Try to answer the questions as well as you can.  ( / ) Mark any incorrect letters with a slash ( Ø ) Circle self-corrections if you already marked the letter incorrect ( ] ) Mark the final letter read with a bracket  () 1 = Correct () 0 = Incorrect () . = No response. Questions [Answers] Amai anapita kumsika m’masana tsiku lina. 6 Ndani anapita kumsika? (Amai) 1 0 . Anasiya mwana ndi mkulu wake Dolika. 12 Mwana anatsala ndi ndani? (Dolika) 1 0 . Anzake a Dolika anabwera kudzamtenga pamodzi ndi mwanayo. Dolika ndi anzake anaphunzitsa mwana kuyimba. Anamuphunzitsa nyimbo ya alifabeti. 30 Kodi mwana anaphunzitsiwa kucita ciani? (Kuyimba, Kuyimba nyimbo ya alifabeti) 1 0 . Atabwerako kumsika amai, anapeza mwana ali kuyimba. 37 Kodi mwana anadziwa bwanji kuyimba nyimbo ya alifabeti? Dolika ndi anzake anamphunzitsa) 1 0 . Amai anakondwera kwambiri. 40 N’cifukwa ciani amai anakondwera? (Mwana anali kuyimba) 1 0 .  Time remaining on stopwatch at completion (number of SECONDS)  Exercise discontinued: the child had no correct answers in the first line Wacita bwino! Tiye tipitirize patsamba lotsatira Good effort! Let’s go on to the next section. 99 Don’t know/no response 58 TASK 5: LISTENING COMPREHENSION (2014)  X  X  Ndidzakuwerengera ka nthano/nkhani mokweza KAMODZI ndipo pambuyo pake ndidzakufunsa mafunso. Conde umvetsere mosamalira ndipo uyankhe mafunso mmene ungakwanitsire. Wakonzeka? Tiye Tiyambe. I am going to read you a short story aloud ONCE and then ask you some questions. Please listen carefully and answer the questions as best as you can. Ready? Let’s begin. Remove the pupil stimuli booklet from the child’s view. Do not allow the child to look at the passage or the questions.  () 1 = Correct () 0 = Incorrect () . = No response. Patsiku Lolemba, Mangani anapita kusukulu. Ananyamula mabuku ndi nyama m’chola cake. Pamene anali kuyenda, anapeza galu wamkulu panjira. Anafuna kuthawira pathengo koma anagwa pansi. Yunifomu yake inada ndipo galu anatenga nyama yake. Mangani anathawira kunyumba. Pamene anafika kunyumba, m’bale wake anamubwereka yunifomu yake. Anakondwera. Ndi tsiku liti pamene Mangani anapita kusukulu? (Pa Lolemba) 1 0 . Ananyamula ciani mu chola cake? (Mabuku ndi nyama) 1 0 . N’ciani cimene anapeza panjira? (Anapeza galu wamkulu) 1 0 . Ndi cifukwa ciani Mangani anathawa galu? (Anaopa kuti galu angamulume, nyama, Anaopa, Galu wamkulu) 1 0 . Ndi cifukwa ciani m’bale wake anamubwereka yunifomu Mangani? (Cifukwa yunifomu yake inada, Anagwa ). 1 0 . Wacita bwino! Tiye tipitirize patsamba lotsatira Good effort! Let’s go on to the next section. 99 Don’t know/no response 59 TASK 6: READING PRACTICES (ZOCHITA 6: MAWERENGEDWE) I am going to ask you some questions about the time you spend reading. If you do not want to answer any question, or do not know the answer to a question, you do not have to answer. Ndizakufunsa mafunso pa nthawi imene umakhala uwerenga. Ngati siufuna kuyanka funso lilionse mwina kapena siudziwa yanko, usayankhe. Can I begin asking you these questions? Kodi ndiyambe kufunsa mafunsowa sopano? (Inde 01, Ai 02 SKIP TO END) |__|__| Some of the questions I am going to ask you are about what you like and do not like to do. I’m going to show faces. Mafunso ena ndizakufunsa ndi a zinthu zimene ukonda ndi zimene siukonda kuchita. Ndizaku sonyedza zikope-kope. chikope ichi: chitanthauza kuti ukonda kuchita zinthu izi, [ENUMERATOR: POINT TO FACE] chikope ichi: chiyimilira pa zinthu zimene sidzikudetsa nkawa pa kuzichita kapena kusazichita koma umazichita. [ENUMERATOR: POINT TO FACE] chikope ichi: chiyimilira pa zinthu zimene siukonda kuchita. [ENUMERATOR: POINT TO FACE] Point to the face that shows how you feel. Sopano lata pa chikope-kope chimene chisonyedza m’mene umvelera. Q1 [SHOW SMILEY FACE SCALE] Do you like to play? Point to how you feel. Kodi ukonda kusewela? Lata zomwe zisonyedza m’mene umvelera. [RECORD ANSWER] |__|__| Q2 [SHOW SMILEY FACE SCALE] |__|__| 01 Ndizikonda. (I like it) 02 Sidzinidetsa nkawa (I do not like it or dislike it) 03 Sindizikonda (Don’t know/No response) 99 Don’t know/no response 60 Do you like to go to school? Point to how you feel. Kodi ukonda kupita ku sukulu? Lata zomwe zisonyedza m’mene umvelera. [RECORD ANSWER] Q3 [SHOW SMILEY FACE SCALE] Do you like to listen to stories? Point to how you feel. Kodi ukonda kunvelera nthano? Lata zomwe zisonyedza m’mene umvelera. [RECORD ANSWER] |__|__| Q4 Do you know how to read? Kodi udziwa kuwerenga? 1 Inde (Yes) 0 Ai (No) 99 = Kulibe yanko (Don’t know/no response) |__|__| Q5 [SHOW SMILEY FACE SCALE] Do you like to read or like to practice reading? Point to how you feel. Kodi ukonda kuwerenga kapena ukonda kuyetselera kuwerenga? Lata zomwe zisonyedza m’mene umvelera. [RECORD ANSWER] |__|__| Q6 Do you read or try to read on your own at home? Kodi umawerenga kapena umayetselera kuwerenga pa iwe wekha ku nyumba? 1 Inde (Yes) 0 Ai (No) [SKIP TO QUESTION 9] |__|__| Q7 [SHOW SMILEY FACE SCALE] Do you like to read or try to read on your own at home? Point to how you feel. Kodi ukonda kuwerenga kapena umayetselera kuwerenga pa iwe wekha ku nyumba? [RECORD ANSWER] |__|__| Q8 [multiple select] Last week, on which days did you read or try to read on your own at home? [ENUMERATOR SELECT ALL THAT APPLY] Mu sabata lata, ndi matsiku ati yomwe unawerenga kapena unayetselera kuwerenga pa iwe weka ku nyumba? 1 Tsiku lolemba. (Monday) 2 Tsiku yaciwiri mu sabata (Tuesday) 3 Tsiku yacitathu mu sabata (Wednesday) 4 Pa cinai (Thursday) 5 Pa cisanu (Friday) 6 Pa ciwero (Saturday) 7 Pa sondo (Sunday) 8 Sindinawerengeko mu sabata latha. (I did not do this activity last week) |__|__| Q9 Do you read or try to read in class alone? Kodi umawerenga kapena umayetselera kuwerenga pa iwe wekha |__|__| 99 Don’t know/no response 61 mu kalasi? 1 Inde (Yes) 0 Ai (No) [SKIP TO QUESTION 12] Q10 [SHOW SMILEY FACE SCALE] Do you like to read or try to read in class alone? Kodi ukonda kuwerenga kapena kumayetselera kuwerenga pa iwe wekha mu kalasi? [RECORD ANSWER] |__|__| Q11 [multiple select] Last week, on which days did you read or try to read in class alone? Mu sabata latha, ndi matsiku ati yomwe unawerenga kapena unayetselera kuwerenga pa iwe wekha mu kalasi? 1 Tsiku lolemba. (Monday) 2 Tsiku yaciwiri mu sabata (Tuesday) 3 Tsiku yacitathu mu sabata (Wedneday) 4 Pa cinai (Thursday) 5 Pa cisanu (Friday) 6 Sindinawerengeko mu sabata latha (I did not do this activity last week) |__|__| Q12 Do you read or try to read in class out loud? Kodi umawerenga kapena kuyetselera kuwerenga mu kalasi mokweza liu? 1 Inde (Yes) 2 Ai (No) [SKIP TO QUESTION 15] |__|__| Q13 [SHOW SMILEY FACE SCALE] Do you like to read or try to read in class out loud? Kodi ukonda kuwerenga kapena kuyetselera kuwerenga mokweza liu mu kalasi? [RECORD ANSWER] |__|__| Q14 [multiple select] Last week, on which days did you read or try to read in class out loud? Mu sabata latha, ndi matsiku ati omwe unawerenga kapena unayetselera kuwerenga mu kalasi mokweza liu? 1 Tsiku lolemba. (Monday) 2 Tsiku yaciwiri mu sabata (Tuesday) 3 Tsiku yacitathu mu sabata (Wednesday) 4 Pa cinai (Thursday) 5 Pa cisanu (Friday) 6 Sindinawerengeko mu sabata latha. (I did not do this activity last week) |__|__| Q15 Does anyone read or help you try to read with you at home? Kodi kuli omwe ama werenga kapena omwe ama kutandiza kuyetselera kuwerenga ku nyumba? 1 Inde (Yes) 0 Ai (No) [SKIP TO QUESTION 21] 99= kulibe yanko (Don’t know/no response) [SKIP TO QUESTION 21] |__|__| 99 Don’t know/no response 62 Q16 [multiple select] Who do you read with, or who helps you try to read, at home the most? Kodi umawerenga ndi ndani, kapena mwina ndani omwe ama kutandizira kuyetselera kuwerenga ku nyumba kawiri-kawiri? 1 Amai (Mother) 2 Atate (Father) 3 Amai opezamo (Stepmother) 4 Atate opezamo (Stepfather) 5 Amai akulu kapena ang’ono (Aunt) 6 Atsibweni (Uncle) 7 Ambuya amuna (Grandfather) 8 Ambuya akadzi (Grandmother) 9 Akalongosi (Sister) 10 Abale (Brother) 11 M’zako (Friend) 12 Ena, simikiza (Other, specify) ________ |__|__| Q17 Do you read or try to read with anyone else at home? Kodi umawerenga kapena kuyetselera kuwerenga ndi munthu wina ku nyumba? 1 Inde 0 Ai [SKIP TO QUESTION 19] |__|__| Q18 [multiple select] Who else do you read with, or who else helps you try to readat home [SELECT ALL THAT APPLY] Ndani wina omwe umawerenga naye kapena omwe ama kutandizira kuyetselera kuwerenga ku nyumba? [sankani mayanko yonse omwe apasa] 1 Amai (Mother) 2 Atate (Father) 3 Amai opezamo (Stepmother) 4 Atate opezamo (Stepfather) 5 Amai akulu kapena ang’ono (Aunt) 6 Atsibweni (Uncle) 7 Ambuya amuna (Grandfather) 8 Ambuya akadzi (Grandmother) 9 Akalongosi (Sister) 10 Abale (Brother) 11 M’zako (Friend) 12 Ena, simikiza (Other, specify) ________ |__|__| Q19 [multiple select] Last week, on which days did anyone read to you at home, or help you try to read? Mu sabata latha, ndi matsiku ati omwe munthu wina anakuwerengela kapena kukutandiza kuyetselera kuwerenga ku nyumba? 1 Tsiku lolemba. (Monday) 2 Tsiku yaciwiri mu sabata (Tuesday) 3 Tsiku yacitathu mu sabata (Wednesday) 4 Pa cinai (Thursday) 5 Pa cisanu (Friday) 6 Pa ciwero (Saturday) 7 Pa sondo (Sunday) 8 Sindinawerengeko mu sabata latha. (I did not do this activity last week) |__|__| Q20 [SHOW SMILEY FACE SCALE] |__|__| 99 Don’t know/no response 63 Do you like to read or try to read with someone at home? Point to how you feel. Kodi ukonda kuwerenga kapena kuyetselera kuwerenga ndi munthu wina ku nyumba? lata zomwe zisonyedza m’mene umvelera. [RECORD ANSWER] Q21 [multiple select] Last week, which days did you attend school? INTERVIEWER: SELECT ALL THAT APPLY Mu sabata latha, ndi matsiku ati omwe unapita ku sukulu? 0 Kosapitako ai (sanapiteko sabata yonse) (Zero days) [SKIP TO END OF INTERVIEW] 1 Tsiku lolemba. (Monday) 2 Tsiku yaciwiri mu sabata (Tuesday) 3 Tsiku yacitathu mu sabata (Wednesday) 4 Pa cinai (Thursday) 5 Pa cisanu (Friday) |__|__| Q22 “Last week, on the days you were in school, was your teacher in school present on all of those days?” Sabata latha pa masiku unapita kusukulu kodi aphunzisi anabwera masiku onse? 1 Inde (Yes) 0 Ai (No) 99 Don’t know/No reponse |__|__| “Wow, you did a great job today! We are done now. Thank you for your help, here is a small token of thanks. You can go back to class now. Have a good day!” Aaaah! Wacita nchito yabwino lelo. Tatsiliza tsopano. Zikomo pathandizo lako. Aka ndikamphatso kako. Ungabwelere ku kilasi tsopano. Tsiku labwino. 99 Don’t know/no response 64