SOMA UMENYE IMPACT EVALUATION AID-696-TO-16-00002 EVALUATION - YEAR 1 BASELINE REPORT EXECUTIVE SUMMARY April 26, 2018 This Year 1 Baseline Report was prepared for USAID/Rwanda by International Business & Technical Consultants, Inc. (IBTCI) under Task Order AID-696-TO-16-00002. The authors are Edward Jay Allan, Magda Raupp, Ali Protik, Kenn Ndirangu, and Nikita Ramchandani. The views expressed in this document do not necessarily reflect the views of the United States Agency for International Development or the United States Government. SOMA UMENYE IMPACT EVALUATION AID-696-TO-16-00002 EVALUATION - YEAR 1 BASELINE REPORT EXECUTIVE SUMMARY APRIL 26, 2018 Edward Jay Allan, IBTCI, Project Director Magda Raupp, IBTCI, Team Leader Ali Protik, Mathematica Policy Research, Methodologist Kenn Ndirangu, Incisive Africa, Data Manager Nikita Ramchandani, Mathematica Policy Research, Research Analyst International Business & Technical Consultants, Inc. (IBTCI) 8618 Westwood Center Drive Suite 400 Vienna, VA 22182 USA DISCLAIMER This Year 1 Report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents are the sole responsibility of IBTCI and do not necessarily reflect the views of USAID or the United States Government. i EXECUTIVE SUMMARY Introduction and Background Soma Umenye (SU), “Read and Know,” is a five-year program implemented by Chemonics International to develop and improve Kinyarwanda reading skills of students in the first three years of primary school. SU is intended to improve quality of early literacy instruction and strengthen the capacity of the education system in Rwanda to implement and sustain these improvements. USAID awarded International Business & Technical Consultants, Inc. (IBTCI) a parallel five-year task order to conduct an Impact Evaluation (IE) of SU, including other studies. The IE planned to incorporate the 2017, 2018, and 2019 school years. Because of acceleration of implementation of SU to include all schools, including the control schools, in 2018, the IE task order was modified to cover only the 2017 school year. This report describes the planning and implementation of the Impact Evaluation starting in October 2016 and presents the impact of Soma Umenye at the end of the first school year (October 2017). To assess the impact of Soma Umenye, the Early Grade Reading Assessment (EGRA) was adapted for Rwanda and Kinyarwanda, Rwanda’s major language. Field tests and revision took place in August 2017, and in September/October 2017 the EGRA was administered to a random sample of 5,457 P1 students in 304 Treatment and Control schools throughout Rwanda. During these school visits, a student context questionnaire and teacher and head teacher surveys were administered. Chemonics International, the implementer, led the EGRA adaptation effort, the development of the additional data collection instruments, and the field-testing of all instruments. IBTCI, the Impact Evaluator, provided feedback to the process and to the instruments as needed. Data collection responsibilities for piloting and full collection were shared by the two organizations: Chemonics collected the data on 2,454 students and on teachers in randomly selected classes at the randomly selected treatment schools while IBTCI did the same on the 3,003 students and on teachers at the control schools. The two organizations shared the data sets to conduct the analyses. However, the focus of the analyses was different—IBTCI examined the impact of SU on student reading competency and developed a baseline for study of change in teacher instructional behavior. The focus of the implementer was primarily on the achievement of SU performance indicators. It is important to note that during the 2017 school year only an abridged version of the full intervention package was implemented. SU provided eight days of training to P1 Kinyarwanda teachers in treatment schools on the principles of effective reading instruction. Head Teachers, Directors of Studies, and District and Sector Education Officers received a two-day orientation on evidence-based reading instruction. Figure ES1 (Figure 2.1 in Section 2) shows the activities implemented in 2017 vis-à-vis the full package of intervention to be implemented at the school level in years 2 – 4. SOMA UMENYE IMPACT EVALUATION - Year 1 Baseline Report – Executive Summary Figure ES1. The school-based package implemented in 2017 vis-à-vis the full package to be implemented in years 2 - 4 Implemented in Year 1 (2017) • Eight days of face-to-face training on the principles of effective reading instruction; • Two-day orientation to evidence-based reading instruction for Head teachers, Directors of Studies, and District and Sector Education Officers; • Distribution of government-selected P1 Kinyarwanda textbooks to all schools. Completed in August 2017, to reduce the overall student-textbook ratio to 3:1. Note: Because all schools received these materials, this component of the package was not included in the Impact Evaluation. Full package to be implemented in Years 2 – 4 • Distribution of essential core of Kinyarwanda teaching and learning materials. For P1, this includes the SU-prepared Kinyarwanda reading textbook (1 textbook per child), teacher guide (1 guide per teacher), read aloud story book (1 book per teacher), and a set of decodable readers (24 titles, one per student in class) and leveled readers (1 set of 76 titles per classroom). For P2, the essential core consists of the student textbook (1 textbook per child), teacher guide (1 guide per teacher), read aloud story book (1 book per teacher), and a set of leveled readers (1 set of 150 titles per classroom). The P3 essential core includes the student textbook (1 textbook per child), teacher guide (1 guide per teacher), read aloud story book (1 book per teacher), and a set of leveled readers (1 set of 150 titles per classroom); • Teacher continuing professional development (CPD)—ten days of face-to-face training focused on use of the materials, plus coaching/mentoring, self-learning videos, and termly community of practice meetings; • Ten days of face-to-face training for Kinyarwanda mentors (directors of studies, head teachers, or Kinyarwanda school subject leaders) to conduct structured classroom observations and mentoring; • Two days of face-to-face training for head teachers to enable them to support reading instruction and become instructional leaders in their schools; • Print-rich classrooms—alphabet charts, pocket boards and flash cards. Methodology and Impact Evaluation Design The Soma Umenye Impact Evaluation was designed as a randomized controlled trial (RCT), where a randomly assigned group that does not receive the intervention—the control group—is used as the counterfactual. The difference in outcomes between the beneficiaries of the intervention (the treatment group) and the control group is the measure of impact. By constructing a counterfactual, an impact evaluation is able to assign observed changes in outcomes to the intervention and clarify what the outcomes would have been in the absence of the intervention. The Methodology and the design of the Impact Evaluation are summarized in Section 3 and detailed in Annex C, Evaluation Design and Year 1 Workplan. The EGRA was administered to individual students in Kinyarwanda and consisted of six sub-tests: (1) Letter Recognition; (2) Syllable Recognition; (3) Familiar Word Reading; (4) Oral Comprehension; (5) Fluency (reading connected text); and, (6) Reading Comprehension. Naming letters, enunciating the sounds of syllables, and reading isolated words are foundational skills needed for reading fluency. These timed sub-tests allow us to assess whether students are achieving the desired level of automaticity in these skill areas. Oral Comprehension and Reading Comprehension required students either to listen or to read a story and answer five questions to assess their understanding of what they heard or read. Students who were unable to perform a single item on a sub-test received a zero score on that sub-test. SOMA UMENYE IMPACT EVALUATION - Year 1 Baseline Report – Executive Summary Findings and Conclusions This report addresses specifically Evaluation Questions 1 (student reading competency in Kinyarwanda) and 2 (the quality of literacy instruction in early grade reading classes). Our analysis provides initial information on these evaluation questions, but more work would be needed to assess the impact of Soma Umenye’s full package on students’ reading ability as well as on the other expected effects of the program. Evaluation Questions 3 and 4 were scheduled to be addressed in subsequent reports using data collected in the 2018 and 2019 school year.1 The key findings are summarized below and detailed in Section 7. ● Evaluation Question # 1: To what extent are changes in Kinyarwanda reading outcomes for students in Grades 1-3 attributable to the Soma Umenye activity (as a package)? 1. Impact of Soma Umenye on student reading competency (EGRA timed sub-tests) The average scores obtained by P1 students in treatment schools were compared to students’ average scores in schools that received no intervention (control schools). Table ES.1 highlights where the differences between the two groups are significant. Note that the number of items per sub-test differs. Our analyses show very low overall performance on the four timed sub-tests. On the average, students identified only 20 percent of the 100 (19 to 23 letters); they recognized a little over 10 percent of the 100 syllables; and were able to read fewer than 5 of the 50 familiar words presented to them. In two of the four sub-tests, the differences noted between students in treatment and in control schools were statistically significant, showing the impact of Soma Umenye. Students in the treatment group had an average score of 4.51 on the reading of familiar words sub-test while students’ in the control groups had an average score of 3.72 words per minute. Even though the difference is statistically significant on the two sub-tests, our findings show that overall, P1 students in both groups are not acquiring the lower-order reading skills necessary to become fluent readers in later grades. 1 In February 2018 USAID communicated to IBTCI that as the result of the accelerated expansion of Soma Umenye, the Impact Evaluation would be discontinued. SOMA UMENYE IMPACT EVALUATION - Year 1 Baseline Report – Executive Summary Table ES.1 Scores on the EGRA Timed Sub-Tests by Treatment and Control Group EGRA timed sub-test scores Treatment Control Difference between groups Letter Identification (MAX 100 clpm) 22.19 19.41 2.77* Syllable identification (MAX 100 csilpm) 12.11 11.19 0.93 Familiar Word Reading (MAX 50 cwpm) 4.51 3.72 0.79* Reading Fluency (MAX 38 orf) 4.98 4.27 0.72 N=5,466 * Differences between treatment and control statistically significant at .01/.05 level. clpm: correct letters per minute; csilpm: correct syllable per minute; cwpm: correct words per minute; orf: oral reading fluency (number of words of connected text read correctly in one minute) Nevertheless, students in the treatment schools achieved gains in reading performance that were not achieved in control schools. Considering the difference between the intended package of intervention planned to be implemented in years 2 – 4 and the abridged package that was possible to implement in 2017, we hypothesize that in 2018, and even more so in 2019, the effects of the project could be larger and that the gap between students in the treatment and in control schools could widen.2 2. Impact of Soma Umenye on student reading competency (EGRA untimed sub￾tests) Table ES.2 shows the average scores for students in the treatment and control groups, the differences observed between the two groups for the sub-tests Listening Comprehension and Reading Comprehension. Table ES.2 Scores on the EGRA Untimed Sub-Tests for Treatment and Control Group EGRA untimed sub-test scores Treatment Control Difference between groups Listening Comprehension (MAX 5 questions) 3.91 3.78 0.13* Reading Comprehension (MAX 5 questions) 0.69 0.60 0.09 N=5,457 * Differences between treatment and control statistically significant at .05 level. Both groups scored high on the Listening Comprehension sub-test—3.84 on average out of a possible score of 5, showing students’ ability to listen to a story, understand what was read to them, and correctly answer oral comprehension questions. Overall students scored 78 percent, close to the proficiency level of 80 percent, answering four questions out of five correctly. It is also possible that the sub-test was too easy for most students and this caused the scores to concentrate heavily at the upper end of the allowable range, restricting the variation in scores (ceiling effect). 2 Since the IE was interrupted, the effective impact of SU may not be knowable. SOMA UMENYE IMPACT EVALUATION - Year 1 Baseline Report – Executive Summary Average scores on the Reading Comprehension sub-test were extremely low for both groups, 0.64 on average out of a possible 5.0. Overall, 3.09 percent students (N=169) scored 80 percent or higher and 0.26 percent (N=14) scored 100 percent. These 183 students total are considered able to read grade-level text with comprehension. In the treatment group, 3.18 percent of the students (N=78) scored 80 percent or higher and in the control group, 3.02 percent (N=91) scored 80 percent or higher. The difference between treatment and control is not statistically significant. ORF scores for proficient readers—students who scored 80 percent or higher in reading comprehension—was 27.8 correct words per minute. Because of the low performance in words per minute (Table ES.1), it is not surprising that scores concentrated heavily at the lower end of the score distribution. EGRA studies have shown the high correlation (0.91) between correct words read per minute (fluency) and reading comprehension.3 We also observed this high correlation between oral reading fluency and reading comprehension in our sample, which was 0.92, Other findings that resulted from the analyses of the EGRA data are summarized below. The findings are detailed in Section 7. 3. EGRA findings for male and female students Soma Umenye had statistically significant positive impacts on female students in the treatment group compared to female students in the control group on three sub-tests: letter identification, syllable identification, and familiar word reading. The intervention also had statistically significant positive impacts on male students in the treatment group compared to male students in the control group on the familiar word reading sub-test. Given that the impacts on male students were not different from the impacts on female students, we conclude that, on average, SU benefitted female and male students equally. 4. EGRA findings for Repeaters and Non-Repeaters Soma Umenye had statistically significant positive impacts on both groups of students. Repeaters and non-repeaters in the treatment group scored better in the letter identification sub-test compared to repeaters and non-repeaters in the control group. These findings suggest that while repeaters may have derived some benefits from their previous academic experience, even an intensive reading intervention such as Soma Umenye did not benefit them any more than it did the non-repeaters. 5. EGRA findings for Students in Urban and Rural Sectors Our sample was overwhelmingly rural, 90.4 percent or 4,932 students were from 275 schools in rural sectors and only 525 (9.6 percent) were from 29 schools in urban sectors.4 Because rural students represented such a high proportion of the total sample, the impacts of SU on them closely resembles the impacts on the total sample and we are not able to conclude that SU benefitted rural students more or less than their urban counterparts. 3 Fuchs,L., Fuchs,D., Hosp,M.K. & Jenkins, J. (2001) Oral reading fluency as an indicator of reading competence: A theoretical, empirical, and historical analysis. 4 We used the official definition of rural-urban status of sectors as provided by the Rwanda Education Board. SOMA UMENYE IMPACT EVALUATION - Year 1 Baseline Report – Executive Summary ● Evaluation Question # 2: To what extent has classroom instruction changed as a result of Soma Umenye? Soma Umenye and IBTCI enumerators conducted a total of 281 observations of the Kinyarwanda reading lesson at the class randomly selected in each school for EGRA participation, using a lesson observation protocol developed by SU. Teachers were scored on five skills: (i) lesson preparation, (ii) teaching methodology, (iii) literacy technical skills, (iv) assessment techniques, and (v) use of teaching materials. Teachers in treatment schools, who participated in SU training received an average composite score that was half a standard deviation better compared to the score obtained by control group teachers and the difference is statistically significant. Because Evaluation Questions 3 and 4 call for data that would have required that the IE be continued, it is not possible to answer them in this Year 1 report. Summary Conclusion The main focus of the impact evaluation is the comparison between the EGRA scores obtained by students in intervention schools and their counterparts in control schools. These comparisons show an initial impact of Soma Umenye on two sub-tests: Letter Identification and Familiar Word Reading. Given the limited intervention implemented in 2017, these findings are encouraging. However, the great majority of students did not demonstrate that they were acquiring the lower￾order reading skills that we can expect at P1—they identified only 20 percent of the 100 letters presented to them, recognized a little over 10 percent of the 100 syllables, and were able to read fewer than 5 of the 50 familiar words. Higher-order reading skills (fluency and comprehension) build on lower order skills (phonemic awareness, letter sound knowledge and, decoding). EGRA tests were developed to be predictive of later reading achievements, and numerous administrations of EGRA in multiple countries and languages have confirmed the correlations between lower- and higher-order reading skills.5 Overall, findings allow us to conclude the great majority of students are not learning the foundational skills that will allow them to become fluent readers by the end of P3. Improving the teaching—and consequently the learning—of foundational reading skills in P1 and P2, using EGRA findings to establish standards and benchmarks, and training reading teachers and headmasters on how to enhance students’ reading outcomes should therefore be an urgent priority. 5 Early Grade Reading Assessment (EGRA) Toolkit. Page 18, 2nd Edition, March 2016. PERFORMANCE EVALUATION OF HICD/R PROJECT – FINAL REPORT 112 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 Tel: (202) 712-0000 Fax: (202) 216-3524 www.usaid.gov